0 Fork 0

mirror of https://github.com/cozystack/cozystack.git synced 2026-11-03 02:27:01 +09:00

Files

T

Marian Koreniuk cd983d416e fix(e2e): take the selector's type test in the allowlist scan and pin its regex

lg_suite_has_test dropped every directory with `! -type d`, but the
selector keeps a chainsaw-test.y*ml whatever its type, so Chainsaw can
fail loudly on a directory by that name. An allowlisted suite in that
shape failed validation and stopped the run before Chainsaw reported the
real error. Use the selector's own `\( ! -type d -o -name
chainsaw-test.yaml -o -name chainsaw-test.yml \)`, which still drops a
directory with a step-file name.

The step-file regex behind the find glob was never reached by a test:
install-01.yaml is already filtered by the glob. 1a-install.yaml passes
the glob and not the regex, so it pins the regex; a directory named
chainsaw-test.yaml and one named 01-install.yaml pin the type test.

The README said the runner recognises the same three layouts. It
recognises none: it hands every top-level directory to Chainsaw.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM

2026-10-28 18:53:07 +01:00

6.9 KiB

Raw Permalink Blame History

Chainsaw E2E tests

The Cozystack E2E app suite, written for Kyverno Chainsaw — a declarative, Kubernetes-native E2E test framework (CNCF, part of the Kyverno project; successor to KUTTL).

This directory replaces the per-app BATS suite that used to live in hack/e2e-apps/*.bats. The CI e2e job runs every suite under hack/e2e-chainsaw/ through hack/e2e-chainsaw-run.sh after make install-cozystack; cluster bootstrap (hack/e2e-prepare-cluster-container.bats, hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.

Layout

hack/e2e-chainsaw/
├── .chainsaw.yaml          # global Configuration (namespace, timeouts, JUnit report)
├── _lib/                   # shell reused by the script-heavy suites
│   ├── run-kubernetes.sh   #   tenant Kubernetes bringup + LB/NFS/ouroboros checks
│   └── remediation-guard.sh
├── <app>/
│   ├── chainsaw-test.yaml  # one or more Test definitions
│   └── <app>.yaml          # the manifests the test applies
└── ...

A suite is a directory directly under hack/e2e-chainsaw/ in or below which Chainsaw finds a Test. Three layouts do that, and the selector (hack/select-e2e.sh) and the leak-guard allowlist both recognise the same three:

chainsaw-test.yaml, the usual one, or chainsaw-test.yml when the .yaml is absent;
numbered step files such as 01-install.yaml, which Chainsaw turns into a Test on its own when neither of the above exists;
either of those in a subdirectory of the suite, since Chainsaw walks every subdirectory of the directory it is handed.

A parked chainsaw-test.yaml.disabled is none of these: the directory is then not a suite, is selected by nothing and cannot be allowlisted.

Suites: postgres, bucket, mariadb, mongodb, redis, valkey, qdrant, clickhouse, kafka, kafka-metadata, rabbitmq, etcd, openbao, harbor, foundationdb, external-dns, kuberture, vminstance, gateway, opensearch, monitoring, kubernetes-latest, kubernetes-previous, securitygroup, seaweedfs, computeplane. kubernetes-proxmox is parked as chainsaw-test.yaml.disabled and runs only in .github/workflows/e2e-proxmox.yaml, which needs a Proxmox hypervisor.

What Chainsaw buys over the BATS suite

Readable tests — a list of resources and expected states, not shell control flow. The timeout N sh -ec "until kubectl get ..." + kubectl wait pair that appeared ~100 times collapses into a single assert that polls existence and state together.
Distinguishable failure modes — "object never created", "created but never ready", and "wrong status" all surface as a structured diff instead of an opaque shell timeout.
Automatic diagnostics — catch blocks dump events, describe, and pod logs on any failure (the BATS suite only did this in harbor.bats, by hand).
Automatic cleanup — resources Chainsaw applied are deleted (and waited on) in reverse order; finally covers operator-created leftovers.
CI-ready output — a JUnit chainsaw-report.xml is emitted for the test summary.

Test anatomy

Each suite is a directory with one or more Test docs (in one of the layouts above, usually a chainsaw-test.yaml) plus the manifests it applies. A step has try (operations: apply, assert, error, script, delete, …), optional catch (diagnostics on failure), and optional finally (teardown regardless of outcome).

Declarative suites assert on status.conditions and concrete fields. Inherently imperative suites keep their logic in script steps:

openbao — bao operator init/unseal via kubectl exec.
kuberture — jq on Service annotations + external-dns split-horizon log assertions.
vminstance — a server-side dry-run eviction of the VM's launcher pod, checked against the VMI's LiveMigratable condition.
gateway — admission-rejection cases that need kubectl --as impersonation or error-message greps; the tenant apex is read from the namespace namespace.cozystack.io/host label at runtime so they pass on any host.
kubernetes-latest/previous — wrap _lib/run-kubernetes.sh (Kamaji bringup, port-forward, LB/NFS/ouroboros). The script cds to the repo root (the test dir is hack/e2e-chainsaw/<suite>, so repo root is ../../..).

Conventions / gotchas (Chainsaw v0.2.15)

Condition asserts use the filter-as-list form, not the documented [0] index:
```
status:
  (conditions[?type == 'Ready']):
  - status: "True"
```
The (conditions[?type == 'Ready'])[0]: {status: "True"} idiom fails with "field not found in the input object".
A reading of a failure diff: the "actual" side is a projection onto the expected fields — status: {} means the JMESPath expression failed, not that status is empty. Read the error line above the diff first.
error with a bare resource ref (name only) asserts the resource does not exist (used for "PVC reclaimed", "no backup schedule", "child has no Gateway HR").
script runs in the test directory with $NAMESPACE set to the config namespace (tenant-test).

Running locally

Requires a cluster with Cozystack installed and a tenant-test namespace (the environment hack/e2e-install-cozystack.bats produces). bucket and seaweedfs additionally need mc, nc, and jq on the host; kuberture/openbao/foundationdb/securitygroup/computeplane/kubernetes-latest/kubernetes-previous/vminstance need jq.

# install chainsaw
go install github.com/kyverno/chainsaw@latest   # or: brew install kyverno/chainsaw/chainsaw

# run all suites (config picked up from .chainsaw.yaml)
cd hack/e2e-chainsaw
chainsaw test

# run a single suite
chainsaw test postgres/

kubernetes-latest/kubernetes-previous each bring up a full tenant Kubernetes cluster (~25 min, real worker nodes) and vminstance boots a nested VM — expect a long run for the full set.

CI integration

The e2e job in .github/workflows/pull-requests.yaml runs make -C packages/core/testing test-chainsaw (which execs hack/e2e-chainsaw-run.sh inside the e2e sandbox) and uploads chainsaw-report.xml. The runner calls chainsaw test once per suite directory, fails a suite whose pods outlive its cleanup by more than the default timeouts.cleanup (LEAKED-PODS suite=<dir> in the log; see docs/agents/e2e-testing.md §8 and _lib/leak-allowlist.txt for known, filed leaks), and merges the per-suite reports into that one file. A local chainsaw test as shown above runs the same tests without the leak check. The sandbox image (packages/core/testing/images/e2e-sandbox/Dockerfile) ships the chainsaw binary alongside kubectl/helm/mc/jq/nc, built from source with a patch that keeps an assert polling through transient API errors such as an etcd leader change; a released chainsaw run locally fails the assert on the first such error instead.

6.9 KiB Raw Permalink Blame History