Aleksei Sviridkin
ad62a8d62b
The tenant node-join wait no longer discriminates. Nested virtualisation on the shared runners degrades far enough that a guest which registers in minutes on real hardware can miss any deadline this test can afford, so a red on that one wait says as much about the machine the run landed on as about the product, and it teaches a reader to re-run the job. The suite still fails on it. That is what keeps Chainsaw's catch running: the previous-instance container logs, the host snapshot, the data-plane capture and the event dump are all keyed on the test failing, and the report stays honest. What changes is one layer up. The wait writes a marker beside its diagnostics, and a new lane-side gate reads that marker to decide whether the job blocks. It opens only when every failed test in the run carries one, and blocks on anything it cannot attribute: a report it could not read, a failed test the report does not name, a failure with no marker. Only the lanes that block a merge ask it. The two pull-request lanes do; nightly and the tag lane run the same suites and stay hard. A soft pass on the tag lane would ship a release candidate whose tenant clusters brought up no worker nodes, and nightly is the only place an e2e failure reaches anyone who did not open a pull request, since no lane in this tree notifies anywhere. A softened red finishes the job green, so the job's result stops answering whether the suite passed, and both kinds of consumer are handed the decision instead of inferring it. A step gated on the job failing would silently stop running: the SSH breakpoint is that case, and it still opens for a pull request marked for debugging. A job that reports the result would silently start reporting a pass: the required commit status now says the deadline was missed and that the merge is not blocked on it, rather than that the suite passed. The deadline goes to 29m. That is the test's own budget and nothing more: its clock starts after the machinedeployment wait, the LoadBalancer assignment and the healthz probe, while the chart's autoscaler and health-check timers have been running since the Machine was created, so no ordering against them is claimed. The chart's 20m and 30m give a scale to pick against, and a budget over the first leaves room for a remediation cycle without promising that one fits. The operation ceiling both suites give the script rises with the deadline, and the diagnostics phase budget rises inside it, because the collectors gated ahead of the serial console cost more between them than the old budget covered. Both budget guards read their terms from the sources that set them. The e2e job cap rises with them, and its guard holds what a guard can hold: the pair's operation and catch ceilings, and one figure across every lane that runs the suite. It also records that the suites' full summed ceilings have never fit any cap in that file, so the margin above the bound is taken from a measured run instead. The prose that restates the cap elsewhere moves with it, and the sweep that holds the deadline's restatements now reads the figure spelled in words as well as in minutes. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Chainsaw E2E tests
The Cozystack E2E app suite, written for Kyverno Chainsaw — a declarative, Kubernetes-native E2E test framework (CNCF, part of the Kyverno project; successor to KUTTL).
This directory replaces the per-app BATS suite that used to live in hack/e2e-apps/*.bats. The CI e2e job runs chainsaw test hack/e2e-chainsaw/ after make install-cozystack; cluster bootstrap (hack/e2e-prepare-cluster.bats, hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.
Layout
hack/e2e-chainsaw/
├── .chainsaw.yaml # global Configuration (namespace, timeouts, JUnit report)
├── _lib/ # shell reused by the script-heavy suites
│ ├── run-kubernetes.sh # tenant Kubernetes bringup + LB/NFS/ouroboros checks
│ └── remediation-guard.sh
├── <app>/
│ ├── chainsaw-test.yaml # one or more Test definitions
│ └── <app>.yaml # the manifests the test applies
└── ...
Suites: postgres, bucket, mariadb, mongodb, redis, qdrant, clickhouse, kafka, etcd, openbao, harbor, foundationdb, external-dns, kuberture, vminstance, gateway, kubernetes-latest, kubernetes-previous, kubernetes-oidc-system, kubernetes-oidc-customconfig, securitygroup.
What Chainsaw buys over the BATS suite
- Readable tests — a list of resources and expected states, not shell control flow. The
timeout N sh -ec "until kubectl get ..."+kubectl waitpair that appeared ~100 times collapses into a singleassertthat polls existence and state together. - Distinguishable failure modes — "object never created", "created but never ready", and "wrong status" all surface as a structured diff instead of an opaque shell timeout.
- Automatic diagnostics —
catchblocks dump events,describe, and pod logs on any failure (the BATS suite only did this inharbor.bats, by hand). - Automatic cleanup — resources Chainsaw applied are deleted (and waited on) in reverse order;
finallycovers operator-created leftovers. - CI-ready output — a JUnit
chainsaw-report.xmlis emitted for the test summary.
Test anatomy
Each suite is a directory with a chainsaw-test.yaml (one or more Test docs) plus the manifests it applies. A step has try (operations: apply, assert, error, script, delete, …), optional catch (diagnostics on failure), and optional finally (teardown regardless of outcome).
Declarative suites assert on status.conditions and concrete fields. Inherently imperative suites keep their logic in script steps:
- openbao —
bao operator init/unsealviakubectl exec. - kuberture —
jqon Service annotations + external-dns split-horizon log assertions. - gateway — admission-rejection cases that need
kubectl --asimpersonation or error-message greps; the tenant apex is read from the namespacenamespace.cozystack.io/hostlabel at runtime so they pass on any host. - kubernetes-latest/previous — wrap
_lib/run-kubernetes.sh(Kamaji bringup, port-forward, LB/NFS/ouroboros). The scriptcds to the repo root (the test dir ishack/e2e-chainsaw/<suite>, so repo root is../../..).
Conventions / gotchas (Chainsaw v0.2.15)
- Condition asserts use the filter-as-list form, not the documented
[0]index:Thestatus: (conditions[?type == 'Ready']): - status: "True"(conditions[?type == 'Ready'])[0]: {status: "True"}idiom fails with "field not found in the input object". - A reading of a failure diff: the "actual" side is a projection onto the expected fields —
status: {}means the JMESPath expression failed, not that status is empty. Read the error line above the diff first. errorwith a bare resource ref (name only) asserts the resource does not exist (used for "PVC reclaimed", "no backup schedule", "child has no Gateway HR").scriptruns in the test directory with$NAMESPACEset to the config namespace (tenant-test).
Running locally
Requires a cluster with Cozystack installed and a tenant-test namespace (the environment hack/e2e-install-cozystack.bats produces). bucket additionally needs mc and nc on the host; kuberture/openbao/harbor need jq.
# install chainsaw
go install github.com/kyverno/chainsaw@latest # or: brew install kyverno/chainsaw/chainsaw
# run all suites (config picked up from .chainsaw.yaml)
cd hack/e2e-chainsaw
chainsaw test
# run a single suite
chainsaw test postgres/
kubernetes-latest/kubernetes-previous each bring up a full tenant Kubernetes cluster (~25 min, real worker nodes) and vminstance boots a nested VM — expect a long run for the full set.
CI integration
The e2e job in .github/workflows/pull-requests.yaml runs make -C packages/core/testing test-chainsaw (which execs chainsaw test hack/e2e-chainsaw/ inside the e2e sandbox) and uploads chainsaw-report.xml. The sandbox image (packages/core/testing/images/e2e-sandbox/Dockerfile) ships the chainsaw binary alongside kubectl/helm/mc/jq/nc.