cozystack/docs/agents/e2e-testing.md

E2E Testing Conventions

Guidance for writing, changing, and reviewing Cozystack's end-to-end (E2E) tests and the CI that runs them. Read this before touching anything under hack/e2e-chainsaw/, the bootstrap/OpenAPI BATS (hack/e2e-*.bats), hack/*.sh test helpers, the E2E CI workflow (.github/workflows/pull-requests.yaml), or packages/core/testing/.

The app suite is Kyverno Chainsaw (hack/e2e-chainsaw/, one directory per app, each with a chainsaw-test.yaml). Cluster bootstrap (hack/e2e-install-cozystack.bats, hack/e2e-prepare-cluster.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.

The core principle

Retries do not recover flakes — they hide deterministic bugs and triple diagnostic wall-time. An audit of 30 PR runs found that across sampled failures, 25/25 retry attempts failed: the retry loop never once turned a red run green, it only delayed surfacing real bugs (Helm namespace conflicts, seaweedfs/harbor races, a vminstance disk race) that looked like flakes because the retry sometimes coincided with transient state clearing.

Every convention below follows from that finding: fail fast, fail loud, make the failure legible. A test that flakes is a test (or a product) with a real race to fix, not a test to wrap in a retry. Chainsaw's per-assertion polling is the structural expression of this: each assert waits on its own condition to its own timeout, and a failure is reported as a structured diff plus auto-captured events/describe/logs — not a generic non-zero exit.

Conventions

1. No retries on deterministic steps; retry only pure infrastructure

• Run E2E tests and Install Cozystack run once. On failure they print ❌ ... (no retry — see diagnostics below) and dump state. Do not reintroduce a retry loop around them. The E2E step is a single chainsaw test invocation — there is no per-app retry loop to bring back, because Chainsaw already polls each assertion to its own timeout.
• Prepare environment keeps its 3× retry — and only it — because it is pure infrastructure (Talos image download, sandbox VM boot, network) where a transient runner hiccup is a genuine flake with no application logic involved.
• Rule of thumb: retry is justified only for a step that contains no product or test logic.

2. Prefer declarative asserts over imperative waits

A Chainsaw assert polls until the resource matches its expected shape or the per-operation timeout fires, so it subsumes both "the resource exists" and "the field/condition reached this value" in one operation — the old until kubectl get …; do sleep 2; done existence backstop followed by kubectl wait --for=… is no longer needed.

- assert:
    timeout: 6m
    resource:
      apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: postgres-test
      status:
        (conditions[?type == 'Ready']):
        - status: "True"

• Condition checks use the filter-as-list form (conditions[?type == 'Ready']). Chainsaw v0.2.15 throws "field not found" on the indexed (...)[0] form.
• Imperative checks that no Kubernetes API condition models — S3 reachability through a port-forward, an external LB HTTP path, MetalLB advertisement — stay in a script step. A bare sleep N inside such a script is allowed only when nothing better models the wait; annotate it with a TODO(e2e-replace-fixed-timeouts): comment. The bootstrap BATS (hack/e2e-install-cozystack.bats, hack/e2e-chainsaw/_lib/run-kubernetes.sh) carries the sanctioned until kubectl get exceptions.

3. Let Chainsaw own cleanup; no test-level EXIT/RETURN traps

Trap-based cleanup was the single biggest source of false failures in the BATS suite: an EXIT trap ran cleanup in a context where shell variables were unset, set -u killed it, and a successful test was marked failed — then retried twice into real failures.

• Chainsaw deletes the resources it apply-ed during its cleanup phase (bounded by the delete/cleanup timeouts in hack/e2e-chainsaw/.chainsaw.yaml). Do not hand-roll teardown for resources Chainsaw created.
• A self-contained trap '… ' EXIT inside a single script step — to kill a port-forward or remove a temp dir — is fine, because it runs in a contained subprocess with its variables in scope. See hack/e2e-chainsaw/bucket/chainsaw-test.yaml. What is banned is test-level trap-based cleanup of the BATS kind.
• The ban extends to the unit BATS under hack/*.bats, for a second reason worth knowing before you debug one: an EXIT trap inside an @test body replaces the one the bats binary installs for its own bookkeeping, and a test that then fails prints no TAP line at all. It does not appear as not ok; it disappears, and the run ends with # bats warning: Executed N instead of expected M tests and a non-zero exit. Verified with Bats 1.14.0. Anyone reading the tail of the output, or grepping it for not ok, sees a green suite — and the CI runner hack/cozytest.sh, which is not the bats binary, reports the same failure correctly, so the two disagree exactly when it matters. Clean up at the end of the test body instead: both runners set -e, so the cleanup is unreachable on failure and the scratch directory is left behind for inspection, which is what you want from a failed test anyway. When a suite reports zero failures, confirm it also reports how many tests it ran: an exit code answers "did anything fail", never "did anything run". A guard test in hack/cozyreport.bats enforces this over a named list of files, which grows as the remaining ones are converted; a file not yet on that list may still carry a trap, so reconcile its 1..N plan against its ok count before believing a green-looking run.

4. Do not mask cleanup or teardown failures

• Resources a test creates that Chainsaw cannot reclaim — because a controller, not the test, owns them (e.g. the Velero Backup, BackupStorageLocation, and credentials secret in cozy-velero in the disabled hack/e2e-chainsaw/backup/ suite) — must be pruned explicitly. Do not || true over a stuck delete and leave stale state for the next run.
• For nested tenants, tear down child → parent with a hard wait for deletion between each. Deleting the parent while a child is still uninstalling wedges the parent's cleanup Job on the child namespace, and both stuck uninstalls occupy helm-controller workers past the end of the test — starving whichever suite runs next. The kubernetes suites encode this ordering in hack/e2e-chainsaw/_lib/run-kubernetes.sh.

5. The install gate must have teeth

The install test fails if any HelmRelease is not Ready. A toothless gate (backgrounded kubectl wait discarding child exit codes + a bare echo instead of exit 1) once shipped a permanently-failing platform HR through green CI for weeks.

• Use a single kubectl wait hr --all -A, then an outcome-based re-list (covers HRs created after the snapshot), and exit 1 on any non-Ready HR.
• Dump the full Ready-condition message per non-Ready HR so the real error is in the test output. See hack/e2e-install-cozystack.bats.

6. Assert the parent HelmRelease did not remediate — via status.history

A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is a silent race we want to catch. Do not check .status.installFailures / .status.upgradeFailures: Flux's ClearFailures() zeroes those on every successful reconcile, so checking them after the HR is Ready is vacuous and passes against a reverted fix.

• Inspect .status.history instead — a failed or uninstalled Snapshot survives a later successful reconcile. Use the shared helper in hack/e2e-chainsaw/_lib/remediation-guard.sh (helmrelease_has_remediation_cycle) from a script step.

7. Test-Impact Analysis (TIA), default-on

hack/select-e2e.sh walks the packages/core/platform/sources/*.yaml dependency graph and runs only the Chainsaw suites affected by a diff. A suite is a directory under hack/e2e-chainsaw/ containing a chainsaw-test.yaml.

• Conservative escalation: edits to packages/library/, packages/core/, api/, cmd/, internal/, top-level hack/*.sh|*.bats helpers, the shared hack/e2e-chainsaw/_lib/, the chainsaw config hack/e2e-chainsaw/.chainsaw.yaml, the Makefile, or the E2E workflows escalate to the full suite. A per-suite edit (any file under hack/e2e-chainsaw/<app>/) selects only that suite.
• The full-e2e PR label forces the whole suite.
• The selection is passed to make test-chainsaw CHAINSAW_SUITES="<names>", which runs chainsaw test <names> from hack/e2e-chainsaw/ (empty = the whole suite).
• The coverage gap TIA opens on PRs is closed at release-cut time: the "Prepare release" commit bakes image digests into packages/core/ (and other packages), which TIA escalates to the full suite — so the release PR exercises the whole suite against the commit the tag will point at.

When adding a new app package, confirm select-e2e.sh maps it correctly (it has a unit test, hack/select-e2e_test.bats). A new app whose suite directory does not yet exist will not be selected — add hack/e2e-chainsaw/<app>/chainsaw-test.yaml first.

What TIA does and does not do

Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs," and the wiring has consequences worth knowing before you rely on it or change it.

• It trims only which suites chainsaw test runs, not the expensive stages. Prepare environment and Install Cozystack (the full platform install) run regardless of the selection. TIA only decides the CHAINSAW_SUITES passed to the final E2E step. A perfect narrow saves the suite tail, not the bulk of wall-time. (Package image builds are scoped separately — by hack/build-matrix.sh, on the PR diff — while the Talos and installer/finalize legs still run for non-docs PRs; that is a distinct mechanism from TIA's suite trimming.)
• Select E2E tests runs after Install Cozystack and has no always(). GitHub implicitly ANDs a step's if: with success(), so if install fails the selector (and Run E2E tests) are skipped entirely. On a red install you will not see TIA narrow anything — that is the ordering, not a selector bug.
• Most PRs legitimately escalate to the full suite. Platform manifests live under packages/core/, Go under api//internal/, helpers under hack/, plus the Makefile — touching any one triggers the full suite. So "all suites selected" is usually correct, not a failure to narrow.
• Two skip layers, with a gap. The plan job computes a coarse docs-only gate (code, true for any change outside docs/**) that skips the whole build+E2E pipeline for docs-only PRs, plus a per-package build matrix (hack/build-matrix.sh) that scopes which images build. TIA's finer skip (select-e2e.sh ignoring docs/, dashboards/, and *.md) only trims the suite selection. A PR touching only dashboards/*.json therefore still runs install, then skips just the Chainsaw suites — near-zero savings. If you need such PRs to skip the pipeline, widen the plan docs-gate, not select-e2e.sh.

8. Failure must be self-explanatory — diagnostics are first-class

• On install failure, the CI step dumps kubectl get hr -A -o wide, a describe of each non-Ready HR, and sorted events, all under collapsible ::group:: blocks.
• On E2E failure, the CI step dumps kubectl get hr -A, sorted events, and COSI bucket/claim/access readiness columns (those CRDs ship no printer columns, so plain get shows only NAME/AGE). The chainsaw test run is one step, so this dump is at run level; Chainsaw's own per-Test catch blocks add scoped events/describe/podLogs for the failing suite.
• Inside a suite, attach scoped diagnostics with a top-level catch: block (events, describe, podLogs, get) rather than letting a bare assert failure through — see hack/e2e-chainsaw/bucket/chainsaw-test.yaml and hack/e2e-chainsaw/harbor/chainsaw-test.yaml. This replaces the BATS ... || { echo …; kubectl get/describe …; false; } idiom.
• make collect-report → hack/cozyreport.sh uploads a full cozyreport.tgz (operator/Flux/cert-manager logs, LINSTOR, Kamaji, Talos dmesg, COSI YAML) on every run, and the Chainsaw JUnit report (chainsaw-report.xml) is uploaded as its own artifact.
• The report's per-pod evidence, under kubernetes/pods/ in cozyreport.tgz, is selected by the READY column rather than by STATUS — so a pod left at 0/1 Running by a readiness probe that never passes is in scope, where before it was not. In scope is not the same as collected: the walk stops at COZY_REPORT_PODS_MAX pods or COZY_REPORT_PODS_BUDGET seconds, and what it did not reach is named in the artifact rather than dropped in silence, as the next point describes. Selecting on the STATUS column alone drops exactly that pod, and it is a common way for a suite to time out: the replica never joins, its HelmRelease sits in InProgress, and the artifact carries no trace of the pod the run died on. Terminal Succeeded/Completed pods stay excluded, since they legitimately end unready. Widening the selection and capping the walk are only safe together if the widening cannot displace what the uncapped walk always kept, so the collection order splits by evidence class before namespace: every pod that is not Running — a crash loop, Pending, Error, the class the collector took before this — is ordered ahead of every pod that is Running but unready, and COZY_REPORT_PREFER_NS orders within each of those two classes rather than across the whole selection. Otherwise a failed install with more not-Ready pods than the cap spends it on newly-in-scope tenant pods and drops the container logs of the platform pod that is actually crashing.
• Every bound that the report's pod collection hits is named in the artifact, not only in the job log. The pod walk stops at COZY_REPORT_PODS_MAX pods or COZY_REPORT_PODS_BUDGET seconds and leaves COLLECTION-TRUNCATED.txt naming whichever fired; a pod it cut off part way through leaves CONTAINER-LOGS-TRUNCATED.txt; a pod list that never came back leaves COLLECTION-FAILED.txt rather than the empty tree a healthy cluster produces; a missing local timeout leaves COLLECTION-UNBOUNDED.txt, because the reads still run, just without a ceiling, and a missing dependency on the machine writing the report is not a statement about the cluster; a kind this cluster does not serve at all — kubectl get vm -A on an install without KubeVirt, which exits non-zero — leaves KIND-NOT-SERVED.txt rather than a failure marker, since that is the cluster answering correctly and a marker that fires on healthy installs is one a reader learns to skip; and a knob value the collector could not use is named too, appended to COLLECTION-TRUNCATED.txt when there is one, written to COLLECTION-NOTES.txt when there is not, and appended to COLLECTION-FAILED.txt instead when the pod list itself never returned, so the note always sits beside the failure it accompanies rather than in a file the reader has no reason to open, and a bound that turns out to be the default is not mistaken for the bound that was asked for. A pod whose container list came back as a prefix — kubectl named some containers and then failed — leaves CONTAINER-LIST-INCOMPLETE.txt beside the logs it did manage to read, because the containers it never named have no file at all and their absence would otherwise read as containers that were silent. A read that succeeded while kubectl also wrote to stderr — a deprecation notice, a partial-result warning — leaves that message in READ-WARNINGS.txt beside the object rather than inside it, since pod.yaml and the logs are machine-read and a warning line in them costs the parser the file, while dropping it altogether would be the silent loss of evidence this tree exists to prevent. A pod directory whose containers produced any log at all also carries CONTAINER-LOGS-BOUND.txt, naming the --tail bound those logs were written under (a pod whose every container was silent has no bounded log to describe, so the file is absent), because --tail drops the oldest lines and nothing in a log says so; it sits beside the logs rather than inside them, since a log read in full is a complete artifact and the platform's controllers log JSON per line, so a trailing prose line would break every parser that could read the file before. A pod whose container list could not be read leaves logs-UNAVAILABLE.txt in its own directory, carrying kubectl's reason, because a pod directory holding pod.yaml and describe.txt and no logs at all otherwise reads as a pod whose containers were silent. The VM, VMI and services loops carry the same pair of bounds as constants rather than knobs, so a failed install that leaves every LoadBalancer without an address collects the first of them and names the rest instead of running the report step past the job's own limit, which would lose the tarball rather than truncate it. Whichever bound stopped a walk says so in COLLECTION-TRUNCATED.txt, count and clock alike, because objects removed by the clock while the count note stood alone would read as a cap that never fired. The read that feeds each loop is bounded too: the budget cannot start until rows arrive, so an apiserver hanging on the cluster-wide list would otherwise spend the whole collection step with the deadline never consulted. The budget is a deadline for starting a read rather than for finishing the walk, and it is rechecked between containers but not between a container's current and previous log, so the ceiling is the budget plus one pod's object reads plus both log reads of one container, which is the price of never leaving a half-written file behind.
• COZY_REPORT_POD_TAIL bounds each container log at 2000 lines by default, and a log that was read successfully records which bound it was written under, since --tail drops the oldest lines and a file that begins mid-stream looks exactly like a complete one; -1 restores kubectl's own default of the whole log. COZY_REPORT_PREFER_NS (default tenant-) is collected first, so a cap that only bites on a broadly degraded cluster is not spent on platform namespaces while the pod the suite died on falls off the end.
• Capture the PREVIOUS container instance, not just the current one. For a crash-looping pod kubectl logs (and the Chainsaw podLogs collector, which in v0.2.15 has no previous option — it does accept name/namespace/selector/container/tail/timeout/cluster/clusters) shows the Nth restart — a fresh replay of the same startup path — while the evidence that explains the failure sits in the immediately preceding instance, reachable only via --previous and lost for good once the kubelet garbage-collects that container. hack/e2e-capture-previous-logs.sh closes this once for every suite from the global error.catch (and from hack/cozytest.sh for the BATS tests) rather than per-suite: it discovers containers with restartCount > 0 at runtime — init containers included, since an interrupted bootstrap lives there — orders the app suites' tenant-test namespace first (a best-effort ordering only — it never restricts the capture, and for a suite whose pods live in a child tenant namespace it simply degrades to input order), and dumps each to the job log and to snapshots/<test>/previous-logs/. Containers that never restarted are skipped, so a healthy run emits one line, not a page of "previous terminated container not found". Every way the capture can come up short is named rather than inferred: the COZY_PREVLOG_MAX overflow, a malformed cap, a COZY_PREVLOG_TAIL that was rejected and fell back to 200 (a zero or leading-zero tail asks kubectl for nothing, which would otherwise be reported as the container having logged nothing), the tail bound each kept log was written under — named in capture-notes.txt beside the logs rather than inside them, since --tail drops the oldest lines and nothing in the output says so while a log read in full is a complete artifact that a trailing prose line would stop any parser from reading, and suppressed on -1 because no bound was applied then — a pod list that never returned (which is NOT the same as "nothing restarted" and must not be reported as such), a read that timed out versus one kubectl itself refused (kubelet GC, RBAC, an unreachable node — different problems that a blanket "kubelet GC" line would hide), a previous instance that produced no output, and a capture the outer backstop cut short. A partial log is kept and marked truncated, never deleted — it is the only copy of something already gone from the cluster. Each of those lines the script itself emits is written to snapshots/<test>/previous-logs/capture-notes.txt as well as to the job log, and the directory is created before the first of them, so a reader who has only the uploaded artifact can still tell a capture that found nothing from one that never ran. The outer backstop firing is the exception: that line comes from the callers, after the script has already been killed, and reaches the job log only.
• A catch script needs an explicit timeout: that exceeds the inner budgets it contains. A Chainsaw script op is bounded by timeouts.exec (5m in .chainsaw.yaml) unless it overrides it, so a catch whose own timeout -k … wrappers sum past that is structurally guaranteed to be killed — it reports SCRIPT | ERROR | signal: killed and yields nothing. That kill is strictly worse than an inner timeout firing: SIGKILL takes out the process group, so the collector dies mid-write and every later step in the script is skipped, whereas an inner timeout exits 124, leaves a partial capture on disk, and lets the rest of the catch run. Size the op timeout above the sum of the inner budgets and let the inner ones be what fires.
• Set crust-gather --duration explicitly. It is crust-gather's own budget for the collection phase and it defaults to 60s; on elapse it abandons the collection and skips its finish step, leaving a partial tree with nothing to say the rest is missing. The outer wall-clock must exceed it, because the API discovery crust-gather runs first is not covered by that budget at all — with an unhealthy aggregated APIService (the state a failed e2e is usually in) aggregated discovery times out and the fallback re-walks every group, which can consume the entire outer timeout before a single object is written. Do not diagnose a killed snapshot by bumping the outer number alone; the two budgets bound different phases. Send the collector's output to a log inside the snapshot rather than /dev/null so "complete or truncated?" is answerable from the artifact.
• Per-failed-test crust-gather snapshots are the richest diagnostic — a crust-gather serve-able archive of the cluster at the moment of failure, before cleanup. The global error.catch in hack/e2e-chainsaw/.chainsaw.yaml captures the host cluster into _out/cozyreport/snapshots/<test>/host on any failure (the Chainsaw analog of the hack/cozytest.sh EXIT-trap snapshot used by the BATS tests); the kubernetes suites additionally snapshot their tenant cluster from hack/e2e-chainsaw/_lib/run-kubernetes.sh while the tenant API LB is still routable. hack/cozyreport.sh folds snapshots/ into the artifact. Do not remove the global catch — without it the Chainsaw suite uploads a report with no per-test cluster state.

9. Keep the test environment deterministic

• Pre-pull every kubeovn / linstor / cert-manager image onto all nodes before those HelmReleases install (hack/e2e-prepull-images.sh), so clustered workloads (OVN raft, LINSTOR, the cert-manager webhook) do not fail on per-node image-pull stagger.
• Exclude loop devices from host LVM scanning in the Talos machine config so the host does not activate volume groups inside loop-mounted e2e disk images.
• Fail fast on node readiness (≈5m, then bail) rather than marching into LB/NFS tests that will also fail — it saves several minutes per attempt and keeps the real failure at the top of the log.

Chainsaw v0.2.15 gotchas

The suite is pinned to Chainsaw v0.2.15 (the latest release as of May 2026); none of the traps below are fixed upstream yet, so the workarounds stay until a bump is possible. Each one has already cost a debugging session — this is the "no one gets it right the first time" list, worth a read before writing a new assert.

• Condition asserts must use the filter-as-list form. (conditions[?type == 'Ready']) evaluates to a list; assert against it with a list body (- status: "True"). The indexed (conditions[?type == 'Ready'])[0] form throws field not found (also enforced by convention 2).
• No number literals / backticks in JMESPath. ports[?port == `443`] mis-evaluates in v0.2.15 — the numeric comparison silently returns the wrong set. Assert ports (and any numeric field) with the exact-ordered projection form instead: (ports[*].port): [80, 443]. sort() and contains(list, <number>) are likewise broken with numeric arguments.
• error: fails hard on an unknown GVK. Unlike the BATS ! kubectl get <kind> idiom (which passes when the type is absent), a Chainsaw error: op against a kind whose CRD is not installed errors the step rather than asserting absence. If a type may legitimately not exist (a version rename, an optional component), gate the check behind a positive precondition or do it in a script step — never a bare error:. This is what broke the etcd suite on the v1alpha1 → v1alpha2 rename.
• No conditional Test-level skip. There is no skip: <expr>; an env-var-gated test must exit 0 early inside a script step (see the ETCD_E2E_S3_ROUNDTRIP gate). A Test with no matching gate always runs.
• finally is step-level, not test-level. To guarantee teardown of something Chainsaw did not create, put the work and its own cleanup in a single script step; there is no test-wide finally block to hang it on.
• Cleanup is bounded and blocking — it surfaces latent teardown bugs. Chainsaw waits for the resources it applied to actually delete (within the delete/cleanup timeouts in hack/e2e-chainsaw/.chainsaw.yaml) and fails the test if they don't. This is deliberate — a stuck uninstall starves the next suite — but it means a teardown path the BATS suite masked with || true now surfaces as context deadline exceeded at cleanup. The failure is honest, not new; fix the teardown rather than widening the timeout (see cozystack/cozystack#3271, a pre-delete hook that hung on a nodeless tenant).
• Reading a failure diff: the "actual" side is a projection. Chainsaw projects the live object onto the expected fields, so status: {} in a diff means "the expression matched nothing / evaluated empty," not "status is literally empty." The real reason is the error line above the diff — read that first.
• A catch/cleanup script's cwd is the failing suite's directory (hack/e2e-chainsaw/<suite>/), so ../../.. is the repo root. The suites cd ../../.. and then source shared helpers by repo-root-relative path (hack/e2e-chainsaw/_lib/…) rather than juggling ../ hops.

Reviewer checklist for a new or changed E2E test

1. No new retry loop unless the step is pure infra (image pull / VM boot / network).
2. Resource readiness uses a Chainsaw assert (not an imperative until kubectl get …; kubectl wait); condition checks use the filter-as-list form (conditions[?type == 'Ready']).
3. Imperative-only waits live in a script step; any bare sleep carries a TODO(e2e-replace-fixed-timeouts): justification.
4. No test-level EXIT/RETURN trap — rely on Chainsaw cleanup; a self-contained trap is allowed only inside a single script step (port-forward / temp dir).
5. Controller-created artifacts the test cannot reclaim are pruned explicitly; nested tenants delete child → parent with a wait-for-deletion between.
6. Standard HR-Ready assert timeout is 5–6m; longer waits (harbor 10m, NFS 10m, VM image pulls, platform-wide install 15m) are justified in-line.
7. Failure path attaches scoped diagnostics via a catch: block, never a silent pass.
8. If it touches parent-HR behavior, add the status.history remediation guard (hack/e2e-chainsaw/_lib/remediation-guard.sh).

In-flight direction (not yet the merged standard)

These are being explored on branches and may become conventions; do not assume they are the current main behavior:

• Cilium orphaned-endpoint self-heal — an interim CI watchdog that evicts a single confirmed-orphan Cilium endpoint ("IP already in use", cilium/cilium#38313). Explicitly a mitigation to remove once a fixed Cilium ships; it refuses to touch an endpoint backing a live pod so real duplicate-IP bugs stay visible.
• Cluster state snapshot/restore between test groups instead of reinstalling.