cozystack/docs/agents/e2e-testing.md

E2E Testing Conventions

Guidance for writing, changing, and reviewing Cozystack's end-to-end (E2E) tests and the CI that runs them. Read this before touching anything under hack/e2e-chainsaw/, the bootstrap/OpenAPI BATS (hack/e2e-*.bats), hack/*.sh test helpers, either E2E CI workflow (.github/workflows/pull-requests.yaml for same-repo pull requests, .github/workflows/e2e-fork.yaml for forks), or packages/core/testing/. §10 covers how the two split the work and which invariants hold them together.

The app suite is Kyverno Chainsaw (hack/e2e-chainsaw/, one directory per app, each with a chainsaw-test.yaml; §7 lists the other layouts Chainsaw accepts). Cluster bootstrap (hack/e2e-install-cozystack.bats, hack/e2e-prepare-cluster.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.

The core principle

Retries do not recover flakes — they hide deterministic bugs and triple diagnostic wall-time. An audit of 30 PR runs found that across sampled failures, 25/25 retry attempts failed: the retry loop never once turned a red run green, it only delayed surfacing real bugs (Helm namespace conflicts, seaweedfs/harbor races, a vminstance disk race) that looked like flakes because the retry sometimes coincided with transient state clearing.

Every convention below follows from that finding: fail fast, fail loud, make the failure legible. A test that flakes is a test (or a product) with a real race to fix, not a test to wrap in a retry. Chainsaw's per-assertion polling is the structural expression of this: each assert waits on its own condition to its own timeout, and a failure is reported as a structured diff plus auto-captured events/describe/logs — not a generic non-zero exit.

The Proxmox lane

hack/e2e-chainsaw/kubernetes-proxmox/ covers the Proxmox substrate and is the one suite in this tree that the normal lanes cannot run. It needs a Proxmox hypervisor, and the oracle-vm-* runners have none, so it ships as chainsaw-test.yaml.disabled: neither Chainsaw nor the selector counts a .disabled file as a test, and nothing else in the directory or below it is one, which registers it nowhere and makes it unselectable rather than merely unselected. .github/workflows/e2e-proxmox.yaml drops the suffix on a self-hosted runner that has one.

Two properties of that workflow are security boundaries rather than preferences. It is never triggered by pull_request, because a self-hosted runner executes what the workflow checks out and this one stages a hypervisor token and a live kubeconfig — a fork PR trigger would hand both to its author. And its concurrency group never cancels in progress, because a killed run leaks VMs on the hypervisor rather than merely losing a result.

It does not install Cozystack. Every other lane builds a sandbox per run; this one asserts against an existing cluster and an existing hypervisor, which is a weaker guarantee and is stated in the suite header rather than left to be discovered.

The imperative half lives in _lib/run-proxmox.sh and is deliberately not a port of _lib/run-kubernetes.sh: that script's diagnostics are virt-launcher Pods, VirtualMachineInstances and guest consoles, none of which exist on an external hypervisor.

Conventions

1. No retries on deterministic steps; retry only pure infrastructure

• Run E2E tests and Install Cozystack run once. On failure they print a ❌ Install failed (no retry …) or ❌ Chainsaw E2E failed (…) line and dump state. Do not reintroduce a retry loop around them. The E2E step runs each suite once through hack/e2e-chainsaw-run.sh (one chainsaw test per suite directory, for the leak guard in §8) — there is no per-app retry loop to bring back, because Chainsaw already polls each assertion to its own timeout.
• Prepare environment keeps its 3× retry — and only it — because it is pure infrastructure (image pull, node container start, network) where a transient runner hiccup is a genuine flake with no application logic involved. The retry tears the compose project down between attempts, because a half-started project the next attempt adopts is not a fresh try.
• Rule of thumb: retry is justified only for a step that contains no product or test logic.

The node-join deadline is a hard failure

The tenant node-join wait in hack/e2e-chainsaw/_lib/run-kubernetes.sh fails the suite and blocks the lane, like any other assertion. It has not always. While the management nodes were QEMU guests, nested virtualisation on the shared runners degraded far enough that a worker registering in minutes elsewhere could miss any deadline this test could afford, and the lane carried a carve-out that let that one failure through without blocking a merge.

cozystack/cozystack#3513 settled what that degradation actually was — the host's kvm_amd vgif flag, a property of the runner nobody here controls — and moving the management nodes to containers removes the nesting level it acted on. Measured across three CI runs on the same commits, the two tenant-Kubernetes suites take 466-809s on containers and pass in both the vgif-present and vgif-absent strata, against 1143-2040s on QEMU where the vgif-absent runs missed the deadline outright. The deadline stopped being the marginal thing the carve-out existed for, so the carve-out went with the substrate: hack/e2e-node-join-soft-red.sh, the SOFT-RED-node-join.txt marker and the soft_red job output no longer exist, and every lane treats a missed deadline as a red.

What survives is the annotation. cozy_note_node_join_deadline emits a ::warning title=node-join:: so the reason a run went red is legible from the run summary without opening the log, and cozy_node_join_deadline_expired still separates timeout firing (124) from the wait failing to run at all (125, 126, 127, 128+n), because those two send a reader to different places. Neither decides whether the job blocks; nothing does any more.

The node-join deadline itself sits at 29m, and that figure is this test's own budget rather than a position in a sequence of product timers. Its clock starts where the wait starts — after the machinedeployment wait, the LoadBalancer assignment and the healthz probe, each with a ceiling of its own — while the chart's timers are already running by then: the autoscaler's maxNodeProvisionTime from the Machine's creation, the MachineHealthCheck's nodeStartupTimeout from the later of creation and InfrastructureReady, restarting at that transition. The offset between them is not fixed and no ordering against them is claimed. What the chart's 20m and 30m give is a scale to pick against: a budget wider than the first leaves room for a remediation cycle to fit inside the window, which is a possibility rather than a guarantee — a run can spend the whole window on one worker that never comes back. The price of that width is that a worker replaced mid-window takes its guest serial console with it; the replacement's console and the Machine history in the host snapshot survive.

2. Prefer declarative asserts over imperative waits

A Chainsaw assert polls until the resource matches its expected shape or the per-operation timeout fires, so it subsumes both "the resource exists" and "the field/condition reached this value" in one operation — the old until kubectl get …; do sleep 2; done existence backstop followed by kubectl wait --for=… is no longer needed.

- assert:
    timeout: 11m
    resource:
      apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: example-test
      status:
        (conditions[?type == 'Ready']):
        - status: "True"

• Condition checks use the filter-as-list form (conditions[?type == 'Ready']). Chainsaw v0.2.15 throws "field not found" on the indexed (...)[0] form.
• The timeout above is what this block demonstrates least: it clears the install-timeout floor a release with no override carries, which is where any HR-Ready number has to start. Copy the condition form from here and take the budget from Sizing an HR-Ready budget.
• Imperative checks that no Kubernetes API condition models — S3 reachability through a port-forward, an external LB HTTP path, MetalLB advertisement — stay in a script step. A bare sleep N inside such a script is allowed only when nothing better models the wait; annotate it with a TODO(e2e-replace-fixed-timeouts): comment. The bootstrap BATS (hack/e2e-install-cozystack.bats, hack/e2e-chainsaw/_lib/run-kubernetes.sh) carries the sanctioned until kubectl get exceptions.
• A measurement interval is not one of those waits and does not take the marker. The rule above governs a sleep standing in for a condition nobody wrote a wait for, and the marker means "replace this once something models it". A gap between two readings of a cumulative counter models nothing and is the quantity itself: the diagnostics phase in hack/e2e-chainsaw/_lib/run-kubernetes.sh reads the tenant worker CPU counters, the sandbox nodes' /proc/stat and QEMU's per-thread /proc/<pid>/task/<tid>/stat on either side of one shared COZY_DIAG_RATE_INTERVAL wait, because a single reading of a counter that accumulates from container start is an average over the whole uptime and no rate at all. There is no event to wait for instead, so a TODO there would promise a replacement that cannot exist. Carve-outs are named here rather than judged case by case, and this is the only measurement-interval one.

3. Let Chainsaw own cleanup; no test-level EXIT/RETURN traps

Trap-based cleanup was the single biggest source of false failures in the BATS suite: an EXIT trap ran cleanup in a context where shell variables were unset, set -u killed it, and a successful test was marked failed — then retried twice into real failures.

• Chainsaw deletes the resources it apply-ed during its cleanup phase (bounded by the delete/cleanup timeouts in hack/e2e-chainsaw/.chainsaw.yaml). Do not hand-roll teardown for resources Chainsaw created.
• A self-contained trap '… ' EXIT inside a single script step — to kill a port-forward or remove a temp dir — is fine, because it runs in a contained subprocess with its variables in scope. See hack/e2e-chainsaw/bucket/chainsaw-test.yaml. What is banned is test-level trap-based cleanup of the BATS kind. The same carve-out holds inside a BATS @test when the trap sits in an explicit subshell — ( … trap "kill $pid" EXIT … ) — because a subshell trap does not replace the one the bats binary installs, so a failure inside it still prints its not ok. hack/e2e-test-openapi.bats relies on this to kill a backgrounded kubectl proxy; moving that cleanup to the end of the body would leak a process holding a fixed port rather than fix anything.
• The ban extends to every BATS file under hack/, subdirectories and the e2e- prefixed ones included, for a second reason worth knowing before you debug one: an EXIT trap inside an @test body replaces the one the bats binary installs for its own bookkeeping, and a test that then fails prints no TAP line at all. It does not appear as not ok; it disappears, and the run ends with # bats warning: Executed N instead of expected M tests and a non-zero exit. Verified with Bats 1.14.0. Anyone reading the tail of the output, or grepping it for not ok, sees a green suite. The former unit CI runner hack/cozytest.sh reported that failure correctly; the unit lane now runs Bats, which is why test-level handlers had to be removed before the runner switch. Clean up at the last reachable point after an assertion whose failure aborts the test body; the scratch directory then survives for inspection on failure, which is what a failed test wants anyway. A failure in an errexit-exempt if or AND-OR context can continue into cleanup and let cleanup supply the test status, so preserve the verdict explicitly in that shape. When a suite reports zero failures, confirm it also reports how many tests it ran: an exit code answers "did anything fail", never "did anything run". hack/bats-no-exit-trap.bats enforces this across every hack/**/*.bats, subdirectories included: a file that carries no # EXIT-TRAP DEBT: N comment must contain no EXIT-trap line at all, and a file that carries one must install exactly N — so a trap appearing or disappearing fails until the file's own number is corrected. Note what that does and does not buy: it is a ratchet on the number's accuracy, not on the debt itself, because adding a trap and raising N in the same change is green. Nothing mechanical stops the count growing — review does, which is the point of the number living in the file being reviewed. A trap inside an explicit subshell is exempt from the ban but still counted, so a declaration is not by itself an admission of debt; hack/e2e-test-openapi.bats and hack/capk-provider_test.bats document their exempt subshell traps in their headers. No file declares conversion debt after the test-level handlers are removed; the remaining declarations count those exempt traps. The scan reads .bats files only, so a handler reaching a test body from a sourced .sh is outside it — hack/e2e-chainsaw/_lib/run-kubernetes.sh installs two, each benign for its own reason rather than by design: the one in cozy_capture_tenant_talos because that function is declared with ( and runs in a subshell, and the one in run_kubernetes_test because no @test calls it despite being declared with {. Several hack/*.bats files source that library. Treat the guard as a ratchet over a common spelling, not as proof that a file installs no handler. Whenever a suite reports zero failures, reconcile its 1..N plan against its ok count before believing the run.

4. Do not mask cleanup or teardown failures

• Resources a test creates that Chainsaw cannot reclaim — because a controller, not the test, owns them (e.g. the Velero Backup/Restore in cozy-velero, which the vminstance backup round-trips prune by deleting the cozystack Backup and RestoreJob CRs, whose finalizers then clean up the Velero objects asynchronously — the BackupJob has no finalizer and its Backup carries no owner reference, so deleting a BackupJob reclaims nothing — plus an explicit delete of the throwaway BackupStorageLocation, sequenced after the Velero Backup is gone so it does not race the finalizer) — must be pruned explicitly. Do not || true over a stuck delete and leave stale state for the next run.
• For nested tenants, tear down child → parent with a hard wait for deletion between each. Deleting the parent while a child is still uninstalling wedges the parent's cleanup Job on the child namespace, and both stuck uninstalls occupy helm-controller workers past the end of the test — starving whichever suite runs next. The kubernetes suites encode this ordering in hack/e2e-chainsaw/_lib/run-kubernetes.sh.
• A test that creates a tenant Kubernetes cluster waits for that release to become Ready before the assertions it ends on — not before every read of the cluster, which is a different rule and a false one. The chart's post-install hook (packages/apps/kubernetes/templates/oidc-rbac-job.yaml) polls for the Kamaji-issued admin kubeconfig in the modes that have one, and that Secret is what the test needs to reach the tenant API, so bringing up a worker or waiting on a child release happens while the parent is still installing. Reading mid-install is fine. What the rule is about is the COMPLETION path: a run that asserts, passes and ends while the parent is still installing turns a green result into a red one in teardown. A failure path that diagnoses and exits mid-install does the same to the install, but that run is already red, so the second red costs noise rather than a wrong verdict — and the script has a number of such exits between the CR and the wait, which the guard does not and should not treat as violations.
• A Helm release cannot be uninstalled while its install action is running, and for this kind the action can outlive the fast assertions — by how much depends on the mode, per the bullet below: release.cozystack.io/helm-install-disable-wait on the ApplicationDefinition drops the wait on the main resources, but the chart's blocking post-install hook still holds the install. A test that asserts and ends therefore hands the remainder of the install to teardown and is reported red on a step that tests nothing.
• Size the wait from the install path rather than from a CI budget, and note that the floor depends on where the wait sits. Placed straight after the CR is applied it has to cover the hook's own kubeconfig deadline, 600s — but only in the modes that have one: the poll sits behind an early exit for spec.oidc.mode: None, so a suite running the default mode has no such floor at all; placed later, after the run has already read and used that kubeconfig, the deadline is spent and only the tail remains — the hook's work after the Secret appears plus helm-controller flipping the parent Ready. The ceiling does not move either, though not because the install stops: the kind is rendered with Install.Strategy RetryOnFailure rather than Remediation{Retries: -1}, so helm-controller does try again after the 20m helm-install-timeout. What makes a longer wait pointless is the guard that follows it in the same script: a timed-out attempt leaves a failed snapshot in .status.history — Install.Remediation is unset, so nothing uninstalls the release and the uninstalled status never appears — and the guard fails on that snapshot, so a success arriving from a later attempt is one the suite has already failed on.
• hack/run-kubernetes-install-wait_test.bats enforces this on the completion path: the CR is applied, the parent release is waited for, and the calls whose names follow the cozy_assert_* / cozy_switch_and_assert_* convention come after that wait. It reads the budget for parse and sign, and for one ceiling far above anything this suite waits for; it never reads it for the floor, which is not derivable from the script text and stays with review. Five costs are worth knowing before writing such a test: the wait and the assertions it guards must sit in one function body, because source position is execution order only within one body; the wait must be unconditional in that body, not under an if, a loop or a case; nothing in that body may switch errexit off above it; no line in it may put a } at column 0, which a multi-line JSON heredoc does unless its braces are indented — the body is delimited by that character, so the guard reads the body as ending there and reports the wait as being at file scope; and no line may close a brace in command position either, which a { ...; } group written across lines does — a brace standing as a command means the body closed before the extraction did, so the guard refuses the range rather than sourcing what follows it. The last two fail differently and say so differently, and indenting fixes the column-0 one and not the other. Which spellings each matcher accepts, which rewrites it refuses rather than judges, and the blind spots it records are all in the guard's own header, beside the code they govern.

5. The install gate must have teeth

The install test fails if any HelmRelease is not Ready. A toothless gate (backgrounded kubectl wait discarding child exit codes + a bare echo instead of exit 1) once shipped a permanently-failing platform HR through green CI for weeks.

• Use a single kubectl wait hr --all -A, then an outcome-based re-list (covers HRs created after the snapshot), and exit 1 on any non-Ready HR.
• Dump the full Ready-condition message per non-Ready HR so the real error is in the test output. See hack/e2e-install-cozystack.bats.

6. Assert no HelmRelease failed an action it retries in place — via status.history

A HelmRelease that hit its wait timeout and failed an action is a silent race we want to catch, and it stays one when a later retry recovers the release. Do not check .status.installFailures / .status.upgradeFailures: Flux's ClearFailures() zeroes those on every successful reconcile, so checking them after the HR is Ready is vacuous and passes against a reverted fix.

• Inspect .status.history instead — a failed or uninstalled Snapshot survives a later successful reconcile, which is what makes it readable at all after the release is Ready. It does not survive an unbounded number of them: the controller truncates history on every in-sync reconcile and again before it retries or remediates a failed release, retaining the newest five Snapshots on a release whose strategy is a retry and, on any other, those down to the previous deployed or superseded Snapshot — falling back to the same newest-five cap when there is no such Snapshot to cut at. An e2e run takes only a handful of actions on each release, so the footprint is still present when the guard reads it — but a guard placed after further release actions, or one asserting on history days later, is reading a window and needs to say so. Use the shared helper in hack/e2e-chainsaw/_lib/remediation-guard.sh (helmrelease_has_remediation_cycle) from a script step. Take the jsonpath from HELMRELEASE_HISTORY_JSONPATH in that same file rather than writing it out: internal/fluxcontract sources that file and runs the resulting value against the upstream Flux type, and a literal copied elsewhere is not covered by it.
• Fail on any footprint of a release that retries in place; report it on the others. Under RetryOnFailure helm-controller never remediates: a failed action is retried as an upgrade with its manifests left in place, so the release never writes an uninstalled Snapshot of its own, and the failed one is the only record of the attempt. cozy_guard_helmrelease therefore fails the run on either status on a release whose .spec.install.strategy.name is RetryOnFailure: the parent, which the aggregated apiserver stamps that way like every Application HelmRelease (pkg/registry/apps/application/rest.go), and the tenant cilium and csi, whose templates set it. A retry that recovered still fails the run, by design, while the Snapshot still reads failed: the failed attempt happened, and on these releases nothing else records it. It does not after a failed install that the very next attempt recovers. With no deployed release to upgrade from, Helm upgrades from the failed one and marks it superseded on success, helm-controller rewrites that Snapshot in place, and the history reads deployed, superseded like any ordinary upgrade, so the guard passes. A failed upgrade keeps its failed Snapshot, since Helm supersedes the last deployed release instead, and so does an install that fails twice before recovering. That makes a late node join a red run here even when it lands inside the 29m node-join deadline: the tenant cilium install cannot complete before a worker registers, because its operator Deployment has nowhere to schedule, so workers arriving more than two of its 10m install timeouts after the install started leave a failed Snapshot, and the guard fails the run on it. This is accepted rather than carved out, since the chart itself treats an install outrunning its budget for want of a node as a problem, but the cause is the join and not the CNI. The guard's message says so and points at the Snapshot times in the describe output it prints. The other addon releases pair the default strategy with retries: -1, a configuration that declares repeated remediation their recovery path. Nothing in the suite measures how often they take it, and failing a whole bringup on a release doing what it is configured to do is how a guard ends up switched off, so the same footprint there is printed as a note and the run stays green. For the CRD addons that configuration is itself a defect, tracked in cozystack/cozystack#3555; tighten when a measured baseline or a strategy change makes the condition knowable.
• The strategy is read through a jsonpath, so something has to notice when it stops resolving. kubectl -o jsonpath prints nothing and exits 0 for a path that no longer matches, and empty takes the lenient branch — a rename of spec.install.strategy in a future helm-controller would report every footprint on the parent, cilium and csi as a note, on a green run, with the unit tests still passing because they feed the value from a fixture rather than through the jsonpath. An empty value is not by itself evidence of that, since most addons legitimately set no strategy, so the check lives on the one release whose strategy the cluster is known to set: the parent, where an empty value, or any value other than RetryOnFailure, fails the run. That is enough, because the same rename empties the field on every release at once.
• "Covers the tenant addon HelmReleases" means reads and classifies, not can fail the run on. Five conditions red a run: a failed or uninstalled Snapshot on a release that retries in place; a Ready HelmRelease reporting no history at all, which is a status shape the helper can no longer read; a read that failed, which is never folded into "no history"; an empty or non-RetryOnFailure .spec.install.strategy.name on the parent; and an addon prefix that selects no HelmRelease at all, which the passing path fails on rather than passing silently over nothing. Every other addon footprint is a note. Which releases a run has at all depends on what it enabled, since most addon templates are gated on their own addons.<name>.enabled, so the breadth of the reading and the breadth of the gate are not the same number.
• The guard runs after the assertions, so the failure paths need the reading without the verdict. Every deadline a failed child action surfaces as — a node that never turns Ready, a HelmRelease wait that expires, a PVC that never binds — ends the script under errexit well before the guard, so on exactly the runs such an action explains, the guard never executes and the history is named nowhere. cozy_report_helmrelease_remediation is the same classification with no verdict, called from the node-join collector and from the exit handler that covers every other failure. Arm that handler where the object it reads starts existing, not where the tenant becomes reachable: the parent HelmRelease is rendered from the CR at the apply, and everything between there and the tenant LB is a management-cluster wait on that release or on what it installs, so a handler armed after those waits misses the window in which an install action times out. It fails nothing: the run is already red, and the tenant crust-gather snapshot hangs off the caller's own exit status. It walks the releases under both the caller's diagnostics phase budget and one of its own, and names whatever it stopped short of. Both budgets decide whether the next read may start, not when the walk ends, so neither is a worst case: the ceiling is the budget plus the reads already committed to when it expires. Write it that way in any collector you add — a budget described as a total is a number a later reader will trust for sizing the step around it.
• Do not infer a child release's state from the parent's. The Kubernetes ApplicationDefinition sets release.cozystack.io/helm-install-disable-wait, which the apiserver turns into DisableWait on both actions, so the parent reaches Ready without waiting for the manifests it applied, the child HelmReleases included (Helm still waits on its post-install hook Job) — and the suite waits on some addons by name while never naming others. An empty .status.history is therefore judged against that release's own Ready condition: Ready with no history is a status shape the helper can no longer read and fails the run, not-Ready with no history is a release that never completed an action and is reported as not inspected. Read readiness before the history, not after, or a release that finishes installing between the two reads presents as Ready with an empty history and fails the run for a shape it never had. That ordering closes the truncation half of the window and not all of it: HelmReleaseStatus.ClearHistory() empties history outright, and the controller that calls it is not a module this tree carries.

7. Test-Impact Analysis (TIA), default-on

hack/select-e2e.sh walks the packages/core/platform/sources/*.yaml dependency graph and runs only the Chainsaw suites affected by a diff. A suite is a directory directly under hack/e2e-chainsaw/ in or below which Chainsaw would find a Test: a chainsaw-test.yaml, or a chainsaw-test.yml, which Chainsaw loads when the .yaml is absent, or, with neither, numbered step files such as 01-install.yaml, which Chainsaw turns into a Test on its own. Chainsaw walks every subdirectory of the directory it is handed, so a Test in a top-level directory or anywhere below it makes that directory a suite. Every one of those layouts runs, so the selector has to see it or it is never selected.

• Conservative escalation: edits to packages/library/, packages/core/, the Go trees (api/, cmd/, internal/, pkg/), the codegen under tools/, go.mod/go.sum, top-level hack/*.sh helpers, the hack/e2e-*.bats the sandbox runs and the hack/lib/ they source, hack/*.mk, hack/buildkitd.toml, hack/e2e-*.yaml, the shared hack/e2e-chainsaw/_lib/, examples/backups/_lib/ (the place for helpers shared by the backup walkthroughs) and any file directly in examples/backups/, the chainsaw config hack/e2e-chainsaw/.chainsaw.yaml, the Makefile, or any workflow that runs the suite escalate to the full suite. Only pull-requests.yaml is executed from the pull request's own head, so escalating on the other three buys generic regression coverage rather than coverage of the edit; the trade and its measured cost are written next to full_suite_pattern rather than repeated here. A per-suite edit (any file under hack/e2e-chainsaw/<app>/) selects only that suite, with one exception: a *.disabled file selects nothing, because a parked suite is executed by nothing and editing it cannot regress a test. The directory name is membership-tested against the suite list on the spot, so a directory holding no suite — shared material beside _lib/ — escalates there and then, naming the file. hack/e2e-apps/<name>.bats — the pre-Chainsaw per-app BATS suites — maps the same way, off the basename instead of the directory. The files formerly kept there were wired to nothing after migration, and the path remains deliberately mapped rather than marked inert: inert would bake the orphan status into the rule and go quietly wrong the day a lane runs those files again. The basename is membership-tested the same way. Deferring either test to the final intersection instead would leave the verdict to the rest of the diff — alone the selection empties and the backstop escalates, but beside any path that contributed a suite the escalation disappears and the run narrows to that suite in silence, which is the merge-before-escalate shape #3330 removed from the graph walk. Anything else under hack/e2e-apps/ — a nested path, a non-.bats file — is shared material for those suites the way hack/e2e-chainsaw/_lib/ is for the Chainsaw ones, and escalates for the same reason.
• Every path is classified, and an unclassified one escalates. The selector has three selection outcomes — full suite, a scoped set, or nothing — and each is reached by an explicit rule; the fourth outcome in the bullet below is not a selection but a refusal to produce one. Several rules produce the empty one, which is worth knowing before reading an empty selection as "the inert list matched": documentation, dashboards and *.md are dropped first wherever they live, a *.disabled Chainsaw suite and a examples/backups/<app>/ edit whose app has no suite are dropped where chainsaw and backup-example paths are matched, and inert_config_pattern covers repo meta, agent config, hack/testdata/, the helm-unittest fixture charts under packages/tests/, the three .gitattributes that hold nothing but linguist-generated markers, which reach no build, chart or test (enumerated rather than matched by filename, because the justification is what those files contain and the name does not carry it — .gitattributes can also set filter, eol, working-tree-encoding and export-subst, each of which changes what lands in the working tree and therefore what gets built, so a by-name rule would make such a file inert in silence), the top-level hack/*.bats that are not hack/e2e-*.bats, and their shared hack/test_helper.bash. These are the unit lane, and the root Makefile is the authority on the split: BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats)) keeps every non-E2E file for make unit-tests, while packages/core/testing runs the E2E-prefixed files, so the sandbox never executes a unit file and no Chainsaw suite can regress from an edit to one. Being inert here does not leave them untested: the checks job runs Bats for both code and docs-only changes. Its code path also runs the remaining unit and controller tests. A bats-only pull request still runs install and the OpenAPI tests, since the e2e job reads code rather than the selection. full_suite_pattern covers shared build inputs and the e2e harness, and the graph lookup handles packages/(apps|system|extra)/. A path matching none of them escalates to the full suite and logs why, because both lanes read an empty selection as "skip Chainsaw" and then post E2E Tests green — so a silent default is a green gate with nothing tested (#3392). When that escalation fires for something genuinely inert, add it to inert_config_pattern; do not relax the fall-through. The escalation list and the inert list are checked in that order, so a specific escalation beats a broad inert directory.
• Coverage is decided per changed path. A path that resolves through the graph but reaches no runnable suite is covered by nothing, so it forces the full run on its own account, whatever the rest of the diff selected. Reading that off the merged selection instead let any other changed path contributing a suite name swallow the escalation, with nothing in the output to record that it happened: change a platform component, adjust the Chainsaw test beside it, and the component's escalation disappeared behind that one suite (#3330). The unit is the path rather than the source, because one path can belong to several PackageSources, not all of which need reach a suite, and deciding per source would escalate on one that reaches none and run everything for every change to the path.
• Two sources are excluded from the reverse-dependency walk as install-ordering hubs. Every *-application source depends on cozystack.cozystack-engine so its *-rd HelmRelease waits for the ApplicationDefinition CRD, and the sources that depend on cozystack.cozystack-basics do so to get a namespace or a platform-wide policy in place first. Neither edge says the dependent's behaviour changes when the hub does, so carrying selection across them goes wrong in both directions: through the engine a cert-manager change fans out to every app in the tree, and through basics the platform's namespace-and-policy package narrows to whichever single suite happens to land downstream of it — 22 suites down to one on #3426, where the new site-router suite sits under kubevirt-cdi, which depends on basics. Dropping the reverse edges leaves both hubs reachable and stops them propagating; a change to either still runs everything, because its own group then reaches no suite and the per-path escalation fires on it. Adding a third one is a real decision, not bookkeeping: it says every existing edge into that source is ordering rather than behaviour, so read them before extending the list.
• There is a fourth outcome, and it is a failed step rather than a selection. It fires when the suite list itself is unavailable: find cannot list hack/e2e-chainsaw/, or that directory holds no suite in any of the layouts above. There is no fallback left to take, because escalating would print an empty list and both lanes read an empty selection as "skip Chainsaw" before posting E2E Tests green — the outcome meaning "run everything" would deliver a run of nothing. So the selector names what broke on stderr and exits non-zero; both lanes run the step under bash -e, so the job fails and the required check goes red. A red Select E2E tests step is a real signal, not a flake: read its stderr line before rerunning. A yq failure is deliberately not in this class — what is missing there is the dependency graph, not the suite list, so the selector still has a safe answer and escalates to the full suite. That is conservative but quiet: the only sign is one stderr line inside a step that passes, so a selector running blind on every PR looks exactly like a project that escalates a lot.
• Every escalation names its cause on stderr. "Run everything" is the same list of suite names whichever rule produced it, so a reason line is the only thing that distinguishes the branches that can reach it, and without one the answer to "why did this pull request run the whole suite" has to be re-derived by hand from the diff. Four of them used to be silent — the shared hack/e2e-chainsaw/_lib/ and .chainsaw.yaml case, the full_suite_pattern match, a packages/ path no PackageSource claims, and the empty-selection backstop — and full_suite_pattern is the commonest cause of a full run by a wide margin, so the usual answer was the one the log never carried. Each line names the path or the sources that caused it, and a unit test asserts each message, because a reason that regresses to silence changes no selection and is invisible to every other test. The backstop is the exception: every rule that selects a suite by name membership-tests it where the path is classified, so nothing reaches the backstop and no test asserts its line; it stays as the guard for a future rule that forgets to. The lines go to stderr and must stay there: stdout is the suite list and both lanes parse it, so a reason line on stdout would be read as a suite name.
• The one path the fall-through cannot catch is a new e2e workflow. inert_config_pattern makes .github/ inert as a directory, and the workflows that run the suite are exempted from it by name in full_suite_pattern. A workflow added under .github/workflows/ therefore reads as inert until someone adds it to that enumeration, and the fall-through never sees it. That is why the list carries a rg -l test-chainsaw .github/workflows/ reminder: the enumeration is the guard, not a convenience.
• The full-e2e PR label forces the whole suite.
• The selection is passed to make test-chainsaw CHAINSAW_SUITES="<names>", which runs hack/e2e-chainsaw-run.sh <names> from hack/e2e-chainsaw/ (empty = every directory there). The runner hands chainsaw one suite at a time and runs the tests a single chainsaw test <names> would discover, no more and no fewer.
• The coverage gap TIA opens on PRs is closed at release-cut time: the "Prepare release" commit bakes image digests into packages/core/ (and other packages), which TIA escalates to the full suite — so the release PR exercises the whole suite against the commit the tag will point at.

When adding a new app package, confirm select-e2e.sh maps it correctly (it has a unit test, hack/select-e2e_test.bats). A new app whose suite directory does not yet exist will not be selected — add hack/e2e-chainsaw/<app>/chainsaw-test.yaml first.

src_to_suites() maps a PackageSource name to the suites it owns and is the inverse of select-install.sh's suite_to_source(). A suite can have several owners: the Kubernetes suites create both Kubernetes and KubernetesNodes resources, and the VM suite creates both VMInstance and VMDisk. A test walks every owner through both tables, so a source whose name fits neither convention — not <suite>-application, not the suite name itself — fails there rather than quietly making its own suite unreachable, which turns every change to that package into a full run (#3665). Packages needed only by a suite, such as its backup APIs, belong to suite_runtime_sources() in the install selector; they do not change reverse ownership.

What TIA does and does not do

Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs," and the wiring has consequences worth knowing before you rely on it or change it.

• It trims only which suites chainsaw test runs, not the expensive stages. Prepare environment and Install Cozystack (the full platform install) run regardless of the selection. TIA only decides the CHAINSAW_SUITES passed to the final E2E step. A perfect narrow saves the suite tail, not the bulk of wall-time. (Package image builds are scoped separately by hack/build-matrix.sh; the Talos leg runs when its direct or shared inputs changed and remains unconditional for forks, while installer/finalize runs for every non-docs PR. That is a distinct mechanism from TIA's suite trimming.)
• Select E2E tests runs after Install Cozystack and has no always(). GitHub implicitly ANDs a step's if: with success(), so if install fails the selector (and Run E2E tests) are skipped entirely. On a red install you will not see TIA narrow anything — that is the ordering, not a selector bug.
• Most PRs legitimately escalate to the full suite. Platform manifests live under packages/core/, Go under api//internal/, helpers under hack/, plus the Makefile — touching any one triggers the full suite. So "all suites selected" is usually correct, not a failure to narrow.
• Two skip layers, with a gap. The plan job computes a coarse docs-only gate (code, true for any change outside docs/**) that skips the whole build+E2E pipeline for docs-only PRs, plus a per-package build matrix (hack/build-matrix.sh) that scopes which images build. TIA's finer skip (select-e2e.sh ignoring docs/, dashboards/, *.md and the rest of inert_config_pattern) only trims the suite selection. A PR touching only dashboards/*.json therefore still runs install, then skips just the Chainsaw suites — near-zero savings. If you need such PRs to skip the pipeline, widen the plan docs-gate, not select-e2e.sh.

8. Failure must be self-explanatory — diagnostics are first-class

• On install failure, the CI step dumps kubectl get hr -A -o wide, a describe of each non-Ready HR, and sorted events, all under collapsible ::group:: blocks.
• On E2E failure, the CI step dumps kubectl get hr -A, sorted events, and COSI bucket/claim/access readiness columns (those CRDs ship no printer columns, so plain get shows only NAME/AGE). The whole suite run is one CI step, so this dump is at run level; Chainsaw's own per-Test catch blocks add scoped events/describe/podLogs for the failing suite.
• A suite whose pods outlive it fails where it leaked, not where the memory ran out. A suite can pass and still leave its workloads running — brokers of Kafka releases whose application was deleted, a TenantControlPlane stuck on its finalizer — and the first thing to notice used to be a later suite failing on Insufficient memory, hours on and under the wrong name. Chainsaw v0.2.15 has no hook that runs after a test's cleanup (its Configuration carries error.catch, which runs only on failure), so hack/e2e-chainsaw-run.sh runs one chainsaw test per suite directory and checks between them, with hack/e2e-chainsaw/_lib/leak-guard.sh. A pod is charged to the suite when it is not terminal, did not exist before the suite, and the root of its ownerReferences chain was created at or after the suite started on the apiserver's clock; a platform pod replaced mid-suite still traces back to an object the install created, so it is not charged. Charged pods get the default timeouts.cleanup of .chainsaw.yaml (5m), counted from when chainsaw test returns and so after any per-suite cleanup override has been spent, to disappear; past that the run prints LEAKED-PODS suite=<dir> with one line per pod and its root, raises a leaked-pods error annotation, and adds a failed leak-guard case to chainsaw-report.xml. Every suite that leaves nothing behind logs how long it took to clear, and leftovers of a suite that already failed are reported once as a warning, without the wait. The wait has three outcomes, and nothing is retried. A timeout or a spent budget gives the verdict from the last good pod list, including when the re-list after it fails: a leak fails the suite. Any other error on a wait or a re-list, before the budget is spent and before any timeout, stops the waiting and is reported as LEAKED-PODS suite=<dir> count=N inconclusive="<error>", a warning rather than a failure, so an apiserver blip while waiting cannot turn into a leak red. A wait that fails only after running for its whole timeout counts as a timeout whatever kubectl printed, since kubectl reports a deadline that passes before its watch starts as the client's context deadline exceeded. An apiserver blip before a suite starts is different: if the reading taken ahead of the suite fails, that suite is not run and is reported as not run, which fails the run. A failed first reading after a suite leaves the suite unchecked, and it fails with a leak-guard annotation and a report case saying the cluster could not be read, not that it leaked. A suite with a known, filed leak is listed in hack/e2e-chainsaw/_lib/leak-allowlist.txt as <suite-dir> <issue-ref>; its leak prints as a warning and does not fail. The exemption covers the whole suite, so the PR that fixes the issue deletes the line, and an entry naming a directory that is not a suite stops the run before the first suite.
• Inside a suite, attach scoped diagnostics with a top-level catch: block (events, describe, podLogs, get) rather than letting a bare assert failure through — see hack/e2e-chainsaw/bucket/chainsaw-test.yaml and hack/e2e-chainsaw/harbor/chainsaw-test.yaml. This replaces the BATS ... || { echo …; kubectl get/describe …; false; } idiom.
• make collect-report → hack/cozyreport.sh uploads a full cozyreport.tgz (operator/Flux/cert-manager logs, LINSTOR resources and volumes, per-node ZFS datasets in exact bytes, Kamaji, Talos dmesg, COSI YAML) on every run, and the Chainsaw JUnit report (chainsaw-report.xml) is uploaded as its own artifact.
• Which machine a run landed on is recorded before the sandbox container exists, in sandbox-host/runner-identity.txt: the OCI shape, both region keys, the availability and fault domain, the numbers of the shapeConfig sizing object, and the runner VM's own kernel release, online CPU count and processor model. A lane asks for a runner class rather than for a machine — a flex shape is filled from whatever capacity the pool has, and the region is a preference with a fallback — so without this a red run and a green one cannot be compared by the hardware they ran on, which is the first question a flake that only some runs have will raise. hack/e2e-runner-identity.sh takes the read on the runner VM, from packages/core/testing/Makefile's apply target, because the metadata service answers on a link-local address the sandbox container has no guaranteed route to, and an "unreachable" answered from in there would look exactly like a capture that had tried. Every e2e lane reaches the sandbox through make prepare-env, so the record is taken once per run in each of them, on the passing path as well as the failing one. It goes to the job log as well as to the file, and the log copy is not a convenience: a run whose sandbox never came up has no cozyreport.tgz at all, so on that one path the log is the only place the record exists.
• The keys are fetched one at a time and the root instance document never is. That document carries the instance and compartment OCIDs, the image OCID, the display name, the hostname and the user-supplied metadata bag, and the report is a public artifact — so asking per key makes the privacy property something the wire settles rather than something a filter has to be trusted for. It also removes a guess: Oracle documents the shape sizing under its own shapeConfig subpath and the root document does not carry that object at all, so a reader hunting it inside the root document would have reported the size of every machine as unknown and blamed the service for it. From the sizing object only the numbers are kept, and only under keys shaped like field names, which makes the rule hold for keys nobody has seen yet: a size is a number, an identifier is a string, and a key is printed into the record exactly as a value is. That object is also refused whole unless it arrives as one complete JSON object, so a truncated answer or a top-level array cannot be read in part and published as a sizing record. A per-key answer that is not one scalar is named as such rather than printed, since a service answering a whole document to a key request is the one way the design could still publish what it never asked for. The first key doubles as the probe and is held to answering like this service does, so a machine that is not an OCI instance costs one ceiling rather than one per key whether the address is silent or something else is answering on it, and either way the run leaves an imds unreachable line to grep for. Every read is taken through a wall-clock ceiling, because a diagnostic that hangs costs the step its whole budget without ever reaching an exit status; where timeout is not on PATH the local reads run unbounded instead of not at all and the record says so in a [bounds] line, the same trade hack/cozyreport.sh makes for its own reads. None of it is fatal, since this runs inside the one retried step in the job — a runner that is not an OCI instance leaves an imds unreachable line and an exit status of 0. The report writes that filename either way, holding the record or the reason there is none, because an unidentified runner and an unremarkable one would otherwise leave the same absence.
• The report's per-pod evidence, under kubernetes/pods/ in cozyreport.tgz, is selected by the READY column rather than by STATUS — so a pod left at 0/1 Running by a readiness probe that never passes is in scope, where before it was not. In scope is not the same as collected: the walk stops at COZY_REPORT_PODS_MAX pods or COZY_REPORT_PODS_BUDGET seconds, and what it did not reach is named in the artifact rather than dropped in silence, as the next point describes. Selecting on the STATUS column alone drops exactly that pod, and it is a common way for a suite to time out: the replica never joins, its HelmRelease sits in InProgress, and the artifact carries no trace of the pod the run died on. Terminal Succeeded/Completed pods stay excluded, since they legitimately end unready. Widening the selection and capping the walk are only safe together if the widening cannot displace what the uncapped walk always kept, so the collection order splits by evidence class before namespace: every pod that is not Running — a crash loop, Pending, Error, the class the collector took before this — is ordered ahead of every pod that is Running but unready, and COZY_REPORT_PREFER_NS orders within each of those two classes rather than across the whole selection. Otherwise a failed install with more not-Ready pods than the cap spends it on newly-in-scope tenant pods and drops the container logs of the platform pod that is actually crashing.
• Every bound that the report's pod collection hits is named in the artifact, not only in the job log. The pod walk stops at COZY_REPORT_PODS_MAX pods or COZY_REPORT_PODS_BUDGET seconds and leaves COLLECTION-TRUNCATED.txt naming whichever fired; a pod it cut off part way through leaves CONTAINER-LOGS-TRUNCATED.txt; a pod list that never came back leaves COLLECTION-FAILED.txt rather than the empty tree a healthy cluster produces; a missing local timeout leaves COLLECTION-UNBOUNDED.txt, because the reads still run, just without a ceiling, and a missing dependency on the machine writing the report is not a statement about the cluster; a kind this cluster does not serve at all — kubectl get vm -A on an install without KubeVirt, which exits non-zero — leaves KIND-NOT-SERVED.txt rather than a failure marker, since that is the cluster answering correctly and a marker that fires on healthy installs is one a reader learns to skip; and a knob value the collector could not use is named too, appended to COLLECTION-TRUNCATED.txt when there is one, written to COLLECTION-NOTES.txt when there is not, and appended to COLLECTION-FAILED.txt instead when the pod list itself never returned, so the note always sits beside the failure it accompanies rather than in a file the reader has no reason to open, and a bound that turns out to be the default is not mistaken for the bound that was asked for. A pod whose container list came back as a prefix — kubectl named some containers and then failed — leaves CONTAINER-LIST-INCOMPLETE.txt beside the logs it did manage to read, because the containers it never named have no file at all and their absence would otherwise read as containers that were silent. A read that succeeded while kubectl also wrote to stderr — a deprecation notice, a partial-result warning — leaves that message in READ-WARNINGS.txt beside the object rather than inside it, since pod.yaml and the logs are machine-read and a warning line in them costs the parser the file, while dropping it altogether would be the silent loss of evidence this tree exists to prevent. A pod directory whose containers produced any log at all also carries CONTAINER-LOGS-BOUND.txt, naming the --tail bound those logs were written under (a pod whose every container was silent has no bounded log to describe, so the file is absent), because --tail drops the oldest lines and nothing in a log says so; it sits beside the logs rather than inside them, since a log read in full is a complete artifact and the platform's controllers log JSON per line, so a trailing prose line would break every parser that could read the file before. A pod whose container list could not be read leaves logs-UNAVAILABLE.txt in its own directory, carrying kubectl's reason, because a pod directory holding pod.yaml and describe.txt and no logs at all otherwise reads as a pod whose containers were silent. The VM, VMI and services loops carry the same pair of bounds as constants rather than knobs, so a failed install that leaves every LoadBalancer without an address collects the first of them and names the rest instead of running the report step past its own timeout-minutes: 30, which would lose the tarball rather than truncate it, since the collector writes it on its last line. That step ceiling is the outer backstop for a read with no bound of its own; it does not save this report, it saves the steps after it, the other uploads included. It carries continue-on-error: true alongside, because the step's || true is its standing statement that collecting diagnostics is never fatal and a step killed at its ceiling never reaches that || true — without it a passing run whose collection ran long would be published as a failed one, and every later failure() in the job would start meaning "the tests failed or the report was slow". That tolerance also hides the overrun, since the step then renders green, so a following step turns the collect step's outcome into a ::warning:: annotation — a bound that fires is recorded, and the outer bound is not exempt from the rule the inner ones follow. The LINSTOR reads that run a command inside a pod carry a ceiling too — they are the only reads whose duration a workload sets rather than the apiserver, and a wedged controller holds the stream open with nothing on either end to close it — and they record a cutoff in COLLECTION-FAILED.txt beside their output, which for the two that stream a gzipped tarball is the only place it can go, since a line of prose inside one is an archive nothing opens; the ones that produce a table carry the marker in the file as well. A LINSTOR table that got nothing at all — the wedged controller killed before it wrote a byte — says so in the file rather than shipping at zero length, and a read whose stderr could not be captured says that too instead of reporting silence. A LINSTOR table that wrote output and then failed carries the marker inside itself whether the clock or the command ended it, in the same # [cozyreport] form every other in-file marker uses so one anchored grep finds them all, and the note about a satellite pod list that came back short lands under linstor/error-reports/ beside the bundles rather than in the shared Kubernetes file, like every other .txt in the tarball, because a short node list whose explanation sits one directory up reads as a small cluster; the two archives cannot take one, since a line of prose inside a gzip stream is a file nothing opens. Off the clock the in-file marker on a table names the exit status and does not say TRUNCATED, because kubectl exec passes the remote status through, and a command that lists everything it can and exits 1 for one argument it could not open — zfs list over a missing pool — wrote a complete listing. A cut-off bundle goes one of two ways and the note names both: a stream that got members across is kept and truncated, so a short member list there is the cutoff rather than a satellite with few reports, while a stream killed before its first byte leaves zero bytes and the emptiness check at the call site removes it. The reader runs before that check, so it says which two outcomes are possible instead of asserting one. The LINSTOR reads whose output is a table keep the remote command's whole message in the file itself when the command produced no table of its own — a partial table plus an error keeps the message beside the file instead, so the table stays a table — because kubectl exec appends its own command terminated with exit code N last — so the last line there is the one line that never carries the cause, and a listing sent away from its own error is a zero-byte file saying nothing. That note states what the status leaves open instead of resolving it: kubectl exec returns the remote command's own exit status, so a 124 there is either the collector's deadline firing or the command itself exiting 124, and nothing separates the two. Every other read in the collector carries a ceiling too, gates included — an if kubectl get deploy … that decides whether a module runs is a read, and an unbounded one ends the whole report before the module it guards. A gate cut off short is recorded rather than treated as an answer, because skipping the module produces the same empty tree as a cluster that never had the component; a gate that answers "not installed" is recorded nowhere, since that is the cluster answering correctly and a marker firing on every install without KubeVirt is one a reader learns to skip. What a per-read ceiling does not give you is a ceiling on the report: the pod, VM, VMI and services walks carry a count cap and a time budget, and every other per-object walk in the collector carries neither, so their cost is the cluster's to set and a failed install with a hundred not-Ready HelmReleases spends four hundred bounded reads there, four per release. Described as "the walks outside those four" rather than listed by name, because a list of them drifts and reads as complete while it does so — which is the defect this same change removed from the collector's own header. The step's timeout-minutes is what bounds that, not the per-read wrapper. Whichever bound stopped a walk says so in COLLECTION-TRUNCATED.txt, count and clock alike, because objects removed by the clock while the count note stood alone would read as a cap that never fired. The read that feeds each loop is bounded too: the budget cannot start until rows arrive, so an apiserver hanging on the cluster-wide list would otherwise spend the whole collection step with the deadline never consulted. The budget is a deadline for starting a read rather than for finishing the walk, and it is rechecked between containers but not between a container's current and previous log, so the ceiling is the budget plus one pod's object reads plus both log reads of one container, which is the price of never leaving a half-written file behind.
• COZY_REPORT_POD_TAIL bounds each container log at 2000 lines by default, and a log that was read successfully records which bound it was written under, since --tail drops the oldest lines and a file that begins mid-stream looks exactly like a complete one; -1 restores kubectl's own default of the whole log. COZY_REPORT_PREFER_NS (default tenant-) is collected first, so a cap that only bites on a broadly degraded cluster is not spent on platform namespaces while the pod the suite died on falls off the end.
• Capture the PREVIOUS container instance, not just the current one. For a crash-looping pod kubectl logs (and the Chainsaw podLogs collector, which in v0.2.15 has no previous option — it does accept name/namespace/selector/container/tail/timeout/cluster/clusters) shows the Nth restart — a fresh replay of the same startup path — while the evidence that explains the failure sits in the immediately preceding instance, reachable only via --previous and lost for good once the kubelet garbage-collects that container. hack/e2e-capture-previous-logs.sh closes this once for every suite from the global error.catch (and from hack/cozytest.sh for the e2e BATS suites) rather than per-suite: it discovers containers with restartCount > 0 at runtime — init containers included, since an interrupted bootstrap lives there — orders the app suites' tenant-test namespace first (a best-effort ordering only — it never restricts the capture, and for a suite whose pods live in a child tenant namespace it simply degrades to input order), and dumps each to the job log and to snapshots/<test>/previous-logs/. Containers that never restarted are skipped, so a healthy run emits one line, not a page of "previous terminated container not found". Every way the capture can come up short is named rather than inferred: the COZY_PREVLOG_MAX overflow, a malformed cap, a COZY_PREVLOG_TAIL that was rejected and fell back to 200 (a zero or leading-zero tail asks kubectl for nothing, which would otherwise be reported as the container having logged nothing), the tail bound each kept log was written under — named in capture-notes.txt beside the logs rather than inside them, since --tail drops the oldest lines and nothing in the output says so while a log read in full is a complete artifact that a trailing prose line would stop any parser from reading, and suppressed on -1 because no bound was applied then — a pod list that never returned (which is NOT the same as "nothing restarted" and must not be reported as such), a read that timed out versus one kubectl itself refused (kubelet GC, RBAC, an unreachable node — different problems that a blanket "kubelet GC" line would hide), a previous instance that produced no output, and a capture the outer backstop cut short. A partial log is kept and marked truncated, never deleted — it is the only copy of something already gone from the cluster. Each of those lines the script itself emits is written to snapshots/<test>/previous-logs/capture-notes.txt as well as to the job log, and the directory is created before the first of them, so a reader who has only the uploaded artifact can still tell a capture that found nothing from one that never ran. The outer backstop firing is the exception: that line comes from the callers, after the script has already been killed, and reaches the job log only. hack/cozytest.sh arms this capture, and the other two legs, only for suites whose own filename or whose parent directory carries the e2e- prefix, because all three read cluster state and a unit runner has kubectl on PATH with no cluster behind it, so a red unit test used to walk into the kubectl-gated legs on its way out, and into the crust-gather one as well on any machine where that binary happened to be installed.
• The data-plane collector is the third capture leg, and the only one that reads forwarding state off the node. hack/e2e-capture-dataplane.sh runs after the crust-gather snapshot, from hack/cozytest.sh for the e2e BATS suites and from the global error.catch in hack/e2e-chainsaw/.chainsaw.yaml for the Chainsaw ones, and it collects what an object-state snapshot structurally cannot: the cilium endpoint and BPF conntrack view from an affected pod's node, the kernel conntrack table that is the authoritative one for the host-to-local-pod path under enable-host-legacy-routing, and the LoadBalancer datapath. Its two callers bound it differently, 600s from cozytest.sh and 300s from the Chainsaw catch, so the collector keeps its own wall-clock budget (COZY_DATAPLANE_BUDGET) under each: 270s by default, under the Chainsaw one, and 570s passed by cozytest.sh. Every call is cut to what is left, nothing starts once it is spent, and the last 90s are kept for the LoadBalancer section. It holds the same notes contract as the previous-logs collector: every read carries its own ceiling, and why a read produced nothing goes to capture-notes.txt beside the capture rather than only to the job log, because an empty result and a read that never answered are the same bytes on disk and only the note tells them apart. The two EndpointSlice reads in the LoadBalancer path follow it too, and write <unknown> rather than <none> when the read did not answer.
• A catch script needs an explicit timeout: that exceeds the inner budgets it contains. A Chainsaw script op is bounded by timeouts.exec (5m in .chainsaw.yaml) unless it overrides it, so a catch whose own timeout -k … wrappers sum past that is structurally guaranteed to hit the ceiling — it reports SCRIPT | ERROR | signal: killed and yields little. That kill is strictly worse than an inner timeout firing, and worse in a way the process tree makes precise: the ceiling SIGKILLs the shell process alone, not its process group. Chainsaw runs the shell under exec.CommandContext (pkg/engine/operations/script/operation.go), whose default cancel is Process.Kill() on the shell PID, so every later step in the script is skipped but any collector the shell already spawned is orphaned rather than killed — it keeps running, and it keeps the script's stdout/stderr pipe open. Because that op wires stdout/stderr to in-process buffers, cmd.Wait() does not return until the pipe reaches EOF, which needs every inherited fd closed, so the op does not even come back at its ceiling until the orphan exits on its own — the ceiling stops being a reliable cut (#4268, where the bucket suite's verify-every-bucket-takes-a-write step ran 606s under a 300s ceiling because mc kept retrying after the shell was gone). An inner timeout instead kills that read at a known budget, exits 124, leaves a partial capture on disk, and lets the rest of the catch run — which is why every read must carry its own cap rather than lean on the op ceiling. Size the op timeout above the sum of the inner budgets and let the inner ones be what fires.
• When the sum cannot fit the op, bound the collection phase as a phase rather than raising the op. The rule above assumes the inner budgets add up to something an op timeout can sit above. On a failure path that has accumulated collectors it stops being true — the tenant-Kubernetes node-join path in hack/e2e-chainsaw/_lib/run-kubernetes.sh sums past what is left of its 67m op once the bringup ahead of it and the snapshot behind it are counted, and several of its reads carry only --request-timeout, which bounds one HTTP request rather than a client retrying against a wedged apiserver, so parts of it have no ceiling to add up in the first place. Raising the op ceiling does not fix that shape, it postpones it: a larger ceiling multiplied by a sum that grows with every collector added is the same problem one collector later. What works instead is a wall-clock budget for the whole phase (COZY_DIAG_PHASE_BUDGET, with cozy_diag_phase_start / cozy_diag_phase_has_time): once it is spent, the collectors that have not started are declined out loud instead of attempted, so whatever must run after the phase — the tenant crust-gather snapshot, in that case — is never queued behind work begun past the deadline. Two consequences worth carrying to the next collector. The budget bounds when a collector may start, not when the phase ends, so it is not a worst case and the comment next to it must not be written as one. And because what runs last is what gets declined, source order becomes load-bearing: put the cheap reads that discriminate between failure modes ahead of the collectors that cost minutes, or the budget will spend itself on the expensive ones and drop the evidence the path exists to produce.
• Set crust-gather --duration explicitly. It is crust-gather's own budget for the collection phase and it defaults to 60s; on elapse it abandons the collection and skips its finish step, leaving a partial tree with nothing to say the rest is missing. The outer wall-clock must exceed it, because the API discovery crust-gather runs first is not covered by that budget at all — with an unhealthy aggregated APIService (the state a failed e2e is usually in) aggregated discovery times out and the fallback re-walks every group, which can consume the entire outer timeout before a single object is written. Do not diagnose a killed snapshot by bumping the outer number alone; the two budgets bound different phases. Send the collector's output to a log inside the snapshot rather than /dev/null so "complete or truncated?" is answerable from the artifact.
• A collector that only runs on failure answers in one direction. Its output has no measured value to be read against, so a claim like "this boot was slow" rests on an impression of what a healthy run looks like rather than on a number from one. Where the evidence is destroyed by teardown — the tenant worker's guest serial console goes with the Pod that holds the stream — the baseline has to be taken on the passing run or it does not exist at all, which is why hack/e2e-chainsaw/_lib/run-kubernetes.sh captures that console on both paths and records which one it was. A passing-path collector is held to two rules the failure path does not need. It must not fail the suite, since everything the suite exists to prove has already passed by the time it runs — and its own cost counts, not just its exit status, so a collector that costs minutes declines itself out loud when the operation's ceiling no longer has room for what it is about to spend, rather than starting a walk that the ceiling cuts off mid-write. That bounds the collector's contribution and nothing else: work already on the path with no ceiling of its own stays unbounded, and no reserve makes it fit by arithmetic. And what it could not collect at all has to reach the job log, not only the report, because a green run's artifact is the one nobody downloads. Per-read outcomes stay beside the read in the report, as everywhere else in that collector: a note about one console is only usable next to that console, while a walk that collected nothing is a fact about the run.
• A diagnostic that carries a relative clock also has to carry the absolute one. A guest kernel stamps its console in seconds since its own boot, so a log that ends at 500 seconds says how far that guest got and nothing about when. Whether it stopped early because it started late or because it stopped printing is decided by the instant the read happened and the instant the guest started, and neither is in the file: the first survives only as the artifact's mtime, which is metadata rather than content and so is invisible to anyone reading the log and lost by anything that regenerates it, and the second only in a describe dump that a healthy-looking capture never triggers. Write both beside the capture, and name the units on each — an epoch integer and an RFC 3339 string subtract to nonsense if a reader takes them as printed.
• Per-failed-test crust-gather snapshots are the richest diagnostic — a crust-gather serve-able archive of the cluster at the moment of failure, before cleanup. The global error.catch in hack/e2e-chainsaw/.chainsaw.yaml captures the host cluster into _out/cozyreport/snapshots/<test>/host on any failure (the Chainsaw analog of the hack/cozytest.sh EXIT-trap snapshot used by the e2e BATS suites); the kubernetes suites additionally snapshot their tenant cluster from hack/e2e-chainsaw/_lib/run-kubernetes.sh while the tenant API LB is still routable. hack/cozyreport.sh folds snapshots/ into the artifact. Do not remove the global catch — without it the Chainsaw suite uploads a report with no per-test cluster state.

9. Keep the test environment deterministic

• Pre-pull every kubeovn / linstor / cert-manager image onto all nodes before those HelmReleases install (hack/e2e-prepull-images.sh), so clustered workloads (OVN raft, LINSTOR, the cert-manager webhook) do not fail on per-node image-pull stagger.
• Exclude loop devices from host LVM scanning in the Talos machine config so the host does not activate volume groups inside loop-mounted e2e disk images.
• Fail fast on node readiness (≈5m, then bail) rather than marching into LB/NFS tests that will also fail — it saves several minutes per attempt and keeps the real failure at the top of the log.
• Give the sandbox a runner class strictly larger than its nodes. The management nodes get fixed CPU and memory (hack/e2e-compose.yaml, holding the same figures hack/e2e-prepare-cluster.bats gives its QEMU guests), and those resources are committed for the whole run, so a lane whose runner only just covers them leaves the host nothing for containerd and the runner agent. A new e2e lane picks its runs-on accordingly; hack/sandbox-runner-headroom.bats derives both sides and fails the unit suite when any lane loses that margin.

10. The E2E workflow split: same-repo vs fork, and the "E2E Tests" commit status

A fork's pull_request run carries no registry credentials, so it cannot push the images the e2e sandbox pulls. E2E therefore runs across two workflows:

• .github/workflows/pull-requests.yaml runs e2e in-tree for same-repo PRs (the e2e job, named E2E (in-tree)). Fork PRs skip this job.
• .github/workflows/e2e-fork.yaml runs e2e for fork PRs via a privileged workflow_run that fires after the fork's Pull Request run completes. It pushes the fork-built images to the registry and runs the suite — the same Chainsaw + TIA path as in-tree (§7) — in an ephemeral sandbox that itself has no push credentials.

Consequences to keep in mind when touching either workflow:

• Fork builds export, they do not push (OCI_EXPORT_DIR). The fork's unprivileged build sets OCI_EXPORT_DIR (see hack/common-envs.mk) so every image is written to a per-image OCI archive artifact instead of being pushed; the privileged e2e-fork.yaml later pushes those archives to the registry by digest. Any build recipe that shells out to push directly — e.g. a skopeo copy … docker://… that bypasses the image-tags/oci-output macros — must honour OCI_EXPORT_DIR too, or a fork build dies with an anonymous-push denied (#3257). This is covered by hack/common-envs_test.bats.

• "E2E Tests" is a commit status — not a job, and not a check-run. The required merge-gate context named E2E Tests is a GitHub commit status, posted by the plan/e2e-report pair for same-repo PRs and by e2e-fork.yaml's resolve/report pair for forks — not the e2e job itself. Each lane opens the context with pending and concludes it when the suite is done, and the opener is not optional: a commit status never expires, so a head SHA that already went green would otherwise keep reading green from the moment a re-run, a reopen or a full-e2e label starts a new run until that run reports, leaving the pull request mergeable across a suite that may be about to fail. While the gate was a job, a fresh non-success check run appeared automatically and closed that window. The two lanes close it at different speeds, and the fork lane does not close it entirely. Same-repo, the opener is plan's first step, so the window lasts seconds. On a fork the opener is resolve, which cannot run until the unprivileged build has finished, so re-running an already-green fork head leaves the old green readable for the length of that build. The gap is structural rather than an oversight: a fork's pull_request token is read-only and cannot write a status at all, which is the same property the rest of this design relies on. It is bounded by the build and it is the one case where "the opener closes the window" is not literally true. Two reasons it is a status:

  • Not a job: a required job that is skipped (as the e2e job is for forks) counts as passed by branch protection — exactly how fork PRs used to merge with e2e never run (#3257). A status is success only when the report step posts it; a never-posted required status stays "Expected" and blocks the merge. If you rename the e2e job, do not name it E2E Tests and do not wire branch protection to a job name — the status must stay the gate.
  • Not a check-run: a check-run created with the default GITHUB_TOKEN floats into an arbitrary GitHub Actions check-suite (in practice the earliest-registered one, a tiny labeler — that is why E2E Tests once rendered under "PR size label"), and an app-token check-run would need the COZYSTACK_CI app granted checks: write. A commit status has neither problem: branch protection matches it by context name and it carries no check-suite grouping. It is safe to post with the default token because a fork's pull_request token is read-only and cannot post a status — so only same-repo runs (plan's opener and e2e-report, both guarded !fork) and the privileged workflow_run in e2e-fork.yaml can set it. Each publishing job carries permissions: statuses: write and calls repos.createCommitStatus. plan's opener is additionally continue-on-error, since every build and test job hangs off plan and a transient status-API failure there must not discard the run; losing it only restores the stale-green window for that one run. Consequence for branch protection: require the E2E Tests status context (no GitHub App and no app-id pin needed).

• Fork TIA reads trusted inputs, but the fork still writes the selector. e2e-fork.yaml's e2e job runs the same select-e2e.sh TIA as in-tree (§7), and its inputs are hardened: the changed-file list and the full-e2e label come from resolve's base-repo API lookups (pulls.listFiles / the PR's labels), the same trusted source as the docs-only decision, passed in as outputs (changed_b64, full_e2e) rather than re-derived from a git diff of the fork-controlled merge ref. Keep that part as it is: re-deriving the selection from the checked-out tree would add a second bypass for no gain.

• What the fork lane is, and is not. The e2e job runs from the fork's tree, so select-e2e.sh, the suites under hack/e2e-chainsaw/, hack/e2e-install-cozystack.bats and packages/core/testing's recipes are all fork-authored. A fork that wants its own suite to report green can edit any of them, and hardening the TIA inputs does not change that. Read the lane accordingly: it is a correctness gate that stops an honest contributor shipping a regression, not a boundary against a hostile one. What contains a hostile fork is that no fork run executes without maintainer approval (below), that the job holds no push credential, and that a diff editing the test harness is visible in review.

• A fork PR that adds a new image cannot go green until that image exists on the base branch. publish derives the set of archive names it will accept from the base parent (HEAD^1), so that a fork cannot mint arbitrary repositories under the CI registry by naming an artifact. An image the PR itself introduces is therefore not in that set, and the job fails with image '<name>' is not in the base-tree allowlist. This is fail-closed by design and there is no label override — land the image on the base branch first, or run the suite from a same-repo branch. If a fork e2e run fails on that line, the contribution is not broken; the allowlist is doing its job. So that a contributor learns this before the build rather than after it, plan in pull-requests.yaml runs hack/fork-new-images.sh on fork PRs and fails with the names of the new images. That check runs from the fork's tree and is a courtesy, not the boundary; hack/fork-new-images.bats holds its derivation to the one publish uses.

• labeled events run in their own concurrency group, and a cancelled run reports nothing. Because the gate is a status rather than a job, a run that dies without posting leaves the head SHA at "Expected" — which blocks the merge rather than passing it. Several separate pieces implement it, and weakening any one of them reopens the gap. Each lane's terminal reporter runs under !cancelled() rather than always(), so a run that was cancelled publishes nothing at all and the pending it opened stands until the run that superseded it reports. Each lane's concurrency key then keeps a run that is going to publish nothing out of the group of the run that is: pull-requests.yaml's appends -label for a labeled event that plan discards, so labelling an open PR cannot cancel a live opened/synchronize run (the replacement run would never post the status the cancelled run was going to) — the condition is the complement of plan's guard, so a full-e2e label, which does publish, stays in the main group and supersedes the head run instead of racing it, e2e-fork.yaml's concurrency key appends the triggering run's id when that run concluded cancelled or skipped so a privileged run that is going to post nothing cannot cancel the one that is going to post, and e2e-fork.yaml's resolve returns early on a cancelled/skipped triggering run instead of posting failure for it. If you touch any of them, keep the invariant: exactly one run per head SHA is responsible for the status, and a run that is not that one must post nothing at all.

Security prerequisite (repo setting). The split is only safe if fork runs require maintainer approval before they execute. The repository must keep Settings → Actions → "Require approval for all outside collaborators" enabled, so a fork's pull_request run — which produces the artifacts e2e-fork.yaml consumes — cannot run untrusted code (or feed the privileged run) without a maintainer's go-ahead.

11. Never negate a BATS assertion with !

An assertion spelled ! cmd inside an @test body cannot fail, whichever runner executes it. POSIX exempts a command whose return value is inverted with ! from set -e, so the failing status is dropped where it is produced. hack/cozytest.sh then appends return 0 to every test function, so it never resurfaces as the test's own status. The bats binary looks stricter and mostly is not: its ERR trap carries the identical exemption, and bash -c 'set -eE; trap "echo TRAP" ERR; ! true; echo REACHED' prints REACHED, runs no trap, and exits zero. The one shape bats does catch is a negated assertion standing last in the body, where the inverted status is what the function returns — put any statement below it and bats passes the test too. This is the same family as the ban on test-level EXIT/RETURN traps, and unlike that one it is not settled by preferring the stricter runner, because there is no stricter runner to prefer.

The exemption is also wider than the "negated pipeline" phrasing found in older comments: a bare ! true, a negated [ … ], one inside a for body, and one following ;, && or || all pass. Write the absence so that it fails — if cmd; then echo "FAIL: <what must not have happened>"; false; fi — and let the false carry the verdict. hack/bats-no-negated-assert.bats enforces this across every hack/**/*.bats, subdirectories included. Inverted conditions (if !, elif !, while !, until !) are untouched, because set -e is not what decides a condition. [ ! -f x ] is untouched for a different reason and stays a normal, failing assertion: the ! there is an operator of test, not negation of a command, so nothing exempts its status. The scan is lexical, with the limits that implies: a statement assembled at runtime is invisible to it, and the form is counted inside a heredoc or a fixture string where no command exists — which reports a file holding nothing rather than passing one that does, and is why that guard's own fixtures build the token from a separate printf argument.

Chainsaw v0.2.15 gotchas

The suite is pinned to Chainsaw v0.2.15 (the latest release as of May 2026); none of the traps below are fixed upstream yet, so the workarounds stay until a bump is possible. Each one has already cost a debugging session — this is the "no one gets it right the first time" list, worth a read before writing a new assert.

• Condition asserts must use the filter-as-list form. (conditions[?type == 'Ready']) evaluates to a list; assert against it with a list body (- status: "True"). The indexed (conditions[?type == 'Ready'])[0] form throws field not found (also enforced by convention 2).
• No number literals / backticks in JMESPath. ports[?port == `443`] mis-evaluates in v0.2.15 — the numeric comparison silently returns the wrong set. Assert ports (and any numeric field) with the exact-ordered projection form instead: (ports[*].port): [80, 443]. sort() and contains(list, <number>) are likewise broken with numeric arguments.
• error: fails hard on an unknown GVK. Unlike the BATS ! kubectl get <kind> idiom (which passes when the type is absent), a Chainsaw error: op against a kind whose CRD is not installed errors the step rather than asserting absence. If a type may legitimately not exist (a version rename, an optional component), gate the check behind a positive precondition or do it in a script step — never a bare error:. This is what broke the etcd suite on the v1alpha1 → v1alpha2 rename.
• No conditional Test-level skip. There is no skip: <expr>; an env-var-gated test must exit 0 early inside a script step (see the ETCD_E2E_S3_ROUNDTRIP gate). A Test with no matching gate always runs.
• finally is step-level, not test-level. To guarantee teardown of something Chainsaw did not create, put the work and its own cleanup in a single script step; there is no test-wide finally block to hang it on.
• Cleanup is bounded and blocking — it surfaces latent teardown bugs. Chainsaw waits for the resources it applied to actually delete (within the delete/cleanup timeouts in hack/e2e-chainsaw/.chainsaw.yaml) and fails the test if they don't. This is deliberate — a stuck uninstall starves the next suite — but it means a teardown path the BATS suite masked with || true now surfaces as context deadline exceeded at cleanup. The failure is honest, not new; fix the teardown rather than widening the timeout (see cozystack/cozystack#3271, a pre-delete hook that hung on a nodeless tenant).
• Reading a failure diff: the "actual" side is a projection. Chainsaw projects the live object onto the expected fields, so status: {} in a diff means "the expression matched nothing / evaluated empty," not "status is literally empty." The real reason is the error line above the diff — read that first.
• A catch/cleanup script's cwd is the failing suite's directory (hack/e2e-chainsaw/<suite>/), so ../../.. is the repo root. The suites cd ../../.. and then source shared helpers by repo-root-relative path (hack/e2e-chainsaw/_lib/…) rather than juggling ../ hops.

Reviewer checklist for a new or changed E2E test

1. No new retry loop unless the step is pure infra (image pull / VM boot / network).
2. Resource readiness uses a Chainsaw assert (not an imperative until kubectl get …; kubectl wait); condition checks use the filter-as-list form (conditions[?type == 'Ready']).
3. Imperative-only waits live in a script step; any bare sleep carries a TODO(e2e-replace-fixed-timeouts): justification, unless it is a measurement interval, which §2 carves out by name and which takes no marker because no event can replace it.
4. No test-level EXIT/RETURN trap — rely on Chainsaw cleanup; a self-contained trap is allowed inside a single script step (port-forward / temp dir), and inside an explicit subshell within a BATS @test, where it does not displace the runner's own handler. A BATS file that still carries test-level traps declares the count itself in # EXIT-TRAP DEBT: N; do not ask for one to be removed without checking that line, because the count is exact in both directions.
5. Controller-created artifacts the test cannot reclaim are pruned explicitly; nested tenants delete child → parent with a wait-for-deletion between.
6. An HR-Ready assert has a floor: the release's own install timeout, not only the app's boot time. Below it a failed install reports as a bare assert timeout, and above it by less than a start margin it can, so a number under the floor buys a shorter red run at the price of the diagnosis. Whether to leave a suite under it is a per-suite call, made against what that suite's red runs cost, unless no budget inside the E2E job's cap can clear the floor at all, as with kafka's 7h install timeout; either way, a Chainsaw assert left under it says why in-line. Derive the floor from Sizing an HR-Ready budget rather than copying a neighbour's number, and justify a raise in-line. The same goes for any other wait longer than the ones around it, HR-Ready or not: it carries its reason in-line.
7. Failure path attaches scoped diagnostics via a catch: block, never a silent pass. The node-join deadline is no exception, and no longer has one: with the soft-red marker gone it fails its suite like any other deadline and its diagnostics run from the same catch:.
8. If it touches HelmRelease install/upgrade behavior, add the status.history guard: fail the run on a failed or uninstalled Snapshot of a release whose install strategy is RetryOnFailure, fail it on a parent whose strategy is anything else, and report the footprint of every other release without failing. What is reusable today is the predicate and the jsonpath in hack/e2e-chainsaw/_lib/remediation-guard.sh; the policy that weighs the footprint against .spec.install.strategy.name, judges an empty history against the release's own readiness, and keeps each read's exit status apart from its output lives in cozy_guard_helmrelease in hack/e2e-chainsaw/_lib/run-kubernetes.sh — read it before rebuilding any of that for another suite. If the suite has a failure path that exits before the guard, give that path cozy_report_helmrelease_remediation as well, or the runs the guard exists for are the runs it does not see.
9. A test that creates a tenant Kubernetes cluster waits for its release to reach Ready before the assertions it ends on, with a --timeout the guard can read as positive and no longer than a day. Reads earlier in the run are expected and are not what this bounds. The mechanism, how to size the budget and what the guard does with it are under "Do not mask cleanup or teardown failures". One thing to carry from there rather than look up: the naming is part of the rule, not a detail of the check, so an end-assertion added under any name outside the cozy_assert_* / cozy_switch_and_assert_* convention is invisible to it and passes before the wait without reddening anything — as long as one recognised call site is left behind it. Rename every recognised call and the guard reds instead, on finding none at all, which is the one shape of this mistake it does catch.
10. No BATS assertion begins with ! — an inverted status is exempt from set -e and the assertion cannot fail; use if cmd; then echo "FAIL: …"; false; fi.

The container lane: what the srv nodes are, and what that costs

The srv1-srv3 nodes of the PR lane run as Talos containers, not QEMU guests. The sandbox used to boot three guests with Cozystack running KubeVirt inside them, which put a tenant worker at L3 — Oracle host, runner VM, srv VM, worker. Running the srv nodes as containers deletes one of those levels and puts the worker at L2.

What that buys, measured in CI on commits where both substrates ran: the two tenant-Kubernetes suites take 466-809s on containers and pass in both the vgif-present and vgif-absent strata, against 1143-2040s on QEMU where the vgif-absent runs missed the node-join deadline outright. Whole-job wall clock came down 24-49 minutes across three such runs, and essentially all of it is in Chainsaw rather than in bringing the stand up: install takes about half an hour on either substrate.

Three properties of container mode drive the whole design, and each one fails quietly rather than loudly if ignored. machine.kernel.modules is a silent no-op — kernel_module_spec.go returns early on ModeContainer with no error and no event — so the host must load openvswitch and zfs before the nodes start, and both the host script and the BATS file assert it rather than trusting it. Docker gives a privileged container its own tmpfs /dev, seeded once at creation, so /dev/kvm works but ZFS zvols created later never appear and linstor-csi fails ControllerPublishVolume; the compose file binds the host devtmpfs instead, which is also why talosctl cluster create docker can never carry LINSTOR. And container mode has no ARP VIP, so 192.168.123.11 stands in for the QEMU lane's 192.168.123.10 and hack/e2e-install-cozystack.bats takes the endpoint from COZY_APISERVER_ENDPOINT, defaulting to the unchanged QEMU value.

In CI this is every lane that runs the suite: the e2e jobs in .github/workflows/pull-requests.yaml and e2e-fork.yaml, which gate a merge, and the e2e jobs in nightly.yaml and e2e-tag.yaml. No workflow boots the QEMU substrate or downloads a nocloud disk any more. make -C packages/core/testing prepare-cluster and hack/e2e-prepare-cluster.bats stay in the tree because unit tests still read the node sizing and addressing from that file, but nothing in CI runs them, so DRBD, the replicated StorageClass and the Cozystack Talos node image have no e2e coverage until a lane with a kernel per node exists (cozystack/cozystack#4642).

Root entry points are make prepare-env-container (host-side compose plus the in-sandbox BATS file) and make -C packages/core/testing delete-cluster-container, which is also the retry cleanup: a half-started compose project the next attempt adopts is not a fresh try.

The nodes run as siblings on the runner's docker daemon, not nested inside the sandbox, because those host binds have to resolve against the real host devtmpfs. That is why provisioning is two steps: hack/e2e-container-up.sh does the host-side half and needs a docker socket, then the sandbox is attached to the compose network and hack/e2e-prepare-cluster-container.bats runs inside it exactly as the QEMU lane's file does.

What the lane gives up is DRBD, and with it more than the replicated StorageClass. DRBD's resource registry is global per kernel, so three containers sharing one kernel cannot represent three DRBD nodes — a second new-resource for the same name returns success while changing nothing, which is worse than an error because LINSTOR then records a replica it does not have. It applies to kind equally; it is a property of one kernel, not of Talos. linstor must therefore run with drbd.enabled: false, without which every satellite CrashLoops and no PVC binds at all. Because linstor-csi couples RWX to DRBD, tenant worker disks become RWO and live migration is unavailable on this lane, which nothing in the tree tests today either way. Everything else survives: kube-ovn runs the real OVS dataplane, and LINSTOR keeps real ZFS pools and the production local StorageClass.

Two further things go with the substrate, and no lane covers them. replicated is volumeBindingMode: Immediate while local is WaitForFirstConsumer, so the Immediate binding path is not exercised; and tenant StorageClass propagation only applies to classes that are not allowRemoteVolumeAccess: "false", which on this lane is none of them. The management nodes also boot upstream ghcr.io/siderolabs/talos rather than the Cozystack image, so the extensions that image carries — drbd, zfs, openvswitch — and the LVM filter are not validated by e2e. Tenant worker nodes still import a Talos disk, so that half of the Talos path is unaffected.

One property of container mode has no fix here and is worth knowing before reading a red run: a node reports the host's CPU and memory as its capacity. hack/e2e-container-up.sh counterbalances that with kubelet systemReserved so the scheduler sees 8 CPU / 24 GiB per node, which corrects the scheduler and not what a pod reads from /proc. A workload that sizes itself from /proc/cpuinfo or /proc/meminfo is therefore configured for the whole runner.

The local-only mode has one mandatory install-time adaptation and keeps a second for a run that turns root monitoring on. CDI otherwise ranks ReadWriteMany/Block first for any LINSTOR StorageClass, so post-install prep waits for StorageProfile/local and pins it to ReadWriteOnce/Block before any tenant DataVolume is created; without that override linstor-csi rejects the first worker disk because RWX requires DRBD. Root VictoriaLogs is the only root-tenant workload that explicitly defaults to replicated instead of inheriting the default class. The installer leaves root monitoring off, so VictoriaLogs does not render by default, but it still writes a logsStorages override into tenant-root/cozystack-values before the Tenant patch, so a run that enables monitoring gets local claims on the first render; patching the generated monitoring HelmRelease afterward is unsafe because it can create an immutable replicated PVC first.

Two capacity edges are handled explicitly. A container node reports the host's CPU and memory as capacity rather than its compose cgroup limits, so hack/e2e-container-up.sh computes kubelet systemReserved from the host totals and caps each node's scheduler-visible allocatable near 8 CPU / 24 GiB; it refuses a host without strict aggregate headroom, and the live preparation suite asserts every Node's resulting allocatable. Each ZFS pool uses a sparse 200 GB backing file, matching the QEMU lane's per-node data disk: root-tenant reservations can otherwise leave less than the ~21 GiB required by CDI scratch on the node where its 20 GiB target disk is already bound. The apparent 600 GB is not preallocated, but blocks written during the run consume the shared runner disk until teardown destroys the pools and backing files.

That second point used to be much worse than a sizing note. An importer pod mounts two node-pinned volumes -- the disk, and a scratch volume CDI creates only after the pod is scheduled -- so the scheduler sizes the node against the disk alone and can place two importers where both disks plus both scratch volumes do not fit. The scratch volume was then provisioned on a different node and the pod became unschedulable forever against pv ... node affinity doesn't match node, with no error surfaced anywhere and the DataVolume reporting ImportScheduled indefinitely. That is fixed by linstorCSI.strictTopology (on by default), which constrains provisioning to the node the scheduler actually chose; the failure becomes a legible 1 node(s) did not have enough free storage that resolves on its own once space frees up. Both the deadlock and the fix were reproduced by measurement on 2026-08-25.

Sizing an HR-Ready budget

Give an HR-Ready wait a budget that clears, by a start margin, both the app's first-boot budget and the install timeout of the release it waits on, read off the ApplicationDefinition for a generated release and off the rendering template otherwise; the rest of this section is how to find those two numbers.

A HelmRelease install waits through kstatus unless the ApplicationDefinition carries release.cozystack.io/helm-install-disable-wait. kubernetes-rd, kubernetes-nodes-rd and vm-instance-rd are the only three that do, so their installs never poll at all — though the kubernetes chart's blocking post-install hook still holds its install open, and sizing a wait on that kind is covered under "Do not mask cleanup or teardown failures" rather than here. Two other things would take an install off kstatus and onto helm's own readiness logic, neither of them in effect: a spec.release.waitStrategy of legacy, which no ApplicationDefinition sets at all — the only two places in the tree setting a strategy are hand-written HelmRelease templates under packages/extra/, and both set poller — and the UseHelm3Defaults feature gate, which flips the controller-wide default to legacy and is not among the gates helm-controller runs with here. Every other release therefore polls, and the wait in front of it has two floors. It has to beat both by a start margin. A wait equal to the install-timeout floor does not merely tie with it: the two clocks start at different moments, the wait's when the step runs and the install's once the Application has been reconciled and the chart artifact fetched, so at equal budgets the wait is the one that expires, and it expires before the release has recorded anything about the failure. The margin has to cover that gap, the helm-controller queue included; nothing in the tree bounds any part of it, so it is a policy value, held in MBW_START_MARGIN in hack/mariadb-first-boot-waits.bats.

The first floor is the app's own first boot. For a chart that sets startupProbe it is initialDelaySeconds + (failureThreshold - 1) * periodSeconds; for a chart that does not, it is whatever its operator needs, and only a measurement will say. mariadb is the case in this tree where this floor is real rather than nominal: its chart grants a 310s startup budget, so a wait that fails earlier fails a boot the probe was still willing to wait for.

The second floor is the release's own install timeout, and it is usually the binding one. Where to read it depends on who created the release, and getting this wrong under-budgets rather than over-budgets. For a release cozystack-api generated from an ApplicationDefinition the default is 10m (pkg/cmd/server/start.go), overridden per definition by release.cozystack.io/helm-install-timeout — kubernetes-rd at 20m, kubernetes-nodes-rd, tenant-rd and monitoring-rd at 15m, kafka-rd at 7h, a floor no suite clears inside the E2E job's 215-minute cap; the first two also disable the wait, so for kubernetes the override bounds its blocking post-install hook rather than a kstatus poll, and kubernetes-nodes, which renders no post-install hook, has nothing in its install for the override to bound. For a HelmRelease an app chart renders itself there is no ApplicationDefinition to read and that default never applies: the floor is the spec.install.timeout on the template that renders it, which takes precedence over its spec.timeout, and where a template sets neither, helm-controller's own documented default of 5m applies instead. There is deliberately no count of that population here. Read the value off the template you are waiting on: they range from 5m to 30m, they are spread across packages/apps, packages/extra and packages/system, and a release carrying spec.kubeConfig installs into a tenant cluster and is not one of these at all. A number in this sentence would be an obligation to revisit it whenever a template is added, and its three qualifiers are each easy to drop from a recursive grep. The nested <name>-system releases the app charts wrap their workloads in are all of this second kind. A HelmRelease that cozystack-operator builds for a Package is a third kind: its spec.install.timeout comes from the operator's --helmrelease-install-timeout, 10m unless cozystackOperator.helmReleaseInstallTimeout in the installer chart overrides it. At or below whichever floor applies, a failed install is reported as a bare assert timeout, because the release has not yet recorded why it failed.

What holds the install open is anything in the rendered manifests kstatus reports InProgress, which is wider than a Ready condition. kstatus carries built-in rules for a list of kinds — the workloads, but also Service, Pod, PersistentVolumeClaim and CustomResourceDefinition — and falls through to reading Ready only for kinds it has no rule for. Read that list rather than recalling it: legacyTypes in pkg/kstatus/status/core.go of fluxcd/cli-utils, at whatever version the helm-controller image in internal/fluxinstall/manifests/fluxcd.yaml was built against — nothing in this repository vendors it, so there is no local copy to open and the version has to come from that image. It does not divide the way intuition suggests, because Secret, ConfigMap and batch/CronJob are in the table and map to always-ready, so being in it is not the same as gating on it. The practical consequence is that "renders no CR publishing Ready" does not exempt a release, and tcp-balancer and vpn show why: each renders a WorkloadMonitor, whose status type declares no conditions at all, so kstatus falls past the Ready fallback and reads that CR as Current — and then gates the install on the Deployment beside it through a built-in rule. The CR is present and contributes nothing; the workload is what holds the install open.

Every mistake in this derivation under-budgets, which is the harmful direction — but the floor is not free either, because a wait that must exceed its release's install timeout means a genuinely failing suite burns that long before it says so, and .chainsaw.yaml already warns about diagnostics eating the job's own 215m — for scale, the timeouts written on the mariadb suite's own steps come to 120m of that, before the defaults its untimed steps inherit from .chainsaw.yaml, and a job cancelled at the cap loses both its log and cozyreport.tgz. Raise a suite when a failed install there needs to say why; leave one alone when the diagnosis is not worth ten minutes added to every red run in that suite. That is the trade, and it is per suite — "only the ones kstatus polls" is not a filter, because almost everything polls. Two suites are worth reading before writing a new wait, and deliberately not as a list of every wait and its floor: a hand-written list of them is incomplete in a way its author cannot see, and goes stale without saying so — a grep for the call misses the invocations split across lines, and reading for it by eye misses the single-release waits among the --all ones. Enumerate with grep -rnE 'kubectl +wait.*(^|[^[:alnum:]])hr([^[:alnum:]]|$)' hack/ and the chainsaw asserts on kind: HelmRelease, not from memory and not from here. mariadb's 11m in hack/e2e-chainsaw/mariadb/chainsaw-test.yaml is the worked example of the derivation above. harbor's two asserts show why one literal cannot serve a whole suite: harbor-test is generated and clears its 600s floor at 11m, while harbor-test-system is rendered by the app chart with a spec.timeout of 15m and needs 16m.

Two things this derivation does not give you. Readiness may span a first boot rather than always doing so — a poll landing before the operator's first status write reads a condition-less resource as Current and can end the wait early, which is why a suite keeps an endpoint assert after the HR-Ready one rather than treating it as redundant. And a number copied from a neighbouring suite says nothing about the release you are waiting on, even where it clears the neighbour's floor: the floor depends on who rendered the release, and kafka's assert sits under its own on purpose. Scan hack/e2e-chainsaw/*/chainsaw-test.yaml for HR-Ready asserts and hold each against its own release's floor plus the start margin; the result is left to that scan rather than written out here, because a list of them goes stale silently while the scan does not. Chainsaw asserts are not the whole surface: wait_hr_ready calls under examples/backups/ carry the same floor, not all of them from a run-all.sh and not all of them with a literal budget — a call that passes none takes the helper's own default, which a grep for the number misses — so scan both places. Derive the budget from the release you are waiting on.

In-flight direction (not yet the merged standard)

These are being explored on branches and may become conventions; do not assume they are the current main behavior:

• Cilium orphaned-endpoint self-heal — an interim CI watchdog that evicts a single confirmed-orphan Cilium endpoint ("IP already in use", cilium/cilium#38313). Explicitly a mitigation to remove once a fixed Cilium ships; it refuses to touch an endpoint backing a live pod so real duplicate-IP bugs stay visible.
• Cluster state snapshot/restore between test groups instead of reinstalling.