# E2E Testing Conventions Guidance for writing, changing, and reviewing Cozystack's end-to-end (E2E) tests and the CI that runs them. Read this **before** touching anything under `hack/e2e-chainsaw/`, the bootstrap/OpenAPI BATS (`hack/e2e-*.bats`), `hack/*.sh` test helpers, either E2E CI workflow (`.github/workflows/pull-requests.yaml` for same-repo pull requests, `.github/workflows/e2e-fork.yaml` for forks), or `packages/core/testing/`. §10 covers how the two split the work and which invariants hold them together. The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per app, each with a `chainsaw-test.yaml`). Cluster bootstrap (`hack/e2e-install-cozystack.bats`, `hack/e2e-prepare-cluster.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS. ## The core principle **Retries do not recover flakes — they hide deterministic bugs and triple diagnostic wall-time.** An audit of 30 PR runs found that across sampled failures, **25/25 retry attempts failed**: the retry loop never once turned a red run green, it only delayed surfacing real bugs (Helm namespace conflicts, seaweedfs/harbor races, a vminstance disk race) that *looked* like flakes because the retry sometimes coincided with transient state clearing. Every convention below follows from that finding: **fail fast, fail loud, make the failure legible.** A test that flakes is a test (or a product) with a real race to fix, not a test to wrap in a retry. Chainsaw's per-assertion polling is the structural expression of this: each `assert` waits on its own condition to its own timeout, and a failure is reported as a structured diff plus auto-captured events/describe/logs — not a generic non-zero exit. ## Conventions ### 1. No retries on deterministic steps; retry only pure infrastructure - `Run E2E tests` and `Install Cozystack` run **once**. On failure they print `❌ ... (no retry — see diagnostics below)` and dump state. Do not reintroduce a retry loop around them. The E2E step is a single `chainsaw test` invocation — there is no per-app retry loop to bring back, because Chainsaw already polls each assertion to its own timeout. - `Prepare environment` keeps its 3× retry — and *only* it — because it is pure infrastructure (Talos image download, sandbox VM boot, network) where a transient runner hiccup is a genuine flake with no application logic involved. - Rule of thumb: retry is justified only for a step that contains **no product or test logic**. #### The node-join deadline fails the suite but does not block the lane This carve-out is named here rather than left to judgement, and it is not a retry: the tenant node-join wait in `hack/e2e-chainsaw/_lib/run-kubernetes.sh`. Nested virtualisation on the shared runners degrades far enough that a worker which registers in minutes elsewhere can miss any deadline this test can afford, so a red on that wait says as much about the machine the run landed on as about the product, and what it produces in practice is a re-run. The test still fails. That is the part worth reading twice, because it is what separates this from hiding a flake: the suite exits non-zero, Chainsaw records a failure, and its whole `catch` runs — previous-instance container logs, the host snapshot, the data-plane capture, the event dump — every one of which is keyed on the test failing. The JUnit report says a test failed. What changes is one layer up: the wait writes a `SOFT-RED-node-join.txt` marker beside its diagnostics, and `hack/e2e-node-join-soft-red.sh` reads that marker to decide whether the job blocks. Softening inside the suite would have bought a green check by throwing all of that evidence away. Anything else the lane gates on a failed job needs the decision handed to it explicitly, since a softened run finishes green: the SSH breakpoint in `pull-requests` is that case today, and it still opens on a soft red for a PR carrying the `debug` or `release` label. The gate opens on one shape only — every failed test in the run carrying that marker — and blocks on everything else, including every way it can fail to answer: a report it cannot read, a failure it cannot name, a failure with no marker, a run whose report records no failure at all. A marker is written only when `timeout` reports 124 on that one wait, so an assertion after the join, or the wait failing to run, is an ordinary red. What the marker attributes is the TEST, not the failure, and the difference is worth knowing before reading a soft run as narrowly scoped. Chainsaw writes one `` per Test and combines the errors of that Test's steps into its single ``, so a second step error, or a failure of the automatic cleanup that runs with a live step report, rides soft alongside an expired deadline. A `catch`, `finally` or step-level `cleanup` operation that fails rides soft for a different reason, and it is worth stating separately because it looks like the same one: at v0.2.15 all three are run with the step report passed as nil, so no operation record is ever built for them and their error never reaches the `` — it fails the run without appearing in what the gate reads. A step report is not what goes missing: for a step-level `cleanup` it is the very one the automatic cleanup writes into, and for `catch` and `finally` it is the step's own; the nil is the argument, not the object. A Test that failed only through `finally` or step-level `cleanup` — `catch` runs only after something else already failed, so it cannot be the sole cause — renders as a testcase with no failure inside a testsuite counting none, so the gate sees a run that did not pass and a report recording nothing that did, and blocks. The granularity is accepted rather than worked around because the suite stops at the wait, so nothing the test exists to prove runs between the marker and the end of that Test. Only the lanes that block a merge ask the gate at all. `pull-requests` and `e2e-fork` do; `nightly` and `e2e-tag` run the same suites and stay hard, and `hack/node-join-soft-red_test.bats` holds that split in both directions. `e2e-tag` is the release candidate's own e2e check, where a soft pass would ship a release whose tenant clusters brought up no worker nodes. `nightly` is the only place an e2e failure reaches anyone who did not open a pull request — nothing in this tree notifies anywhere, so the red job is the notification. What is given up is real and worth stating: a regression whose only symptom is that workers never reach Ready no longer blocks a merge, and these two suites are the only ones in the tree that boot tenant worker nodes. The mechanism sees a timer, not a cause, so a deterministic node-join regression rides soft as readily as a slow runner does. The failed job also stops being the thing that opens a debugging path by itself, which is why the breakpoint is wired to the decision rather than to the job's result. That is a debt, not a design, and it has a stated way out. The owner is the node-join epic, cozystack/cozystack#3513. The route back to a hard gate is to calibrate the runner canary's thresholds against real lane data and then move the tolerance under `_COZY_CANARY_ALERT`, so the lane softens a run the canary measured as degraded and blocks one it measured as healthy. Until that calibration exists the canary cannot carry the decision, which is why it does not carry it today. Like the Cilium orphaned-endpoint watchdog below, this is a mitigation to remove rather than a convention to extend: a second assertion joining it is a decision to take in review, with its own reason for why a red there has stopped discriminating. The node-join deadline itself sits at 29m, and that figure is this test's own budget rather than a position in a sequence of product timers. Its clock starts where the wait starts — after the machinedeployment wait, the LoadBalancer assignment and the healthz probe, each with a ceiling of its own — while the chart's timers are already running by then: the autoscaler's `maxNodeProvisionTime` from the Machine's creation, the MachineHealthCheck's `nodeStartupTimeout` from the later of creation and `InfrastructureReady`, restarting at that transition. The offset between them is not fixed and no ordering against them is claimed. What the chart's 20m and 30m give is a scale to pick against: a budget wider than the first leaves room for a remediation cycle to fit inside the window, which is a possibility rather than a guarantee — a run can spend the whole window on one worker that never comes back. The price of that width is that a worker replaced mid-window takes its guest serial console with it; the replacement's console and the Machine history in the host snapshot survive. ### 2. Prefer declarative asserts over imperative waits A Chainsaw `assert` polls until the resource matches its expected shape or the per-operation timeout fires, so it subsumes both "the resource exists" and "the field/condition reached this value" in one operation — the old `until kubectl get …; do sleep 2; done` existence backstop followed by `kubectl wait --for=…` is no longer needed. ```yaml - assert: timeout: 6m resource: apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: postgres-test status: (conditions[?type == 'Ready']): - status: "True" ``` - Condition checks use the **filter-as-list** form `(conditions[?type == 'Ready'])`. Chainsaw v0.2.15 throws "field not found" on the indexed `(...)[0]` form. - Imperative checks that no Kubernetes API condition models — S3 reachability through a port-forward, an external LB HTTP path, MetalLB advertisement — stay in a `script` step. A bare `sleep N` inside such a script is allowed **only** when nothing better models the wait; annotate it with a `TODO(e2e-replace-fixed-timeouts):` comment. The bootstrap BATS (`hack/e2e-install-cozystack.bats`, `hack/e2e-chainsaw/_lib/run-kubernetes.sh`) carries the sanctioned `until kubectl get` exceptions. - A **measurement interval** is not one of those waits and does not take the marker. The rule above governs a `sleep` standing in for a condition nobody wrote a wait for, and the marker means "replace this once something models it". A gap between two readings of a cumulative counter models nothing and is the quantity itself: the diagnostics phase in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` reads the tenant worker CPU counters, the sandbox nodes' `/proc/stat` and QEMU's per-thread `/proc//task//stat` on either side of one shared `COZY_DIAG_RATE_INTERVAL` wait, because a single reading of a counter that accumulates from container start is an average over the whole uptime and no rate at all. There is no event to wait for instead, so a `TODO` there would promise a replacement that cannot exist. Carve-outs are named here rather than judged case by case, and this is the only measurement-interval one. ### 3. Let Chainsaw own cleanup; no test-level EXIT/RETURN traps Trap-based cleanup was the single biggest source of false failures in the BATS suite: an `EXIT` trap ran cleanup in a context where shell variables were unset, `set -u` killed it, and a **successful** test was marked failed — then retried twice into real failures. - Chainsaw deletes the resources it `apply`-ed during its cleanup phase (bounded by the `delete`/`cleanup` timeouts in `hack/e2e-chainsaw/.chainsaw.yaml`). Do not hand-roll teardown for resources Chainsaw created. - A self-contained `trap '… ' EXIT` **inside a single `script` step** — to kill a port-forward or remove a temp dir — is fine, because it runs in a contained subprocess with its variables in scope. See `hack/e2e-chainsaw/bucket/chainsaw-test.yaml`. What is banned is test-level trap-based cleanup of the BATS kind. The same carve-out holds inside a BATS `@test` when the trap sits in an explicit subshell — `( … trap "kill $pid" EXIT … )` — because a subshell trap does not replace the one the `bats` binary installs, so a failure inside it still prints its `not ok`. `hack/e2e-test-openapi.bats` relies on this to kill a backgrounded `kubectl proxy`; moving that cleanup to the end of the body would leak a process holding a fixed port rather than fix anything. - The ban extends to every BATS file under `hack/`, subdirectories and the `e2e-` prefixed ones included, for a second reason worth knowing before you debug one: an `EXIT` trap inside an `@test` body replaces the one the `bats` binary installs for its own bookkeeping, and a test that then **fails** prints no TAP line at all. It does not appear as `not ok`; it disappears, and the run ends with `# bats warning: Executed N instead of expected M tests` and a non-zero exit. Verified with Bats 1.14.0. Anyone reading the tail of the output, or grepping it for `not ok`, sees a green suite — and the CI runner `hack/cozytest.sh`, which is not the `bats` binary, reports the same failure correctly, so the two disagree exactly when it matters. Clean up at the end of the test body instead: both runners set `-e`, so the cleanup is unreachable on failure and the scratch directory is left behind for inspection, which is what you want from a failed test anyway. When a suite reports zero failures, confirm it also reports how many tests it ran: an exit code answers "did anything fail", never "did anything run". `hack/bats-no-exit-trap.bats` enforces this across every `hack/**/*.bats`, subdirectories included: a file that carries no `# EXIT-TRAP DEBT: N` comment must contain no EXIT-trap line at all, and a file that carries one must install exactly N — so a trap appearing or disappearing fails until the file's own number is corrected. Note what that does and does not buy: it is a ratchet on the number's *accuracy*, not on the debt itself, because adding a trap and raising `N` in the same change is green. Nothing mechanical stops the count growing — review does, which is the point of the number living in the file being reviewed. A trap inside an explicit subshell is exempt from the ban but still counted, so a declaration is not by itself an admission of debt; `hack/e2e-test-openapi.bats` is the current example and says so in its own header. The scan reads `.bats` files only, so a handler reaching a test body from a sourced `.sh` is outside it — `hack/e2e-chainsaw/_lib/run-kubernetes.sh` installs two, each benign for its own reason rather than by design: the one in `cozy_capture_tenant_talos` because that function is declared with `(` and runs in a subshell, and the one in `run_kubernetes_test` because no `@test` calls it despite being declared with `{`. Seven `hack/*.bats` source that library. Treat the guard as a ratchet over a common spelling, not as proof that a file installs no handler. A handful of files still declare a debt while their conversion waits on the branches that own them; when one of those reports zero failures, reconcile its `1..N` plan against its `ok` count before believing the run. ### 4. Do not mask cleanup or teardown failures - Resources a test creates that Chainsaw cannot reclaim — because a controller, not the test, owns them (e.g. the Velero `Backup`, `BackupStorageLocation`, and credentials secret in `cozy-velero` in the disabled `hack/e2e-chainsaw/backup/` suite) — must be pruned explicitly. Do not `|| true` over a stuck delete and leave stale state for the next run. - For nested tenants, tear down **child → parent with a hard wait for deletion between** each. Deleting the parent while a child is still uninstalling wedges the parent's cleanup Job on the child namespace, and both stuck uninstalls occupy helm-controller workers past the end of the test — starving whichever suite runs next. The kubernetes suites encode this ordering in `hack/e2e-chainsaw/_lib/run-kubernetes.sh`. ### 5. The install gate must have teeth The install test fails if **any** HelmRelease is not Ready. A toothless gate (backgrounded `kubectl wait` discarding child exit codes + a bare `echo` instead of `exit 1`) once shipped a permanently-failing platform HR through green CI for weeks. - Use a single `kubectl wait hr --all -A`, then an outcome-based re-list (covers HRs created after the snapshot), and `exit 1` on any non-Ready HR. - Dump the full Ready-condition message per non-Ready HR so the real error is in the test output. See `hack/e2e-install-cozystack.bats`. ### 6. Assert the parent HelmRelease did not remediate — via `status.history` A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is a silent race we want to catch. **Do not** check `.status.installFailures` / `.status.upgradeFailures`: Flux's `ClearFailures()` zeroes those on every successful reconcile, so checking them after the HR is Ready is **vacuous** and passes against a reverted fix. - Inspect `.status.history` instead — a `failed` or `uninstalled` Snapshot survives a later successful reconcile. Use the shared helper in `hack/e2e-chainsaw/_lib/remediation-guard.sh` (`helmrelease_has_remediation_cycle`) from a `script` step. ### 7. Test-Impact Analysis (TIA), default-on `hack/select-e2e.sh` walks the `packages/core/platform/sources/*.yaml` dependency graph and runs only the Chainsaw suites affected by a diff. A suite is a directory under `hack/e2e-chainsaw/` containing a `chainsaw-test.yaml`. - Conservative escalation: edits to `packages/library/`, `packages/core/`, the Go trees (`api/`, `cmd/`, `internal/`, `pkg/`), the codegen under `tools/`, `go.mod`/`go.sum`, top-level `hack/*.sh` helpers, the `hack/e2e-*.bats` the sandbox runs and the `hack/lib/` they source, `hack/*.mk`, `hack/buildkitd.toml`, `hack/e2e-*.yaml`, the shared `hack/e2e-chainsaw/_lib/`, the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or any workflow that runs the suite escalate to the **full suite**. Only `pull-requests.yaml` is executed from the pull request's own head, so escalating on the other three buys generic regression coverage rather than coverage of the edit; the trade and its measured cost are written next to `full_suite_pattern` rather than repeated here. A per-suite edit (any file under `hack/e2e-chainsaw//`) selects **only** that suite, with one exception: a `*.disabled` file selects nothing, because a parked suite is executed by nothing and editing it cannot regress a test. `hack/e2e-apps/.bats` — the pre-Chainsaw per-app BATS suites — maps the same way, off the basename instead of the directory. What remains in that directory is wired to nothing after the migration, and it is deliberately mapped rather than marked inert: inert would bake the orphan status into the rule and go quietly wrong the day a lane runs those files again. The basename is membership-tested against the suite list on the spot, and a name no suite carries escalates there and then, naming the file. Deferring that to the final intersection instead would leave the verdict to the rest of the diff — alone the selection empties and the backstop escalates, but beside any path that contributed a suite the escalation disappears and the run narrows to that suite in silence, which is the merge-before-escalate shape #3330 removed from the graph walk. Anything else under `hack/e2e-apps/` — a nested path, a non-`.bats` file — is shared material for those suites the way `hack/e2e-chainsaw/_lib/` is for the Chainsaw ones, and escalates for the same reason. - **Every path is classified, and an unclassified one escalates.** The selector has three selection outcomes — full suite, a scoped set, or nothing — and each is reached by an explicit rule; the fourth outcome in the bullet below is not a selection but a refusal to produce one. Several rules produce the empty one, which is worth knowing before reading an empty selection as "the inert list matched": documentation, dashboards and `*.md` are dropped first wherever they live, a `*.disabled` Chainsaw suite and a `examples/backups//` edit whose app has no suite are dropped where chainsaw and backup-example paths are matched, and `inert_config_pattern` covers repo meta, agent config, `hack/testdata/`, the helm-unittest fixture charts under `packages/tests/`, the three `.gitattributes` that hold nothing but `linguist-generated` markers, which reach no build, chart or test (enumerated rather than matched by filename, because the justification is what those files contain and the name does not carry it — `.gitattributes` can also set `filter`, `eol`, `working-tree-encoding` and `export-subst`, each of which changes what lands in the working tree and therefore what gets built, so a by-name rule would make such a file inert in silence) and the top-level `hack/*.bats` that are not `hack/e2e-*.bats`. That last one is the unit lane, and the root `Makefile` is the authority on the split: `BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats))` keeps 60 files for `make unit-tests` while `packages/core/testing`'s recipes run only the three it filters out, so the sandbox never executes one of the 60 and no Chainsaw suite can regress from an edit to one. Being inert here does not leave them untested: the `unit-tests` job is gated on the `plan` job's `code` output, which `pull-requests.yaml` computes as "any changed path outside `docs/`" and never from `select-e2e.sh`, so a bats-only pull request still runs the whole unit lane — plus install and the OpenAPI tests, since the `e2e` job also reads `code` rather than the selection. `full_suite_pattern` covers shared build inputs and the e2e harness, and the graph lookup handles `packages/(apps|system|extra)/`. A path matching none of them escalates to the full suite and logs why, because both lanes read an empty selection as "skip Chainsaw" and then post `E2E Tests` green — so a silent default is a green gate with nothing tested (#3392). When that escalation fires for something genuinely inert, add it to `inert_config_pattern`; do not relax the fall-through. The escalation list and the inert list are checked in that order, so a specific escalation beats a broad inert directory. - **Coverage is decided per changed path.** A path that resolves through the graph but reaches no runnable suite is covered by nothing, so it forces the full run on its own account, whatever the rest of the diff selected. Reading that off the merged selection instead let any other changed path contributing a suite name swallow the escalation, with nothing in the output to record that it happened: change a platform component, adjust the Chainsaw test beside it, and the component's escalation disappeared behind that one suite (#3330). The unit is the path rather than the source, because one path can belong to several PackageSources — `system/postgres-operator` belongs to two, one of which reaches no suite — and deciding per source would escalate on that half and run everything for every change to it. - **Two sources are excluded from the reverse-dependency walk as install-ordering hubs.** Every `*-application` source depends on `cozystack.cozystack-engine` so its `*-rd` HelmRelease waits for the ApplicationDefinition CRD, and the sources that depend on `cozystack.cozystack-basics` do so to get a namespace or a platform-wide policy in place first. Neither edge says the dependent's behaviour changes when the hub does, so carrying selection across them goes wrong in both directions: through the engine a `cert-manager` change fans out to every app in the tree, and through basics the platform's namespace-and-policy package narrows to whichever single suite happens to land downstream of it — 22 suites down to one on #3426, where the new site-router suite sits under `kubevirt-cdi`, which depends on basics. Dropping the reverse edges leaves both hubs reachable and stops them propagating; a change to either still runs everything, because its own group then reaches no suite and the per-path escalation fires on it. Adding a third one is a real decision, not bookkeeping: it says every existing edge into that source is ordering rather than behaviour, so read them before extending the list. - **There is a fourth outcome, and it is a failed step rather than a selection.** It fires when the suite list itself is unavailable: `find` cannot list `hack/e2e-chainsaw/`, or that directory holds no `chainsaw-test.yaml`. There is no fallback left to take, because escalating would print an empty list and both lanes read an empty selection as "skip Chainsaw" before posting `E2E Tests` green — the outcome meaning "run everything" would deliver a run of nothing. So the selector names what broke on stderr and exits non-zero; both lanes run the step under `bash -e`, so the job fails and the required check goes red. A red `Select E2E tests` step is a real signal, not a flake: read its stderr line before rerunning. A `yq` failure is deliberately not in this class — what is missing there is the dependency graph, not the suite list, so the selector still has a safe answer and escalates to the full suite. That is conservative but quiet: the only sign is one stderr line inside a step that passes, so a selector running blind on every PR looks exactly like a project that escalates a lot. - **Every escalation names its cause on stderr.** "Run everything" is the same list of suite names whichever rule produced it, so a reason line is the only thing that distinguishes the branches that can reach it, and without one the answer to "why did this pull request run the whole suite" has to be re-derived by hand from the diff. Four of them used to be silent — the shared `hack/e2e-chainsaw/_lib/` and `.chainsaw.yaml` case, the `full_suite_pattern` match, a `packages/` path no PackageSource claims, and the empty-selection backstop — and `full_suite_pattern` is the commonest cause of a full run by a wide margin, so the usual answer was the one the log never carried. Each line names the path or the sources that caused it, and a unit test asserts each message, because a reason that regresses to silence changes no selection and is invisible to every other test. The lines go to **stderr** and must stay there: stdout is the suite list and both lanes parse it, so a reason line on stdout would be read as a suite name. - **The one path the fall-through cannot catch is a new e2e workflow.** `inert_config_pattern` makes `.github/` inert as a directory, and the workflows that run the suite are exempted from it by name in `full_suite_pattern`. A workflow added under `.github/workflows/` therefore reads as inert until someone adds it to that enumeration, and the fall-through never sees it. That is why the list carries a `rg -l test-chainsaw .github/workflows/` reminder: the enumeration is the guard, not a convenience. - The `full-e2e` PR label forces the whole suite. - The selection is passed to `make test-chainsaw CHAINSAW_SUITES=""`, which runs `chainsaw test ` from `hack/e2e-chainsaw/` (empty = the whole suite). - The coverage gap TIA opens on PRs is closed at release-cut time: the "Prepare release" commit bakes image digests into `packages/core/` (and other packages), which TIA escalates to the **full** suite — so the release PR exercises the whole suite against the commit the tag will point at. When adding a new app package, confirm `select-e2e.sh` maps it correctly (it has a unit test, `hack/select-e2e_test.bats`). A new app whose suite directory does not yet exist will not be selected — add `hack/e2e-chainsaw//chainsaw-test.yaml` first. `src_to_suites()` maps a PackageSource name to the suites it owns and is the inverse of `select-install.sh`'s `suite_to_source()`. A test walks every suite through both tables, so a source whose name fits neither convention — not `-application`, not the suite name itself — fails there rather than quietly making its own suite unreachable, which turns every change to that package into a full run (#3665). #### What TIA does and does not do Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs," and the wiring has consequences worth knowing before you rely on it or change it. - **It trims only which suites `chainsaw test` runs, not the expensive stages.** `Prepare environment` and `Install Cozystack` (the full platform install) run regardless of the selection. TIA only decides the `CHAINSAW_SUITES` passed to the final E2E step. A perfect narrow saves the suite tail, not the bulk of wall-time. (Package image builds are scoped separately — by `hack/build-matrix.sh`, on the PR diff — while the Talos and installer/finalize legs still run for non-docs PRs; that is a distinct mechanism from TIA's suite trimming.) - **`Select E2E tests` runs *after* `Install Cozystack` and has no `always()`.** GitHub implicitly ANDs a step's `if:` with `success()`, so if install fails the selector (and `Run E2E tests`) are skipped entirely. On a red install you will not see TIA narrow anything — that is the ordering, not a selector bug. - **Most PRs legitimately escalate to the full suite.** Platform manifests live under `packages/core/`, Go under `api/`/`internal/`, helpers under `hack/`, plus the `Makefile` — touching any one triggers the full suite. So "all suites selected" is usually correct, not a failure to narrow. - **Two skip layers, with a gap.** The `plan` job computes a coarse docs-only gate (`code`, true for any change outside `docs/**`) that skips the whole build+E2E pipeline for docs-only PRs, plus a per-package build matrix (`hack/build-matrix.sh`) that scopes which images build. TIA's finer skip (`select-e2e.sh` ignoring `docs/`, `dashboards/`, `*.md` and the rest of `inert_config_pattern`) only trims the suite selection. A PR touching only `dashboards/*.json` therefore still runs install, then skips just the Chainsaw suites — near-zero savings. If you need such PRs to skip the pipeline, widen the `plan` docs-gate, not `select-e2e.sh`. ### 8. Failure must be self-explanatory — diagnostics are first-class - On install failure, the CI step dumps `kubectl get hr -A -o wide`, a `describe` of each non-Ready HR, and sorted events, all under collapsible `::group::` blocks. - On E2E failure, the CI step dumps `kubectl get hr -A`, sorted events, and COSI bucket/claim/access readiness columns (those CRDs ship no printer columns, so plain `get` shows only NAME/AGE). The `chainsaw test` run is one step, so this dump is at run level; Chainsaw's own per-`Test` `catch` blocks add scoped events/describe/podLogs for the failing suite. - Inside a suite, attach scoped diagnostics with a top-level `catch:` block (`events`, `describe`, `podLogs`, `get`) rather than letting a bare assert failure through — see `hack/e2e-chainsaw/bucket/chainsaw-test.yaml` and `hack/e2e-chainsaw/harbor/chainsaw-test.yaml`. This replaces the BATS `... || { echo …; kubectl get/describe …; false; }` idiom. - `make collect-report` → `hack/cozyreport.sh` uploads a full `cozyreport.tgz` (operator/Flux/cert-manager logs, LINSTOR, Kamaji, Talos dmesg, COSI YAML) on every run, and the Chainsaw JUnit report (`chainsaw-report.xml`) is uploaded as its own artifact. - **Which machine a run landed on is recorded before the sandbox container exists**, in `sandbox-host/runner-identity.txt`: the OCI shape, both region keys, the availability and fault domain, the numbers of the `shapeConfig` sizing object, and the runner VM's own kernel release, online CPU count and processor model. A lane asks for a runner class rather than for a machine — a flex shape is filled from whatever capacity the pool has, and the region is a preference with a fallback — so without this a red run and a green one cannot be compared by the hardware they ran on, which is the first question a flake that only some runs have will raise. `hack/e2e-runner-identity.sh` takes the read on the runner VM, from `packages/core/testing/Makefile`'s `apply` target, because the metadata service answers on a link-local address the sandbox container has no guaranteed route to, and an "unreachable" answered from in there would look exactly like a capture that had tried. Every e2e lane reaches the sandbox through `make prepare-env`, so the record is taken once per run in each of them, on the passing path as well as the failing one. It goes to the job log as well as to the file, and the log copy is not a convenience: a run whose sandbox never came up has no `cozyreport.tgz` at all, so on that one path the log is the only place the record exists. - **The keys are fetched one at a time and the root instance document never is.** That document carries the instance and compartment OCIDs, the image OCID, the display name, the hostname and the user-supplied metadata bag, and the report is a public artifact — so asking per key makes the privacy property something the wire settles rather than something a filter has to be trusted for. It also removes a guess: Oracle documents the shape sizing under its own `shapeConfig` subpath and the root document does not carry that object at all, so a reader hunting it inside the root document would have reported the size of every machine as unknown and blamed the service for it. From the sizing object only the numbers are kept, and only under keys shaped like field names, which makes the rule hold for keys nobody has seen yet: a size is a number, an identifier is a string, and a key is printed into the record exactly as a value is. That object is also refused whole unless it arrives as one complete JSON object, so a truncated answer or a top-level array cannot be read in part and published as a sizing record. A per-key answer that is not one scalar is named as such rather than printed, since a service answering a whole document to a key request is the one way the design could still publish what it never asked for. The first key doubles as the probe and is held to answering like this service does, so a machine that is not an OCI instance costs one ceiling rather than one per key whether the address is silent or something else is answering on it, and either way the run leaves an `imds unreachable` line to grep for. Every read is taken through a wall-clock ceiling, because a diagnostic that hangs costs the step its whole budget without ever reaching an exit status; where `timeout` is not on PATH the local reads run unbounded instead of not at all and the record says so in a `[bounds]` line, the same trade `hack/cozyreport.sh` makes for its own reads. None of it is fatal, since this runs inside the one retried step in the job — a runner that is not an OCI instance leaves an `imds unreachable` line and an exit status of 0. The report writes that filename either way, holding the record or the reason there is none, because an unidentified runner and an unremarkable one would otherwise leave the same absence. - **The report's per-pod evidence, under `kubernetes/pods/` in `cozyreport.tgz`, is selected by the READY column rather than by STATUS** — so a pod left at `0/1 Running` by a readiness probe that never passes is in scope, where before it was not. In scope is not the same as collected: the walk stops at `COZY_REPORT_PODS_MAX` pods or `COZY_REPORT_PODS_BUDGET` seconds, and what it did not reach is named in the artifact rather than dropped in silence, as the next point describes. Selecting on the STATUS column alone drops exactly that pod, and it is a common way for a suite to time out: the replica never joins, its HelmRelease sits in `InProgress`, and the artifact carries no trace of the pod the run died on. Terminal `Succeeded`/`Completed` pods stay excluded, since they legitimately end unready. Widening the selection and capping the walk are only safe together if the widening cannot displace what the uncapped walk always kept, so the collection order splits by evidence class before namespace: every pod that is not `Running` — a crash loop, `Pending`, `Error`, the class the collector took before this — is ordered ahead of every pod that is `Running` but unready, and `COZY_REPORT_PREFER_NS` orders within each of those two classes rather than across the whole selection. Otherwise a failed install with more not-Ready pods than the cap spends it on newly-in-scope tenant pods and drops the container logs of the platform pod that is actually crashing. - **Every bound that the report's pod collection hits is named in the artifact, not only in the job log.** The pod walk stops at `COZY_REPORT_PODS_MAX` pods or `COZY_REPORT_PODS_BUDGET` seconds and leaves `COLLECTION-TRUNCATED.txt` naming whichever fired; a pod it cut off part way through leaves `CONTAINER-LOGS-TRUNCATED.txt`; a pod list that never came back leaves `COLLECTION-FAILED.txt` rather than the empty tree a healthy cluster produces; a missing local `timeout` leaves `COLLECTION-UNBOUNDED.txt`, because the reads still run, just without a ceiling, and a missing dependency on the machine writing the report is not a statement about the cluster; a kind this cluster does not serve at all — `kubectl get vm -A` on an install without KubeVirt, which exits non-zero — leaves `KIND-NOT-SERVED.txt` rather than a failure marker, since that is the cluster answering correctly and a marker that fires on healthy installs is one a reader learns to skip; and a knob value the collector could not use is named too, appended to `COLLECTION-TRUNCATED.txt` when there is one, written to `COLLECTION-NOTES.txt` when there is not, and appended to `COLLECTION-FAILED.txt` instead when the pod list itself never returned, so the note always sits beside the failure it accompanies rather than in a file the reader has no reason to open, and a bound that turns out to be the default is not mistaken for the bound that was asked for. A pod whose container list came back as a prefix — kubectl named some containers and then failed — leaves `CONTAINER-LIST-INCOMPLETE.txt` beside the logs it did manage to read, because the containers it never named have no file at all and their absence would otherwise read as containers that were silent. A read that succeeded while kubectl also wrote to stderr — a deprecation notice, a partial-result warning — leaves that message in `READ-WARNINGS.txt` beside the object rather than inside it, since `pod.yaml` and the logs are machine-read and a warning line in them costs the parser the file, while dropping it altogether would be the silent loss of evidence this tree exists to prevent. A pod directory whose containers produced any log at all also carries `CONTAINER-LOGS-BOUND.txt`, naming the `--tail` bound those logs were written under (a pod whose every container was silent has no bounded log to describe, so the file is absent), because `--tail` drops the oldest lines and nothing in a log says so; it sits beside the logs rather than inside them, since a log read in full is a complete artifact and the platform's controllers log JSON per line, so a trailing prose line would break every parser that could read the file before. A pod whose container list could not be read leaves `logs-UNAVAILABLE.txt` in its own directory, carrying kubectl's reason, because a pod directory holding `pod.yaml` and `describe.txt` and no logs at all otherwise reads as a pod whose containers were silent. The VM, VMI and services loops carry the same pair of bounds as constants rather than knobs, so a failed install that leaves every LoadBalancer without an address collects the first of them and names the rest instead of running the report step past its own `timeout-minutes: 30`, which would lose the tarball rather than truncate it, since the collector writes it on its last line. That step ceiling is the outer backstop for a read with no bound of its own; it does not save this report, it saves the steps after it, the other uploads included. It carries `continue-on-error: true` alongside, because the step's `|| true` is its standing statement that collecting diagnostics is never fatal and a step killed at its ceiling never reaches that `|| true` — without it a passing run whose collection ran long would be published as a failed one, and every later `failure()` in the job would start meaning "the tests failed or the report was slow". That tolerance also hides the overrun, since the step then renders green, so a following step turns the collect step's `outcome` into a `::warning::` annotation — a bound that fires is recorded, and the outer bound is not exempt from the rule the inner ones follow. The LINSTOR reads that run a command inside a pod carry a ceiling too — they are the only reads whose duration a workload sets rather than the apiserver, and a wedged controller holds the stream open with nothing on either end to close it — and they record a cutoff in `COLLECTION-FAILED.txt` beside their output, which for the two that stream a gzipped tarball is the only place it can go, since a line of prose inside one is an archive nothing opens; the four that produce a table carry the marker in the file as well. A LINSTOR table that got nothing at all — the wedged controller killed before it wrote a byte — says so in the file rather than shipping at zero length, and a read whose stderr could not be captured says that too instead of reporting silence. A LINSTOR table cut short carries the marker inside itself whether the clock or the command ended it, in the same `# [cozyreport]` form every other in-file marker uses so one anchored grep finds them all, and the note about a satellite pod list that came back short lands under `linstor/error-reports/` beside the bundles rather than in the shared Kubernetes file, like every other `.txt` in the tarball, because a short node list whose explanation sits one directory up reads as a small cluster; the two archives cannot take one, since a line of prose inside a gzip stream is a file nothing opens. A cut-off bundle goes one of two ways and the note names both: a stream that got members across is kept and truncated, so a short member list there is the cutoff rather than a satellite with few reports, while a stream killed before its first byte leaves zero bytes and the emptiness check at the call site removes it. The reader runs before that check, so it says which two outcomes are possible instead of asserting one. The four LINSTOR reads whose output is a table keep the remote command's whole message in the file itself when the command produced no table of its own — a partial table plus an error keeps the message beside the file instead, so the table stays a table — because `kubectl exec` appends its own `command terminated with exit code N` last — so the last line there is the one line that never carries the cause, and a listing sent away from its own error is a zero-byte file saying nothing. That note states what the status leaves open instead of resolving it: `kubectl exec` returns the remote command's own exit status, so a 124 there is either the collector's deadline firing or the command itself exiting 124, and nothing separates the two. Every other read in the collector carries a ceiling too, gates included — an `if kubectl get deploy …` that decides whether a module runs is a read, and an unbounded one ends the whole report before the module it guards. A gate cut off short is recorded rather than treated as an answer, because skipping the module produces the same empty tree as a cluster that never had the component; a gate that answers "not installed" is recorded nowhere, since that is the cluster answering correctly and a marker firing on every install without KubeVirt is one a reader learns to skip. What a per-read ceiling does **not** give you is a ceiling on the report: the pod, VM, VMI and services walks carry a count cap and a time budget, and every other per-object walk in the collector carries neither, so their cost is the cluster's to set and a failed install with a hundred not-Ready HelmReleases spends four hundred bounded reads there, four per release. Described as "the walks outside those four" rather than listed by name, because a list of them drifts and reads as complete while it does so — which is the defect this same change removed from the collector's own header. The step's `timeout-minutes` is what bounds that, not the per-read wrapper. Whichever bound stopped a walk says so in `COLLECTION-TRUNCATED.txt`, count and clock alike, because objects removed by the clock while the count note stood alone would read as a cap that never fired. The read that feeds each loop is bounded too: the budget cannot start until rows arrive, so an apiserver hanging on the cluster-wide list would otherwise spend the whole collection step with the deadline never consulted. The budget is a deadline for starting a read rather than for finishing the walk, and it is rechecked between containers but not between a container's current and previous log, so the ceiling is the budget plus one pod's object reads plus both log reads of one container, which is the price of never leaving a half-written file behind. - **`COZY_REPORT_POD_TAIL` bounds each container log** at 2000 lines by default, and a log that was read successfully records which bound it was written under, since `--tail` drops the oldest lines and a file that begins mid-stream looks exactly like a complete one; `-1` restores kubectl's own default of the whole log. `COZY_REPORT_PREFER_NS` (default `tenant-`) is collected first, so a cap that only bites on a broadly degraded cluster is not spent on platform namespaces while the pod the suite died on falls off the end. - **Capture the PREVIOUS container instance, not just the current one.** For a crash-looping pod `kubectl logs` (and the Chainsaw `podLogs` collector, which in v0.2.15 has no `previous` option — it does accept `name`/`namespace`/`selector`/`container`/`tail`/`timeout`/`cluster`/`clusters`) shows the Nth restart — a fresh replay of the same startup path — while the evidence that explains the failure sits in the immediately preceding instance, reachable only via `--previous` and lost for good once the kubelet garbage-collects that container. `hack/e2e-capture-previous-logs.sh` closes this once for every suite from the global `error.catch` (and from `hack/cozytest.sh` for the e2e BATS suites) rather than per-suite: it discovers containers with `restartCount > 0` at runtime — init containers included, since an interrupted bootstrap lives there — orders the app suites' `tenant-test` namespace first (a best-effort ordering only — it never restricts the capture, and for a suite whose pods live in a child tenant namespace it simply degrades to input order), and dumps each to the job log and to `snapshots//previous-logs/`. Containers that never restarted are skipped, so a healthy run emits one line, not a page of "previous terminated container not found". Every way the capture can come up short is named rather than inferred: the `COZY_PREVLOG_MAX` overflow, a malformed cap, a `COZY_PREVLOG_TAIL` that was rejected and fell back to 200 (a zero or leading-zero tail asks kubectl for nothing, which would otherwise be reported as the container having logged nothing), the tail bound each kept log was written under — named in `capture-notes.txt` beside the logs rather than inside them, since `--tail` drops the oldest lines and nothing in the output says so while a log read in full is a complete artifact that a trailing prose line would stop any parser from reading, and suppressed on `-1` because no bound was applied then — a pod list that never returned (which is NOT the same as "nothing restarted" and must not be reported as such), a read that timed out versus one kubectl itself refused (kubelet GC, RBAC, an unreachable node — different problems that a blanket "kubelet GC" line would hide), a previous instance that produced no output, and a capture the outer backstop cut short. A partial log is kept and marked truncated, never deleted — it is the only copy of something already gone from the cluster. Each of those lines the script itself emits is written to `snapshots//previous-logs/capture-notes.txt` as well as to the job log, and the directory is created before the first of them, so a reader who has only the uploaded artifact can still tell a capture that found nothing from one that never ran. The outer backstop firing is the exception: that line comes from the callers, after the script has already been killed, and reaches the job log only. `hack/cozytest.sh` arms this capture, and the other two legs, only for suites whose own filename or whose parent directory carries the `e2e-` prefix, because all three read cluster state and a unit runner has `kubectl` on PATH with no cluster behind it, so a red unit test used to walk into the kubectl-gated legs on its way out, and into the crust-gather one as well on any machine where that binary happened to be installed. - **The data-plane collector is the third capture leg, and the only one that reads forwarding state off the node.** `hack/e2e-capture-dataplane.sh` runs after the crust-gather snapshot, from `hack/cozytest.sh` for the e2e BATS suites and from the global `error.catch` in `hack/e2e-chainsaw/.chainsaw.yaml` for the Chainsaw ones, and it collects what an object-state snapshot structurally cannot: the cilium endpoint and BPF conntrack view from an affected pod's node, the kernel conntrack table that is the authoritative one for the host-to-local-pod path under `enable-host-legacy-routing`, and the LoadBalancer datapath. Its two callers bound it differently, 600s from `cozytest.sh` and 300s from the Chainsaw catch, so anything added to what it collects has to fit the smaller of the two rather than the one you happened to test under. It holds the same notes contract as the previous-logs collector, with one stated exception: every read carries its own ceiling, and why a read produced nothing goes to `capture-notes.txt` beside the capture rather than only to the job log, because an empty result and a read that never answered are the same bytes on disk and only the note tells them apart; the two EndpointSlice reads in the LoadBalancer path are the exception and report nothing either way, so a `` backend line there does not distinguish an endpointless Service from a read that never answered. - **A catch script needs an explicit `timeout:` that exceeds the inner budgets it contains.** A Chainsaw `script` op is bounded by `timeouts.exec` (5m in `.chainsaw.yaml`) unless it overrides it, so a catch whose own `timeout -k …` wrappers sum past that is *structurally guaranteed* to be killed — it reports `SCRIPT | ERROR | signal: killed` and yields nothing. That kill is strictly worse than an inner `timeout` firing: SIGKILL takes out the process group, so the collector dies mid-write **and** every later step in the script is skipped, whereas an inner `timeout` exits 124, leaves a partial capture on disk, and lets the rest of the catch run. Size the op timeout above the sum of the inner budgets and let the inner ones be what fires. - **When the sum cannot fit the op, bound the collection phase as a phase rather than raising the op.** The rule above assumes the inner budgets add up to something an op timeout can sit above. On a failure path that has accumulated collectors it stops being true — the tenant-Kubernetes node-join path in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` sums past what is left of its 67m op once the bringup ahead of it and the snapshot behind it are counted, and several of its reads carry only `--request-timeout`, which bounds one HTTP request rather than a client retrying against a wedged apiserver, so parts of it have no ceiling to add up in the first place. Raising the op ceiling does not fix that shape, it postpones it: a larger ceiling multiplied by a sum that grows with every collector added is the same problem one collector later. What works instead is a wall-clock budget for the whole phase (`COZY_DIAG_PHASE_BUDGET`, with `cozy_diag_phase_start` / `cozy_diag_phase_has_time`): once it is spent, the collectors that have not started are declined out loud instead of attempted, so whatever must run after the phase — the tenant crust-gather snapshot, in that case — is never queued behind work begun past the deadline. Two consequences worth carrying to the next collector. The budget bounds when a collector may *start*, not when the phase ends, so it is not a worst case and the comment next to it must not be written as one. And because what runs last is what gets declined, source order becomes load-bearing: put the cheap reads that discriminate between failure modes ahead of the collectors that cost minutes, or the budget will spend itself on the expensive ones and drop the evidence the path exists to produce. - **Set `crust-gather --duration` explicitly.** It is crust-gather's own budget for the collection phase and it defaults to **60s**; on elapse it abandons the collection and skips its finish step, leaving a partial tree with nothing to say the rest is missing. The outer wall-clock must exceed it, because the API discovery crust-gather runs *first* is not covered by that budget at all — with an unhealthy aggregated APIService (the state a failed e2e is usually in) aggregated discovery times out and the fallback re-walks every group, which can consume the entire outer timeout before a single object is written. Do not diagnose a killed snapshot by bumping the outer number alone; the two budgets bound different phases. Send the collector's output to a log inside the snapshot rather than `/dev/null` so "complete or truncated?" is answerable from the artifact. - **A collector that only runs on failure answers in one direction.** Its output has no measured value to be read against, so a claim like "this boot was slow" rests on an impression of what a healthy run looks like rather than on a number from one. Where the evidence is destroyed by teardown — the tenant worker's guest serial console goes with the Pod that holds the stream — the baseline has to be taken on the passing run or it does not exist at all, which is why `hack/e2e-chainsaw/_lib/run-kubernetes.sh` captures that console on both paths and records which one it was. A passing-path collector is held to two rules the failure path does not need. It must not fail the suite, since everything the suite exists to prove has already passed by the time it runs — and its own cost counts, not just its exit status, so a collector that costs minutes declines itself out loud when the operation's ceiling no longer has room for what it is about to spend, rather than starting a walk that the ceiling cuts off mid-write. That bounds the collector's contribution and nothing else: work already on the path with no ceiling of its own stays unbounded, and no reserve makes it fit by arithmetic. And what it could not collect **at all** has to reach the **job log**, not only the report, because a green run's artifact is the one nobody downloads. Per-read outcomes stay beside the read in the report, as everywhere else in that collector: a note about one console is only usable next to that console, while a walk that collected nothing is a fact about the run. - **A diagnostic that carries a relative clock also has to carry the absolute one.** A guest kernel stamps its console in seconds since its own boot, so a log that ends at 500 seconds says how far that guest got and nothing about when. Whether it stopped early because it started late or because it stopped printing is decided by the instant the read happened and the instant the guest started, and neither is in the file: the first survives only as the artifact's mtime, which is metadata rather than content and so is invisible to anyone reading the log and lost by anything that regenerates it, and the second only in a `describe` dump that a healthy-looking capture never triggers. Write both beside the capture, and name the units on each — an epoch integer and an RFC 3339 string subtract to nonsense if a reader takes them as printed. - **Per-failed-test crust-gather snapshots** are the richest diagnostic — a `crust-gather serve`-able archive of the cluster *at the moment of failure*, before cleanup. The global `error.catch` in `hack/e2e-chainsaw/.chainsaw.yaml` captures the **host** cluster into `_out/cozyreport/snapshots//host` on any failure (the Chainsaw analog of the `hack/cozytest.sh` EXIT-trap snapshot used by the e2e BATS suites); the kubernetes suites additionally snapshot their **tenant** cluster from `hack/e2e-chainsaw/_lib/run-kubernetes.sh` while the tenant API LB is still routable. `hack/cozyreport.sh` folds `snapshots/` into the artifact. Do not remove the global catch — without it the Chainsaw suite uploads a report with no per-test cluster state. ### 9. Keep the test environment deterministic - **Pre-pull** every kubeovn / linstor / cert-manager image onto all nodes before those HelmReleases install (`hack/e2e-prepull-images.sh`), so clustered workloads (OVN raft, LINSTOR, the cert-manager webhook) do not fail on per-node image-pull stagger. - **Exclude loop devices** from host LVM scanning in the Talos machine config so the host does not activate volume groups inside loop-mounted e2e disk images. - **Fail fast on node readiness** (≈5m, then bail) rather than marching into LB/NFS tests that will also fail — it saves several minutes per attempt and keeps the real failure at the top of the log. - **Give the sandbox a runner class strictly larger than its guests.** `hack/e2e-prepare-cluster.bats` boots the management nodes with fixed `-smp` and `-m`, and those resources are committed for the whole run, so a lane whose runner only just covers them leaves the host nothing for QEMU's own threads, containerd and the runner agent. A new e2e lane picks its `runs-on` accordingly; `hack/sandbox-runner-headroom.bats` derives both sides and fails the unit suite when any lane loses that margin. ### 10. The E2E workflow split: same-repo vs fork, and the "E2E Tests" commit status A fork's `pull_request` run carries no registry credentials, so it cannot push the images the e2e sandbox pulls. E2E therefore runs across **two** workflows: - **`.github/workflows/pull-requests.yaml`** runs e2e in-tree for **same-repo** PRs (the `e2e` job, named `E2E (in-tree)`). Fork PRs skip this job. - **`.github/workflows/e2e-fork.yaml`** runs e2e for **fork** PRs via a privileged `workflow_run` that fires after the fork's Pull Request run completes. It pushes the fork-built images to the registry and runs the suite — the same Chainsaw + TIA path as in-tree (§7) — in an ephemeral sandbox that itself has no push credentials. Consequences to keep in mind when touching either workflow: - **Fork builds export, they do not push (`OCI_EXPORT_DIR`).** The fork's unprivileged build sets `OCI_EXPORT_DIR` (see `hack/common-envs.mk`) so every image is written to a per-image OCI archive artifact instead of being pushed; the privileged `e2e-fork.yaml` later pushes those archives to the registry **by digest**. Any build recipe that shells out to push directly — e.g. a `skopeo copy … docker://…` that bypasses the `image-tags`/`oci-output` macros — must honour `OCI_EXPORT_DIR` too, or a fork build dies with an anonymous-push `denied` (#3257). This is covered by `hack/common-envs_test.bats`. - **"E2E Tests" is a commit status — not a job, and not a check-run.** The required merge-gate context named `E2E Tests` is a GitHub **commit status**, posted by the `plan`/`e2e-report` pair for same-repo PRs and by `e2e-fork.yaml`'s `resolve`/`report` pair for forks — **not** the `e2e` job itself. Each lane opens the context with `pending` and concludes it when the suite is done, and the opener is not optional: a commit status never expires, so a head SHA that already went green would otherwise keep reading green from the moment a re-run, a reopen or a `full-e2e` label starts a new run until that run reports, leaving the pull request mergeable across a suite that may be about to fail. While the gate was a job, a fresh non-success check run appeared automatically and closed that window. The two lanes close it at different speeds, and the fork lane does not close it entirely. Same-repo, the opener is `plan`'s first step, so the window lasts seconds. On a fork the opener is `resolve`, which cannot run until the unprivileged build has finished, so re-running an already-green fork head leaves the old green readable for the length of that build. The gap is structural rather than an oversight: a fork's `pull_request` token is read-only and cannot write a status at all, which is the same property the rest of this design relies on. It is bounded by the build and it is the one case where "the opener closes the window" is not literally true. Two reasons it is a status: - *Not a job:* a required *job* that is skipped (as the `e2e` job is for forks) counts as passed by branch protection — exactly how fork PRs used to merge with e2e never run (#3257). A status is `success` only when the report step posts it; a never-posted required status stays "Expected" and blocks the merge. If you rename the `e2e` job, do **not** name it `E2E Tests` and do not wire branch protection to a job name — the status must stay the gate. - *Not a check-run:* a check-run created with the default `GITHUB_TOKEN` floats into an arbitrary GitHub Actions check-suite (in practice the earliest-registered one, a tiny labeler — that is why `E2E Tests` once rendered under "PR size label"), and an app-token check-run would need the COZYSTACK_CI app granted `checks: write`. A commit status has neither problem: branch protection matches it by context name and it carries no check-suite grouping. It is safe to post with the default token because a fork's `pull_request` token is **read-only** and cannot post a status — so only same-repo runs (`plan`'s opener and `e2e-report`, both guarded `!fork`) and the privileged `workflow_run` in `e2e-fork.yaml` can set it. Each publishing job carries `permissions: statuses: write` and calls `repos.createCommitStatus`. `plan`'s opener is additionally `continue-on-error`, since every build and test job hangs off `plan` and a transient status-API failure there must not discard the run; losing it only restores the stale-green window for that one run. **Consequence for branch protection:** require the `E2E Tests` **status context** (no GitHub App and no app-id pin needed). - **Fork TIA reads trusted inputs, but the fork still writes the selector.** `e2e-fork.yaml`'s `e2e` job runs the same `select-e2e.sh` TIA as in-tree (§7), and its inputs are hardened: the changed-file list and the `full-e2e` label come from `resolve`'s base-repo API lookups (`pulls.listFiles` / the PR's labels), the same trusted source as the docs-only decision, passed in as outputs (`changed_b64`, `full_e2e`) rather than re-derived from a `git diff` of the fork-controlled merge ref. Keep that part as it is: re-deriving the selection from the checked-out tree would add a second bypass for no gain. - **What the fork lane is, and is not.** The `e2e` job runs from the fork's tree, so `select-e2e.sh`, the suites under `hack/e2e-chainsaw/`, `hack/e2e-install-cozystack.bats` and `packages/core/testing`'s recipes are all fork-authored. A fork that wants its own suite to report green can edit any of them, and hardening the TIA inputs does not change that. Read the lane accordingly: it is a correctness gate that stops an honest contributor shipping a regression, not a boundary against a hostile one. What contains a hostile fork is that no fork run executes without maintainer approval (below), that the job holds no push credential, and that a diff editing the test harness is visible in review. - **A fork PR that adds a new image cannot go green until that image exists on the base branch.** `publish` derives the set of archive names it will accept from the base parent (`HEAD^1`), so that a fork cannot mint arbitrary repositories under the CI registry by naming an artifact. An image the PR itself introduces is therefore not in that set, and the job fails with `image '' is not in the base-tree allowlist`. This is fail-closed by design and there is no label override — land the image on the base branch first, or run the suite from a same-repo branch. If a fork e2e run fails on that line, the contribution is not broken; the allowlist is doing its job. - **`labeled` events run in their own concurrency group, and a cancelled run reports nothing.** Because the gate is a status rather than a job, a run that dies without posting leaves the head SHA at "Expected" — which blocks the merge rather than passing it. Several separate pieces implement it, and weakening any one of them reopens the gap. Each lane's terminal reporter runs under `!cancelled()` rather than `always()`, so a run that was cancelled publishes nothing at all and the `pending` it opened stands until the run that superseded it reports. Each lane's concurrency key then keeps a run that is going to publish nothing out of the group of the run that is: `pull-requests.yaml`'s appends `-label` for a `labeled` event that `plan` discards, so labelling an open PR cannot cancel a live `opened`/`synchronize` run (the replacement run would never post the status the cancelled run was going to) — the condition is the complement of `plan`'s guard, so a `full-e2e` label, which does publish, stays in the main group and supersedes the head run instead of racing it, `e2e-fork.yaml`'s concurrency key appends the triggering run's id when that run concluded `cancelled` or `skipped` so a privileged run that is going to post nothing cannot cancel the one that is going to post, and `e2e-fork.yaml`'s `resolve` returns early on a `cancelled`/`skipped` triggering run instead of posting `failure` for it. If you touch any of them, keep the invariant: exactly one run per head SHA is responsible for the status, and a run that is not that one must post nothing at all. **Security prerequisite (repo setting).** The split is only safe if fork runs require maintainer approval before they execute. The repository must keep **Settings → Actions → "Require approval for all outside collaborators"** enabled, so a fork's `pull_request` run — which produces the artifacts `e2e-fork.yaml` consumes — cannot run untrusted code (or feed the privileged run) without a maintainer's go-ahead. ## Chainsaw v0.2.15 gotchas The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026); none of the traps below are fixed upstream yet, so the workarounds stay until a bump is possible. Each one has already cost a debugging session — this is the "no one gets it right the first time" list, worth a read before writing a new assert. - **Condition asserts must use the filter-as-list form.** `(conditions[?type == 'Ready'])` evaluates to a *list*; assert against it with a list body (`- status: "True"`). The indexed `(conditions[?type == 'Ready'])[0]` form throws `field not found` (also enforced by convention 2). - **No number literals / backticks in JMESPath.** `` ports[?port == `443`] `` mis-evaluates in v0.2.15 — the numeric comparison silently returns the wrong set. Assert ports (and any numeric field) with the exact-ordered projection form instead: `(ports[*].port): [80, 443]`. `sort()` and `contains(list, )` are likewise broken with numeric arguments. - **`error:` fails hard on an unknown GVK.** Unlike the BATS `! kubectl get ` idiom (which passes when the type is absent), a Chainsaw `error:` op against a kind whose CRD is not installed *errors the step* rather than asserting absence. If a type may legitimately not exist (a version rename, an optional component), gate the check behind a positive precondition or do it in a `script` step — never a bare `error:`. This is what broke the etcd suite on the `v1alpha1` → `v1alpha2` rename. - **No conditional Test-level `skip`.** There is no `skip: `; an env-var-gated test must `exit 0` early inside a `script` step (see the `ETCD_E2E_S3_ROUNDTRIP` gate). A Test with no matching gate always runs. - **`finally` is step-level, not test-level.** To guarantee teardown of something Chainsaw did not create, put the work *and* its own cleanup in a single `script` step; there is no test-wide `finally` block to hang it on. - **Cleanup is bounded and blocking — it surfaces latent teardown bugs.** Chainsaw waits for the resources it applied to actually delete (within the `delete`/`cleanup` timeouts in `hack/e2e-chainsaw/.chainsaw.yaml`) and *fails* the test if they don't. This is deliberate — a stuck uninstall starves the next suite — but it means a teardown path the BATS suite masked with `|| true` now surfaces as `context deadline exceeded` at cleanup. The failure is honest, not new; fix the teardown rather than widening the timeout (see cozystack/cozystack#3271, a pre-delete hook that hung on a nodeless tenant). - **Reading a failure diff: the "actual" side is a projection.** Chainsaw projects the live object onto the expected fields, so `status: {}` in a diff means "the expression matched nothing / evaluated empty," not "status is literally empty." The real reason is the `error` line *above* the diff — read that first. - **A `catch`/cleanup `script`'s cwd is the failing suite's directory** (`hack/e2e-chainsaw//`), so `../../..` is the repo root. The suites `cd ../../..` and then source shared helpers by repo-root-relative path (`hack/e2e-chainsaw/_lib/…`) rather than juggling `../` hops. ## Reviewer checklist for a new or changed E2E test 1. No new retry loop unless the step is pure infra (image pull / VM boot / network). 2. Resource readiness uses a Chainsaw `assert` (not an imperative `until kubectl get …; kubectl wait`); condition checks use the filter-as-list form `(conditions[?type == 'Ready'])`. 3. Imperative-only waits live in a `script` step; any bare `sleep` carries a `TODO(e2e-replace-fixed-timeouts):` justification, unless it is a measurement interval, which §2 carves out by name and which takes no marker because no event can replace it. 4. No test-level `EXIT`/`RETURN` trap — rely on Chainsaw cleanup; a self-contained trap is allowed inside a single `script` step (port-forward / temp dir), and inside an explicit subshell within a BATS `@test`, where it does not displace the runner's own handler. A BATS file that still carries test-level traps declares the count itself in `# EXIT-TRAP DEBT: N`; do not ask for one to be removed without checking that line, because the count is exact in both directions. 5. Controller-created artifacts the test cannot reclaim are pruned explicitly; nested tenants delete child → parent with a wait-for-deletion between. 6. Standard HR-Ready assert timeout is **5–6m**; longer waits (harbor 10m, NFS 10m, VM image pulls, platform-wide install 15m) are justified in-line. 7. Failure path attaches scoped diagnostics via a `catch:` block, never a silent pass. The node-join carve-out in §1 does not bend this: that suite still fails and still runs its catch, and only the lane's verdict changes. 8. If it touches parent-HR behavior, add the `status.history` remediation guard (`hack/e2e-chainsaw/_lib/remediation-guard.sh`). ## In-flight direction (not yet the merged standard) These are being explored on branches and may become conventions; do not assume they are the current `main` behavior: - **Cilium orphaned-endpoint self-heal** — an interim CI watchdog that evicts a single confirmed-orphan Cilium endpoint ("IP already in use", cilium/cilium#38313). Explicitly a mitigation to remove once a fixed Cilium ships; it refuses to touch an endpoint backing a live pod so real duplicate-IP bugs stay visible. - **Cluster state snapshot/restore** between test groups instead of reinstalling.