Files
cozystack/hack/e2e-chainsaw
Aleksei Sviridkin b1519acfe7 fix(e2e): cap the bucket S3 reads so the verify step honors its timeout (#4281)
## What this PR does

Refs #4268 — a plain reference, not `Fixes`: the issue asks for a
`script` op to be cut off at its declared ceiling, and that is Chainsaw
behaviour this repo cannot change. Chainsaw runs the op's shell under
Go's `exec.CommandContext`, whose default cancel SIGKILLs the shell PID
alone and then blocks on the inherited stdout/stderr pipe until an
orphaned `mc` closes it, so the op returns only when `mc` exits on its
own rather than at the ceiling. This PR mitigates that where it bit —
per-command caps so no descendant outlives its own budget — but the
op-timeout gap itself lives in Chainsaw, so the link stays a reference
rather than a `Fixes`. The sibling step that produced the 606s,
`verify-every-bucket-takes-a-write` in the `bucket-1-capacity` Test, is
on `main` and already carries its own per-call caps; this PR applies the
same shape to `verify-s3-access`.

On run 34948279363 the bucket suite's
`verify-every-bucket-takes-a-write` step declared `timeout: 5m` but ran
606s and ended on `mc`'s own non-zero status rather than being cut at
the ceiling. `verify-s3-access` in
`hack/e2e-chainsaw/bucket/chainsaw-test.yaml` is the same-shaped step in
this suite. This PR wraps every S3 (`mc`) and apiserver (`kubectl`) read
in `verify-s3-access` in its own `timeout`, so a hung read dies at a
known budget instead of holding the step open, and corrects the two
places (`hack/e2e-chainsaw/.chainsaw.yaml` and
`docs/agents/e2e-testing.md`) that explained the op ceiling with a
mechanism that does not hold. It caps `verify-s3-access` for sharing the
shape, not for producing the 606s number.

### Which of the issue's two candidate mechanisms holds

The issue named two candidates and stated the log could not separate
them. The Chainsaw v0.2.15 source separates them, and it is the second
one, with the exact refinement the issue proposed.

**Candidate 1 — "an operation `timeout` does not bound a `script` op" —
is refuted.** The op timeout is applied and does reach the process:
`pkg/runner/operations/script.go:24` copies the step's `timeout` into
`Timeouts.Exec`, and `:38` wraps the op context with `ctx, cancel :=
context.WithTimeout(ctx, tc.Timeouts().Exec)` before calling
`op.Exec(ctx, …)`. So a 5m step produces a real 300s deadline on the
context the command runs under.

**Candidate 2 — "it bounds it and kills the shell, but Chainsaw waits on
a pipe a grandchild (`mc`) still holds open" — is confirmed.** In
`pkg/engine/operations/script/operation.go` the command is built with
`exec.CommandContext(ctx, shell, …)` (`:105`), and in `execute` its
stdout/stderr are set to `*bytes.Buffer` (`:128`–`:129`, the buffers are
`internal.CommandOutput` in
`pkg/engine/operations/internal/command_output.go:12`–`13`) and run via
`cmd.Run()` (`:130`). Two Go `os/exec` properties then produce the
overrun: `exec.CommandContext`'s default cancel is `Process.Kill()` on
the shell PID alone — not a process-group kill, and the operation sets
no `SysProcAttr.Setpgid` — and because stdout/stderr are not `*os.File`,
`os/exec` allocates OS pipes and copier goroutines, so `cmd.Wait()`
blocks until those pipes reach EOF. EOF needs every inherited write-end
closed. When the shell forks `mc`, `mc` inherits those fds; at 300s the
deadline SIGKILLs the shell but `mc` is orphaned, keeps retrying the S3
PUT, and holds the pipe, so the op does not return until `mc` exits on
its own at 606s. `cmd.Run()` then returns `signal: killed`, which is
stored in the `$error` binding and never printed — only the `($error ==
null): true` check failure is logged — which is why grepping the run for
`signal: killed` found nothing.

**Empirical confirmation.** A standalone Go program reproduces it
exactly. `exec.CommandContext` with a 1s timeout, `*bytes.Buffer`
stdout/stderr, running `sh -c 'sleep 2 & sleep 4'` (a descendant that
inherits the pipe): `cmd.Run()` returns after ~4s, not 1s, with
`err=signal: killed` and `ctx.Err()=context deadline exceeded` — the
ceiling fired at 1s, control came back only when the descendant closed
the pipe. Redirecting the descendants' stdout/stderr away from the pipe
(`… >/dev/null 2>&1`) returns at 1.00s. That is the whole mechanism: the
ceiling cuts the shell but not a descendant that still holds the pipe.

### The fix

Per-command caps, the remedy the issue itself points to ("every read
carries its own cap") and the shape already used elsewhere in the suite.
The bucket step now defines a `k()` wrapper (`timeout 30 kubectl`) and
an `mc()` wrapper (`timeout 15 mc`) for the reads, and calls `timeout 45
mc cp` directly for the two PUTs; a hung read is killed at its own
budget, releasing the inherited pipe promptly, so even when the op
ceiling fires the residual is one inner cap rather than `mc`'s full
retry exhaustion. The terminal precedents for these budgets are the
redis suite's terminal `k()` (30s, `redis/chainsaw-test.yaml:298`) and
the sibling `verify-every-bucket-takes-a-write` step's terminal `mc cp`
(45s, `:390`, sized for a volume grow); the 15s cap on `mc` in
`hack/e2e-chainsaw/_lib/backup-access-preflight.sh` is not a precedent,
because it sits inside a `while :;` retry loop where a 124 costs one
attempt, whereas here under `set -e` a 124 ends the step. The inner
budgets — 2×30 for the two `k` reads + 2×45 for the two `mc cp` PUTs +
at most 7×15 for the other `mc` calls (eight other call sites, but the
two cleanup `mc rm` are mutually exclusive) + the 30s port-forward
readiness wait = at most 285s — sit under the 5m op ceiling, which stays
as a backstop, per the "op above the sum of inner budgets" rule in
`docs/agents/e2e-testing.md`. The long-lived `kubectl port-forward`
keeps no cap by design (it must persist for the whole step, and its
stdout/stderr are already redirected to `/dev/null`, so it cannot hold
the step's pipe). The `mc()` wrapper is non-recursive because `timeout`
execs its argument through a PATH lookup that bypasses shell functions —
verified in `dash`, `bash` and `sh`. The `catch` block's diagnostic read
gets the same treatment — `timeout 30 kubectl` under a 1m op `timeout` —
for the same pipe reason.

### The readonly deny probe fails on a timeout

The readonly-upload probe used to branch on `mc` directly (`if mc cp …;
then`), which suspends `set -e`, so a `timeout` there (exit 124) was
indistinguishable from the access-denied non-zero the probe expects: a
45s hang on the deny path would exit 124, take the false branch, reach
cleanup and exit 0 — a long hang turned green, the exact window this PR
exists to bound. This is the first `ro-user` write in the step, so no
earlier call proved that path healthy. The probe now captures the
status: exit 0 is the readonly regression, 124 is a timed-out probe,
both fail; any other non-zero is the expected denial.

### No unit surface; why the evidence is the proof

This is an e2e-config and doc-correctness change. The embedded Chainsaw
script is not extracted into a unit harness anywhere in the tree, so
there is no red-to-green unit test to add without inventing a bespoke
script-linter (which would be a new checker standing in for the fix).
The regression is instead pinned by the mechanism proof above: the
source citations show the ceiling cannot cut a descendant that holds the
pipe, and the caps remove the only way a descendant outlives its budget.
The deny-probe status handling and the pipe-vs-file secret reads were
exercised against stub exit codes (0 → regression-fail, 124 →
timeout-fail, other non-zero → expected denial) and the whole script
passes `sh -n` and `bash -n`.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The change touches only `hack/e2e-chainsaw/` suite content and
`docs/agents/e2e-testing.md`; it moves or renames nothing under `hack/`,
and touches no node-prerequisite or `package.mk` path. No downstream
repository is reached.

- [x] No downstream repository is affected by this change

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved end-to-end S3 access verification so stalled Kubernetes API
and S3 operations fail promptly.
- Correctly distinguishes timed-out access checks from expected
permission denials.
- Preserves partial command output and allows subsequent test steps to
continue after individual timeouts.
- Applies separate time limits to bucket reads, uploads, and access
checks for more predictable test behavior.

- **Documentation**
- Clarified operation timeout behavior and recommended per-command
limits within the overall operation limit.
- Documented timeout handling for orphaned processes and updated
worst-case timing guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 17:14:56 +04:00
..

Chainsaw E2E tests

The Cozystack E2E app suite, written for Kyverno Chainsaw — a declarative, Kubernetes-native E2E test framework (CNCF, part of the Kyverno project; successor to KUTTL).

This directory replaces the per-app BATS suite that used to live in hack/e2e-apps/*.bats. The CI e2e job runs chainsaw test hack/e2e-chainsaw/ after make install-cozystack; cluster bootstrap (hack/e2e-prepare-cluster.bats, hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.

Layout

hack/e2e-chainsaw/
├── .chainsaw.yaml          # global Configuration (namespace, timeouts, JUnit report)
├── _lib/                   # shell reused by the script-heavy suites
│   ├── run-kubernetes.sh   #   tenant Kubernetes bringup + LB/NFS/ouroboros checks
│   └── remediation-guard.sh
├── <app>/
│   ├── chainsaw-test.yaml  # one or more Test definitions
│   └── <app>.yaml          # the manifests the test applies
└── ...

Suites: postgres, bucket, mariadb, mongodb, redis, valkey, qdrant, clickhouse, kafka, rabbitmq, etcd, openbao, harbor, foundationdb, external-dns, kuberture, vminstance, gateway, opensearch, kubernetes-latest, kubernetes-previous, securitygroup.

What Chainsaw buys over the BATS suite

  • Readable tests — a list of resources and expected states, not shell control flow. The timeout N sh -ec "until kubectl get ..." + kubectl wait pair that appeared ~100 times collapses into a single assert that polls existence and state together.
  • Distinguishable failure modes — "object never created", "created but never ready", and "wrong status" all surface as a structured diff instead of an opaque shell timeout.
  • Automatic diagnostics — catch blocks dump events, describe, and pod logs on any failure (the BATS suite only did this in harbor.bats, by hand).
  • Automatic cleanup — resources Chainsaw applied are deleted (and waited on) in reverse order; finally covers operator-created leftovers.
  • CI-ready output — a JUnit chainsaw-report.xml is emitted for the test summary.

Test anatomy

Each suite is a directory with a chainsaw-test.yaml (one or more Test docs) plus the manifests it applies. A step has try (operations: apply, assert, error, script, delete, …), optional catch (diagnostics on failure), and optional finally (teardown regardless of outcome).

Declarative suites assert on status.conditions and concrete fields. Inherently imperative suites keep their logic in script steps:

  • openbao — bao operator init/unseal via kubectl exec.
  • kuberture — jq on Service annotations + external-dns split-horizon log assertions.
  • gateway — admission-rejection cases that need kubectl --as impersonation or error-message greps; the tenant apex is read from the namespace namespace.cozystack.io/host label at runtime so they pass on any host.
  • kubernetes-latest/previous — wrap _lib/run-kubernetes.sh (Kamaji bringup, port-forward, LB/NFS/ouroboros). The script cds to the repo root (the test dir is hack/e2e-chainsaw/<suite>, so repo root is ../../..).

Conventions / gotchas (Chainsaw v0.2.15)

  • Condition asserts use the filter-as-list form, not the documented [0] index:
    status:
      (conditions[?type == 'Ready']):
      - status: "True"
    
    The (conditions[?type == 'Ready'])[0]: {status: "True"} idiom fails with "field not found in the input object".
  • A reading of a failure diff: the "actual" side is a projection onto the expected fields — status: {} means the JMESPath expression failed, not that status is empty. Read the error line above the diff first.
  • error with a bare resource ref (name only) asserts the resource does not exist (used for "PVC reclaimed", "no backup schedule", "child has no Gateway HR").
  • script runs in the test directory with $NAMESPACE set to the config namespace (tenant-test).

Running locally

Requires a cluster with Cozystack installed and a tenant-test namespace (the environment hack/e2e-install-cozystack.bats produces). bucket additionally needs mc and nc on the host; kuberture/openbao/harbor need jq.

# install chainsaw
go install github.com/kyverno/chainsaw@latest   # or: brew install kyverno/chainsaw/chainsaw

# run all suites (config picked up from .chainsaw.yaml)
cd hack/e2e-chainsaw
chainsaw test

# run a single suite
chainsaw test postgres/

kubernetes-latest/kubernetes-previous each bring up a full tenant Kubernetes cluster (~25 min, real worker nodes) and vminstance boots a nested VM — expect a long run for the full set.

CI integration

The e2e job in .github/workflows/pull-requests.yaml runs make -C packages/core/testing test-chainsaw (which execs chainsaw test hack/e2e-chainsaw/ inside the e2e sandbox) and uploads chainsaw-report.xml. The sandbox image (packages/core/testing/images/e2e-sandbox/Dockerfile) ships the chainsaw binary alongside kubectl/helm/mc/jq/nc.