Files
cozystack/hack/e2e-chainsaw
Aleksei Sviridkin 8cb1023788 fix(e2e): fail on a failed release of cilium and csi as on the parent
The history guard covered only the parent Kubernetes HelmRelease. A
failed action on one of the addon releases it installs left no mark
the suite inspected, so a run surfaced only the downstream symptom,
typically the node-join deadline, with nothing naming the release
behind it. The tenant CNI and CSI are the sharp cases, since they
carry a cluster-wide precondition.

The guard now reads every addon release of the parent, selected from
the cluster by name prefix so a release added later is covered too.
A "failed" or "uninstalled" Snapshot fails the run on a release whose
install strategy is RetryOnFailure: the parent, which the aggregated
apiserver stamps that way, and cilium and csi. The same footprint on
the other addons, which pair the default strategy with retries: -1,
is printed as a note and leaves the run green.

The footprint is not split into a teardown and a retried failure.
In helm-controller v1.5.0, the version the platform installs, and
unchanged in v1.5.1, a release with an active retry strategy never
remediates: atomic_release.go returns an upgrade for a failed
release before it reaches the remediation switch (lines 465-475), and
the two other paths that uninstall, a changed release target and
deletion, both call ClearHistory afterwards. On a RetryOnFailure
release an "uninstalled" Snapshot is therefore unreachable, and the
"failed" one is the only signal. A retry that later recovered still
fails the run, by design, with one blind spot the statuses cannot
close: a failed install that the very next attempt recovers is
marked superseded by Helm, because there is no deployed release to
supersede instead, and its history reads like an ordinary upgrade.

The guard runs after the assertions, while every deadline a failed
addon breaks ends the script earlier, so the same classification is
also printed without a verdict from the node-join collector and from
the exit handler. That handler is now armed at the tenant CR apply,
so the management-cluster waits before the tenant answers are
covered too. Every read is bounded and never folded into "no
history", and the parent must read back RetryOnFailure: an empty
value there means the jsonpath stopped resolving, and any other value
means the apiserver stamp is gone, either of which would turn its
footprint into a note.

The history read takes its jsonpath from HELMRELEASE_HISTORY_JSONPATH,
and the upstream-type pin accepts the bounded wrapper the read goes
through.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 06:59:31 +03:00
..

Chainsaw E2E tests

The Cozystack E2E app suite, written for Kyverno Chainsaw — a declarative, Kubernetes-native E2E test framework (CNCF, part of the Kyverno project; successor to KUTTL).

This directory replaces the per-app BATS suite that used to live in hack/e2e-apps/*.bats. The CI e2e job runs every suite under hack/e2e-chainsaw/ through hack/e2e-chainsaw-run.sh after make install-cozystack; cluster bootstrap (hack/e2e-prepare-cluster.bats, hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.

Layout

hack/e2e-chainsaw/
├── .chainsaw.yaml          # global Configuration (namespace, timeouts, JUnit report)
├── _lib/                   # shell reused by the script-heavy suites
│   ├── run-kubernetes.sh   #   tenant Kubernetes bringup + LB/NFS/ouroboros checks
│   └── remediation-guard.sh
├── <app>/
│   ├── chainsaw-test.yaml  # one or more Test definitions
│   └── <app>.yaml          # the manifests the test applies
└── ...

Suites: postgres, bucket, mariadb, mongodb, redis, valkey, qdrant, clickhouse, kafka, rabbitmq, etcd, openbao, harbor, foundationdb, external-dns, kuberture, vminstance, gateway, opensearch, kubernetes-latest, kubernetes-previous, securitygroup, seaweedfs.

What Chainsaw buys over the BATS suite

  • Readable tests — a list of resources and expected states, not shell control flow. The timeout N sh -ec "until kubectl get ..." + kubectl wait pair that appeared ~100 times collapses into a single assert that polls existence and state together.
  • Distinguishable failure modes — "object never created", "created but never ready", and "wrong status" all surface as a structured diff instead of an opaque shell timeout.
  • Automatic diagnostics — catch blocks dump events, describe, and pod logs on any failure (the BATS suite only did this in harbor.bats, by hand).
  • Automatic cleanup — resources Chainsaw applied are deleted (and waited on) in reverse order; finally covers operator-created leftovers.
  • CI-ready output — a JUnit chainsaw-report.xml is emitted for the test summary.

Test anatomy

Each suite is a directory with a chainsaw-test.yaml (one or more Test docs) plus the manifests it applies. A step has try (operations: apply, assert, error, script, delete, …), optional catch (diagnostics on failure), and optional finally (teardown regardless of outcome).

Declarative suites assert on status.conditions and concrete fields. Inherently imperative suites keep their logic in script steps:

  • openbao — bao operator init/unseal via kubectl exec.
  • kuberture — jq on Service annotations + external-dns split-horizon log assertions.
  • gateway — admission-rejection cases that need kubectl --as impersonation or error-message greps; the tenant apex is read from the namespace namespace.cozystack.io/host label at runtime so they pass on any host.
  • kubernetes-latest/previous — wrap _lib/run-kubernetes.sh (Kamaji bringup, port-forward, LB/NFS/ouroboros). The script cds to the repo root (the test dir is hack/e2e-chainsaw/<suite>, so repo root is ../../..).

Conventions / gotchas (Chainsaw v0.2.15)

  • Condition asserts use the filter-as-list form, not the documented [0] index:
    status:
      (conditions[?type == 'Ready']):
      - status: "True"
    
    The (conditions[?type == 'Ready'])[0]: {status: "True"} idiom fails with "field not found in the input object".
  • A reading of a failure diff: the "actual" side is a projection onto the expected fields — status: {} means the JMESPath expression failed, not that status is empty. Read the error line above the diff first.
  • error with a bare resource ref (name only) asserts the resource does not exist (used for "PVC reclaimed", "no backup schedule", "child has no Gateway HR").
  • script runs in the test directory with $NAMESPACE set to the config namespace (tenant-test).

Running locally

Requires a cluster with Cozystack installed and a tenant-test namespace (the environment hack/e2e-install-cozystack.bats produces). bucket and seaweedfs additionally need mc, nc, and jq on the host; kuberture/openbao/harbor need jq.

# install chainsaw
go install github.com/kyverno/chainsaw@latest   # or: brew install kyverno/chainsaw/chainsaw

# run all suites (config picked up from .chainsaw.yaml)
cd hack/e2e-chainsaw
chainsaw test

# run a single suite
chainsaw test postgres/

kubernetes-latest/kubernetes-previous each bring up a full tenant Kubernetes cluster (~25 min, real worker nodes) and vminstance boots a nested VM — expect a long run for the full set.

CI integration

The e2e job in .github/workflows/pull-requests.yaml runs make -C packages/core/testing test-chainsaw (which execs hack/e2e-chainsaw-run.sh inside the e2e sandbox) and uploads chainsaw-report.xml. The runner calls chainsaw test once per suite directory, fails a suite whose pods outlive its cleanup by more than the default timeouts.cleanup (LEAKED-PODS suite=<dir> in the log; see docs/agents/e2e-testing.md §8 and _lib/leak-allowlist.txt for known, filed leaks), and merges the per-suite reports into that one file. A local chainsaw test as shown above runs the same tests without the leak check. The sandbox image (packages/core/testing/images/e2e-sandbox/Dockerfile) ships the chainsaw binary alongside kubectl/helm/mc/jq/nc.