Files
cozystack/hack/e2e-chainsaw
Andrey KolkovandClaude Opus 4.8 0c14d6cf43 test(e2e): sharpen round-trip diagnostics and make teardown settle
Recommendations from the third review pass (which returned LGTM; none of these
blocked, all three are cheap and make the failure paths honest).

1. The round-trip's catch named nothing. An unnamed `describe HelmRelease` in
   tenant-root dumps every release on the root tenant of a live cluster and
   buries the two that matter, so name them (mariadb-2-backup-roundtrip names
   its release for the same reason). Raised podLogs tail 100 -> 200, since the
   sidecar logs every action and an S3 error sits well back from the tail by the
   time a step fails, and added the Keeper Pods: chkeeper.yaml labels them
   `app: <release>-keeper`, so the existing app.kubernetes.io/instance selectors
   missed them entirely — and a source ClickHouse that never reaches a ready
   replica is usually Keeper failing to form a quorum, which explains itself
   only there.

2. SSL_CERT_DIR is now `/etc/ssl/cozy-s3-ca:/etc/ssl/certs`. The single-directory
   form works — Go reads the default bundle FILE regardless, which is what the
   earlier measurement showed — but that left the public-CA half of the contract
   resting on a file this image happens to ship. Naming the system directory
   costs nothing and survives a base-image bump that moves the bundle; verified
   against public AWS S3 that both forms still verify (403 auth, no x509). The
   mount path stays the bare directory, and the unit test now pins both halves
   separately — it caught the over-broad edit that had briefly made mountPath a
   colon list.

3. cleanup.sh returned while the uninstalls it triggered were still running, so
   the next suite (etcd) could start with helm-controller still draining this
   one, and the pre-clean could hand step 04 an application whose predecessor
   was still going away. It now waits for both ClickHouse releases and the
   Bucket to actually disappear, and — since the script deliberately runs
   without `set -e` so one absent resource cannot abandon the rest — records
   those failures and exits non-zero, rather than reporting a teardown that did
   not settle as success. wait_deleted polls `get` instead of using
   `kubectl wait --for=delete`, which errors on an already-absent resource: the
   normal case here, and indistinguishable from a real failure.

Sized the two cleanup step budgets to 12m for the same reason the run step is
60m: cleanup.sh's waits now sum to 540s, which the previous 8m did not clear.

The Keeper-sized-from-the-server-resources bug the review found outside this
diff is filed as #3487.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00
..

Chainsaw E2E tests

The Cozystack E2E app suite, written for Kyverno Chainsaw — a declarative, Kubernetes-native E2E test framework (CNCF, part of the Kyverno project; successor to KUTTL).

This directory replaces the per-app BATS suite that used to live in hack/e2e-apps/*.bats. The CI e2e job runs chainsaw test hack/e2e-chainsaw/ after make install-cozystack; cluster bootstrap (hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.

Layout

hack/e2e-chainsaw/
├── .chainsaw.yaml          # global Configuration (namespace, timeouts, JUnit report)
├── _lib/                   # shell reused by the script-heavy suites
│   ├── run-kubernetes.sh   #   tenant Kubernetes bringup + LB/NFS/ouroboros checks
│   └── remediation-guard.sh
├── <app>/
│   ├── chainsaw-test.yaml  # one or more Test definitions
│   └── <app>.yaml          # the manifests the test applies
└── ...

Suites: postgres, bucket, mariadb, mongodb, redis, qdrant, clickhouse, kafka, etcd, openbao, harbor, foundationdb, external-dns, kuberture, vminstance, gateway, kubernetes-latest, kubernetes-previous, kubernetes-oidc-system, kubernetes-oidc-customconfig, securitygroup.

What Chainsaw buys over the BATS suite

  • Readable tests — a list of resources and expected states, not shell control flow. The timeout N sh -ec "until kubectl get ..." + kubectl wait pair that appeared ~100 times collapses into a single assert that polls existence and state together.
  • Distinguishable failure modes — "object never created", "created but never ready", and "wrong status" all surface as a structured diff instead of an opaque shell timeout.
  • Automatic diagnostics — catch blocks dump events, describe, and pod logs on any failure (the BATS suite only did this in harbor.bats, by hand).
  • Automatic cleanup — resources Chainsaw applied are deleted (and waited on) in reverse order; finally covers operator-created leftovers.
  • CI-ready output — a JUnit chainsaw-report.xml is emitted for the test summary.

Test anatomy

Each suite is a directory with a chainsaw-test.yaml (one or more Test docs) plus the manifests it applies. A step has try (operations: apply, assert, error, script, delete, …), optional catch (diagnostics on failure), and optional finally (teardown regardless of outcome).

Declarative suites assert on status.conditions and concrete fields. Inherently imperative suites keep their logic in script steps:

  • openbao — bao operator init/unseal via kubectl exec.
  • kuberture — jq on Service annotations + external-dns split-horizon log assertions.
  • gateway — admission-rejection cases that need kubectl --as impersonation or error-message greps; the tenant apex is read from the namespace namespace.cozystack.io/host label at runtime so they pass on any host.
  • kubernetes-latest/previous — wrap _lib/run-kubernetes.sh (Kamaji bringup, port-forward, LB/NFS/ouroboros). The script cds to the repo root (the test dir is hack/e2e-chainsaw/<suite>, so repo root is ../../..).

Conventions / gotchas (Chainsaw v0.2.15)

  • Condition asserts use the filter-as-list form, not the documented [0] index:
    status:
      (conditions[?type == 'Ready']):
      - status: "True"
    
    The (conditions[?type == 'Ready'])[0]: {status: "True"} idiom fails with "field not found in the input object".
  • A reading of a failure diff: the "actual" side is a projection onto the expected fields — status: {} means the JMESPath expression failed, not that status is empty. Read the error line above the diff first.
  • error with a bare resource ref (name only) asserts the resource does not exist (used for "PVC reclaimed", "no backup schedule", "child has no Gateway HR").
  • script runs in the test directory with $NAMESPACE set to the config namespace (tenant-test).

Running locally

Requires a cluster with Cozystack installed and a tenant-test namespace (the environment hack/e2e-install-cozystack.bats produces). bucket additionally needs mc and nc on the host; kuberture/openbao/harbor need jq.

# install chainsaw
go install github.com/kyverno/chainsaw@latest   # or: brew install kyverno/chainsaw/chainsaw

# run all suites (config picked up from .chainsaw.yaml)
cd hack/e2e-chainsaw
chainsaw test

# run a single suite
chainsaw test postgres/

kubernetes-latest/kubernetes-previous each bring up a full tenant Kubernetes cluster (~25 min, real worker nodes) and vminstance boots a nested VM — expect a long run for the full set.

CI integration

The e2e job in .github/workflows/pull-requests.yaml runs make -C packages/core/testing test-chainsaw (which execs chainsaw test hack/e2e-chainsaw/ inside the e2e sandbox) and uploads chainsaw-report.xml. The sandbox image (packages/core/testing/images/e2e-sandbox/Dockerfile) ships the chainsaw binary alongside kubectl/helm/mc/jq/nc.