Myasnikov Daniil
0034da0181
Four blockers, plus the immutability finding carried over from the second round and the minors alongside it. The optional-package helper emitted a Package carrying a variant and nothing else, and package_reconciler takes a component's HelmRelease values from the Package alone, so the worker image catalog documented four operator knobs that no supported path could set. That left the two scenarios the feature exists for -- air-gapped and non-replicated storage -- with a render guard naming a fix the operator could not apply. The helper now takes the same components argument its non-optional twin already had, iaas.yaml forwards a new platform-level kubernetesWorkerImage block, and the nine other optional packages get the same path. The transient headroom a clone needs was written down only in the CI harness. It is a property of the feature, so it belongs in diskSize and in the catalog's storage description, with the arithmetic spelled out. The catalog told operators to set nodeGroups.<name> on a Kubernetes CR, inert since the Phase 2 split, one paragraph above the query that gets it right. talos.version promised the matrix covers every pairing; the helper's own comment says the opposite three paragraphs above the code, and cozyvalues-gen copies the optimistic half into the schema, the README and the dashboard. Both now say what the guard does. An entry whose golden exists is now enforced immutable rather than merely documented so, because the documented recovery -- delete the golden -- runs into the deletion blast radius the same file warns about. Beside it the catalog refuses a duplicate (schematicID, version) and pattern-checks both fields, so it can no longer create a golden whose name a pool cannot reconstruct. The absence and Failed golden guards now fire only for a pool still waiting for its first disk. A pool that cloned months ago has no stake in the source, and aborting its whole HelmRelease on a periodic reconcile was the blast radius the mid-import case already refuses. An empty storageClass on either side now resolves against the cluster default before the cross-class comparison, which is the one corner that cannot be a clone and was passing unchecked. The image resolution is a named template both nodegroup.yaml and talos-reconcile-job.yaml call, carrying the format checks with it, so the boot disk and the in-guest installer cannot drift apart -- the argument this PR already made for the support matrix. The pool field is renamed image -> osImage. It landed one character from the existing images (chart-internal helper overrides) with an unrelated meaning and the same air-gapped motivation in its description. This is new CR surface that has not shipped, so the rename is free now and a migration later. Also: the schema enum drift guard iterated over a command substitution whose failure looked like an empty result, so a renamed property disarmed it; the e2e golden wait reported nothing when the outer timeout fired before the inner one; and ComputePlane's pool-shape allowlist now records the exclusion of osImage rather than leaving it merely absent. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Chainsaw E2E tests
The Cozystack E2E app suite, written for Kyverno Chainsaw — a declarative, Kubernetes-native E2E test framework (CNCF, part of the Kyverno project; successor to KUTTL).
This directory replaces the per-app BATS suite that used to live in hack/e2e-apps/*.bats. The CI e2e job runs chainsaw test hack/e2e-chainsaw/ after make install-cozystack; cluster bootstrap (hack/e2e-prepare-cluster.bats, hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.
Layout
hack/e2e-chainsaw/
├── .chainsaw.yaml # global Configuration (namespace, timeouts, JUnit report)
├── _lib/ # shell reused by the script-heavy suites
│ ├── run-kubernetes.sh # tenant Kubernetes bringup + LB/NFS/ouroboros checks
│ └── remediation-guard.sh
├── <app>/
│ ├── chainsaw-test.yaml # one or more Test definitions
│ └── <app>.yaml # the manifests the test applies
└── ...
Suites: postgres, bucket, mariadb, mongodb, redis, qdrant, clickhouse, kafka, etcd, openbao, harbor, foundationdb, external-dns, kuberture, vminstance, gateway, kubernetes-latest, kubernetes-previous, kubernetes-oidc-system, kubernetes-oidc-customconfig, securitygroup.
What Chainsaw buys over the BATS suite
- Readable tests — a list of resources and expected states, not shell control flow. The
timeout N sh -ec "until kubectl get ..."+kubectl waitpair that appeared ~100 times collapses into a singleassertthat polls existence and state together. - Distinguishable failure modes — "object never created", "created but never ready", and "wrong status" all surface as a structured diff instead of an opaque shell timeout.
- Automatic diagnostics —
catchblocks dump events,describe, and pod logs on any failure (the BATS suite only did this inharbor.bats, by hand). - Automatic cleanup — resources Chainsaw applied are deleted (and waited on) in reverse order;
finallycovers operator-created leftovers. - CI-ready output — a JUnit
chainsaw-report.xmlis emitted for the test summary.
Test anatomy
Each suite is a directory with a chainsaw-test.yaml (one or more Test docs) plus the manifests it applies. A step has try (operations: apply, assert, error, script, delete, …), optional catch (diagnostics on failure), and optional finally (teardown regardless of outcome).
Declarative suites assert on status.conditions and concrete fields. Inherently imperative suites keep their logic in script steps:
- openbao —
bao operator init/unsealviakubectl exec. - kuberture —
jqon Service annotations + external-dns split-horizon log assertions. - gateway — admission-rejection cases that need
kubectl --asimpersonation or error-message greps; the tenant apex is read from the namespacenamespace.cozystack.io/hostlabel at runtime so they pass on any host. - kubernetes-latest/previous — wrap
_lib/run-kubernetes.sh(Kamaji bringup, port-forward, LB/NFS/ouroboros). The scriptcds to the repo root (the test dir ishack/e2e-chainsaw/<suite>, so repo root is../../..).
Conventions / gotchas (Chainsaw v0.2.15)
- Condition asserts use the filter-as-list form, not the documented
[0]index:Thestatus: (conditions[?type == 'Ready']): - status: "True"(conditions[?type == 'Ready'])[0]: {status: "True"}idiom fails with "field not found in the input object". - A reading of a failure diff: the "actual" side is a projection onto the expected fields —
status: {}means the JMESPath expression failed, not that status is empty. Read the error line above the diff first. errorwith a bare resource ref (name only) asserts the resource does not exist (used for "PVC reclaimed", "no backup schedule", "child has no Gateway HR").scriptruns in the test directory with$NAMESPACEset to the config namespace (tenant-test).
Running locally
Requires a cluster with Cozystack installed and a tenant-test namespace (the environment hack/e2e-install-cozystack.bats produces). bucket additionally needs mc and nc on the host; kuberture/openbao/harbor need jq.
# install chainsaw
go install github.com/kyverno/chainsaw@latest # or: brew install kyverno/chainsaw/chainsaw
# run all suites (config picked up from .chainsaw.yaml)
cd hack/e2e-chainsaw
chainsaw test
# run a single suite
chainsaw test postgres/
kubernetes-latest/kubernetes-previous each bring up a full tenant Kubernetes cluster (~25 min, real worker nodes) and vminstance boots a nested VM — expect a long run for the full set.
CI integration
The e2e job in .github/workflows/pull-requests.yaml runs make -C packages/core/testing test-chainsaw (which execs chainsaw test hack/e2e-chainsaw/ inside the e2e sandbox) and uploads chainsaw-report.xml. The sandbox image (packages/core/testing/images/e2e-sandbox/Dockerfile) ships the chainsaw binary alongside kubectl/helm/mc/jq/nc.