Timofei Larkin
1d3ab0bb4c
A managed application's tenant-facing Services are its endpoints, and the only way to publish one externally has been the chart boolean `external`, which publishes one chart-chosen endpoint on one address from no particular pool through five different render paths. EndpointAttachment attaches one address, minted from an IPAddressClass or bound from a reserved IPAddressClaim, to one tenant-facing Service by rendering an additive LoadBalancer Service that mirrors the endpoint's selector and ports. The mechanism therefore needs no per-engine render knowledge and never touches the application's own objects, and a reserved address survives every attachment and moves between applications. The endpoint check requires both the lineage labels and the tenantresource marker: lineage alone also covers operator-internal Services. Identity is by owner reference throughout, so a Service carrying the ownership label without the reference is never adopted, and a same-name recreated application does not inherit the previous incarnation's attachments. The VM datapath modes render the cozy-proxy Service contract unchanged and are gated on a platform declaration rather than CNI detection: a proxy skips a Service delegated to a name it does not claim while the load balancer still attracts the address. Exclusivity is keyed on the backend pod in both modes because cozy-proxy's egress map holds one address per pod, and a whole-IP attachment with several ready backends keeps its address dark instead of picking one. The e2e suite is parked as disabled until the address substrate is packaged; the unit tests carry the hold/move and identity proofs. The kind lives in local.sdn.cozystack.io, the CRD-served half of the sdn family the address substrate and cozyplane share, so stock ResourceQuota counts keep working; the proposal text is being corrected from cozystack.io to match. Assisted-by: LLM Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Chainsaw E2E tests
The Cozystack E2E app suite, written for Kyverno Chainsaw — a declarative, Kubernetes-native E2E test framework (CNCF, part of the Kyverno project; successor to KUTTL).
This directory replaces the per-app BATS suite that used to live in hack/e2e-apps/*.bats. The CI e2e job runs chainsaw test hack/e2e-chainsaw/ after make install-cozystack; cluster bootstrap (hack/e2e-prepare-cluster.bats, hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.
Layout
hack/e2e-chainsaw/
├── .chainsaw.yaml # global Configuration (namespace, timeouts, JUnit report)
├── _lib/ # shell reused by the script-heavy suites
│ ├── run-kubernetes.sh # tenant Kubernetes bringup + LB/NFS/ouroboros checks
│ └── remediation-guard.sh
├── <app>/
│ ├── chainsaw-test.yaml # one or more Test definitions
│ └── <app>.yaml # the manifests the test applies
└── ...
Suites: postgres, bucket, mariadb, mongodb, redis, valkey, qdrant, clickhouse, kafka, rabbitmq, etcd, openbao, harbor, foundationdb, external-dns, kuberture, vminstance, gateway, opensearch, kubernetes-latest, kubernetes-previous, securitygroup.
What Chainsaw buys over the BATS suite
- Readable tests — a list of resources and expected states, not shell control flow. The
timeout N sh -ec "until kubectl get ..."+kubectl waitpair that appeared ~100 times collapses into a singleassertthat polls existence and state together. - Distinguishable failure modes — "object never created", "created but never ready", and "wrong status" all surface as a structured diff instead of an opaque shell timeout.
- Automatic diagnostics —
catchblocks dump events,describe, and pod logs on any failure (the BATS suite only did this inharbor.bats, by hand). - Automatic cleanup — resources Chainsaw applied are deleted (and waited on) in reverse order;
finallycovers operator-created leftovers. - CI-ready output — a JUnit
chainsaw-report.xmlis emitted for the test summary.
Test anatomy
Each suite is a directory with a chainsaw-test.yaml (one or more Test docs) plus the manifests it applies. A step has try (operations: apply, assert, error, script, delete, …), optional catch (diagnostics on failure), and optional finally (teardown regardless of outcome).
Declarative suites assert on status.conditions and concrete fields. Inherently imperative suites keep their logic in script steps:
- openbao —
bao operator init/unsealviakubectl exec. - kuberture —
jqon Service annotations + external-dns split-horizon log assertions. - gateway — admission-rejection cases that need
kubectl --asimpersonation or error-message greps; the tenant apex is read from the namespacenamespace.cozystack.io/hostlabel at runtime so they pass on any host. - kubernetes-latest/previous — wrap
_lib/run-kubernetes.sh(Kamaji bringup, port-forward, LB/NFS/ouroboros). The scriptcds to the repo root (the test dir ishack/e2e-chainsaw/<suite>, so repo root is../../..).
Conventions / gotchas (Chainsaw v0.2.15)
- Condition asserts use the filter-as-list form, not the documented
[0]index:Thestatus: (conditions[?type == 'Ready']): - status: "True"(conditions[?type == 'Ready'])[0]: {status: "True"}idiom fails with "field not found in the input object". - A reading of a failure diff: the "actual" side is a projection onto the expected fields —
status: {}means the JMESPath expression failed, not that status is empty. Read the error line above the diff first. errorwith a bare resource ref (name only) asserts the resource does not exist (used for "PVC reclaimed", "no backup schedule", "child has no Gateway HR").scriptruns in the test directory with$NAMESPACEset to the config namespace (tenant-test).
Running locally
Requires a cluster with Cozystack installed and a tenant-test namespace (the environment hack/e2e-install-cozystack.bats produces). bucket additionally needs mc and nc on the host; kuberture/openbao/harbor need jq.
# install chainsaw
go install github.com/kyverno/chainsaw@latest # or: brew install kyverno/chainsaw/chainsaw
# run all suites (config picked up from .chainsaw.yaml)
cd hack/e2e-chainsaw
chainsaw test
# run a single suite
chainsaw test postgres/
kubernetes-latest/kubernetes-previous each bring up a full tenant Kubernetes cluster (~25 min, real worker nodes) and vminstance boots a nested VM — expect a long run for the full set.
CI integration
The e2e job in .github/workflows/pull-requests.yaml runs make -C packages/core/testing test-chainsaw (which execs chainsaw test hack/e2e-chainsaw/ inside the e2e sandbox) and uploads chainsaw-report.xml. The sandbox image (packages/core/testing/images/e2e-sandbox/Dockerfile) ships the chainsaw binary alongside kubectl/helm/mc/jq/nc.