SANDBOX_NAME is per-checkout and these are not, which reads like an
oversight until you notice that hack/e2e-compose.yaml's container names
and the data-srvN zpools are global too. Two checkouts on one host
therefore share nodes and pools, and the second one's teardown destroys
the first one's storage.
Recorded rather than fixed. CI cannot reach it -- the lane already asks
for 24 vCPU and 72 GiB against a 32 vCPU / 128 GiB runner, so a second
one does not fit beside it -- and isolating it properly means renaming
the pools, which the LINSTOR storage-pool registration reads.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.
The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.
Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.
Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.
The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.
Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.
One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.
Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
An e2e lane asks for a runner class, not for a machine. A flex shape is
filled from whatever capacity the pool has at the time, so one lane can
be served by two processor generations, and the region is a preference
with a fallback. None of that reached the report, so a red run and a
green one could not be compared by the hardware they ran on.
The read is taken on the runner VM, before the sandbox container is
created, because the metadata service answers on a link-local address
the container has no guaranteed route to: an "unreachable" answered from
in there would look exactly like a capture that had tried. It is wired
into the sandbox apply target rather than into a workflow step, so every
lane picks it up through `make prepare-env` on both the passing and the
failing path, and cozyreport folds the file into the artifact from
inside without going near the endpoint.
The keys are fetched one at a time and the root instance document never
is. That document carries the instance and compartment OCIDs, the image
OCID, the display name, the hostname and the user metadata bag, and the
report is a public artifact, so asking per key makes that property
something the wire settles rather than something a filter has to be
trusted for. It also matches where the shape sizing is published: it has
its own subpath and is not in the root document at all. From the sizing
object only the numbers are kept, so the rule holds for keys nobody has
seen yet, and an answer that is not one scalar is named rather than
printed.
Every read carries a wall-clock ceiling, since a read that never returns
costs the step its whole budget without reaching any exit status, and the
first key doubles as the probe so a machine that is not an OCI instance
pays one ceiling rather than one per key. Nothing here is fatal: this
runs inside the one retried step in the job, so a runner with no
metadata service leaves an "imds unreachable" line and an exit status
of 0.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
The E2E job once ran as several state-sharing self-hosted-runner jobs
that rendezvoused through a sha256(REPO:WORKFLOW:REF) key and a per-run
/tmp/$SANDBOX_NAME copy of the checkout. Since the move to the ephemeral
pool it is a single job: there is no cross-job rendezvous and the
checkout is never reused, so copying it into /tmp and removing it
afterwards is dead weight.
Run every step in $GITHUB_WORKSPACE instead: drop the Prepare/Remove
workspace steps and the `cd /tmp/$SANDBOX_NAME` prefixes, and point the
upload paths at _out/. This deletes the unguarded `rm -rf
/tmp/$SANDBOX_NAME` rather than guarding it: when an upstream step fails
or the job is cancelled, the unconditional `Set sandbox ID` is skipped,
SANDBOX_NAME stays empty, and the `if: always()` steps interpolate it —
the removal expanding to `rm -rf /tmp/` and failing the cleanup step.
`Set sandbox ID` stays, since the container still needs a name.
Apply the same treatment to nightly.yaml, and drop the Makefile dead
code the single-runner retirement left behind: the world-writable
mkdir/chmod of /tmp/$SANDBOX_NAME (backed a bind mount removed long ago)
and the `-e SANDBOX_NAME` container env (nothing in the e2e-sandbox
image reads it).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
## What this PR does
Migrates the E2E **app** test suite from BATS to [Kyverno
Chainsaw](https://github.com/kyverno/chainsaw) — a declarative,
Kubernetes-native E2E framework (CNCF, part of the Kyverno project;
successor to KUTTL) — and wires it into CI as the replacement for the
per-app BATS loop. This grows the original pilot (postgres + bucket)
into the full suite.
All 22 app suites are ported under `hack/e2e-chainsaw/` and the
`hack/e2e-apps/*.bats` files are removed:
`postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `qdrant`,
`clickhouse`, `kafka`, `etcd`, `openbao`, `harbor`, `foundationdb`,
`external-dns`, `kuberture`, `vminstance`, `gateway`,
`kubernetes-latest`, `kubernetes-previous`, `kubernetes-oidc-system`,
`kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure`.
**Approach**
- **Declarative** suites assert on `status.conditions` and concrete
fields. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait`
pair that appeared ~100 times collapses into a single `assert` that
polls existence and state together, with a structured diff on failure
and automatic `events`/`describe`/`podLogs` capture via `catch`
(previously only `harbor.bats` did this, by hand).
- **Imperative** suites keep their logic in `script` steps: `openbao`
init/unseal, `kuberture` external-dns split-horizon probes,
`vminstance`, the `gateway` admission/impersonation cases, and
`kubernetes-latest`/`previous`, which wrap the relocated
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` verbatim (Kamaji bring-up,
LB/NFS/ouroboros checks).
- `gateway` tests derive the tenant apex from the namespace
`namespace.cozystack.io/host` label at runtime, so they are
host-independent.
**CI**
- The `e2e` job now runs `chainsaw test hack/e2e-chainsaw/` via a new
`test-chainsaw` target and uploads the JUnit `chainsaw-report.xml`.
- The `chainsaw` binary is added to the e2e-sandbox image.
- `install-cozystack` and `test-openapi` stay BATS (cluster bootstrap +
OpenAPI checks).
- The per-app 3-retry loop is dropped — assertion polling replaces the
fixed-timeout flakiness it papered over.
**Validation**
Ran against a development cluster: the DB/app, storage, and VM suites
pass; the three suites that depend on platform features not present on
that cluster (`gateway`, `kuberture`, `external-dns`) and the two
heavyweight `kubernetes-*` suites are exercised by this PR's CI run on a
freshly installed platform.
Note for reviewers: service-port asserts use the `(ports[*].port)`
projection form rather than a number-literal filter (`` ports[?port ==
`N`] ``), because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal
comparisons.
**2026-07-09: reconciled with main (~520 commits of drift)**
- etcd suite re-ported to the v1alpha2 operator contracts (#2859):
`readyMembers=3` gate, pod-label/`/scale`/WorkloadMonitor/metrics/defrag
contracts, plus the `ETCD_E2E_S3_ROUNDTRIP`-gated backup round-trip
driving `examples/backups/etcd`
- 12 new `ingress-hostname-policy` gateway cases ported (apex derived at
runtime from the tenant-root namespace label)
- post-branch bats drift folded in: `mariadb-single` webhook guard,
kafka `c1.small` presets, bucket/harbor 2m BucketClaim fail-fast, bucket
readonly-denial promoted to a hard fail (cosi-driver v0.3.1), qdrant
PVC-reclaim guard (#3059), kuberture fail-loud negations,
SC-fallback-default test (#2872 B1)
- 4 suites that only exist on main since the branch was cut are ported:
`kubernetes-oidc-system`, `kubernetes-oidc-customconfig`,
`securitygroup`, `serviceexposure`
- CI seams: `nightly.yaml` converted to the chainsaw invocation,
`run-kubernetes.sh` reconciled (Talos/CABPT waits, tenant drain, LINSTOR
pool wait, talos-image-cache under `_lib/`), `e2e-capture-dataplane.sh`
replicated in the Chainsaw global catch
### Release note
```release-note
fix(kubernetes): tenant Kubernetes teardown no longer hangs when the cluster has no working nodes — the pre-delete hook now bounds its wait for the in-tenant HelmReleases and force-clears their Flux finalizers on timeout (#3271)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* End-to-end testing is now driven by a unified Chainsaw suite (platform
apps, Kubernetes, networking, security groups, service exposure).
* Test Impact Analysis selects affected suites, with an option to run
the full suite.
* CI produces a JUnit report artifact and collects suite-scoped
diagnostics on failure.
* **Bug Fixes**
* CI E2E now fails fast and emits clearer Kubernetes/HelmRelease and
sandbox bucket state diagnostics.
* Improved teardown reliability with bounded wait + finalizer handling
to avoid hangs.
* **Documentation**
* Updated E2E testing docs and local guidance to reflect the new
Chainsaw-based workflow and suite paths.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.
Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.
The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Rebuilds the e2e test sandbox on current upstream to clear the published
advisories (containerd, helm, go-git, moby/spdystream, Go stdlib) carried by the
stale tool binaries and base layers:
- base: ubuntu 22.04 -> 24.04; netcat -> netcat-openbsd, since `netcat` is a
candidate-less virtual package on 24.04
- kubectl: 1.33.2 -> 1.33.13, staying on the 1.33 line that the e2e control
plane pins at v1.33.12, so client/server skew is unchanged
- talosctl: 1.10.4 -> 1.13.5, aligning the client with the v1.13.x Talos nodes
the sandbox provisions
- helm: 3.18.3 -> 3.21.2 (latest helm 3); the e2e flow drives the v3 CLI, so it
stays on the 3.x line rather than moving to the breaking 4.x
- yq: 4.44.3 -> 4.53.3
- values.yaml: re-pinned to the rebuilt multi-arch digest
crust-gather and cozyhr already track their latest releases; flux and mc install
the current release at build time.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The crust-gather install.sh substitutes the version verbatim into both
the release-tag path and the asset filename:
releases/download/<ver>/kubectl-crust-gather_<ver>_<target>.tar.gz
The release tag is 'v0.15.2', so passing '0.15.2' 404s on the tag path
(curl exit 22) and fails the image build. The 'v0.15.2' asset variant
exists too, so the leading v satisfies both. Verified HTTP 200.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
cozytest.sh runs each .bats as one invocation, exits on the first failing @test, and never invokes bats setup/teardown. Add a single EXIT-trap hook: on failure, snapshot the host AND every nested tenant cluster (tenantkubeconfig-*) with crust-gather into per-app dirs BEFORE cleanup — so each failure is independently inspectable (crust-gather serve) instead of one end-of-suite host-only dump; then ALWAYS run the file's cozy_cleanup() if defined.
run-kubernetes.sh gains cozy_cleanup() (delete tenant clusters, wait for PVCs to free): a failed kubernetes test otherwise leaves tens of GiB of worker-VM PVCs in tenant-test, exhausting the shared tenant-quota and cascade-failing every storage-heavy app after it. The kubernetes .bats now source run-kubernetes.sh at file scope so the parent-shell trap can reach the hook.
Ship pinned crust-gather 0.15.2 in the e2e-sandbox image; fold per-test snapshots into cozyreport.tgz.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The minio client was downloaded without -L, so when dl.min.io served
a 302 redirect, curl wrote the HTML body ("<a href=...>Found</a>") to
/usr/local/bin/mc, chmod +x'd it, and shipped that as the binary. When
the bucket E2E test ran "mc alias set ...", the shell tried to interpret
the HTML as a script and errored with:
/usr/local/bin/mc: 1: cannot open a: No such file
(dash reading "<a" as an input redirect from a file named "a").
The image is rebuilt per CI run from cached layers; the broken mc layer
was cached and reused. Adding -fsSL fixes the download and the textual
change busts the cache so the next build pulls a real ELF binary.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The test-cluster target references non-existent hack/e2e-cluster.bats
file. Remove it and its dependency from the test target.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace pre-rendered static YAML application with direct helm chart
installation in e2e tests. The chart directory with correct values is
already present in the sandbox after pr.patch application.
- Remove CRD/operator artifact upload/download from CI workflow
- Remove copy-installer-manifest target from testing Makefile
- Use helm upgrade --install from local chart in e2e-install-cozystack.bats
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add -talos suffix to the default variant output file for consistency
with -generic and -hosted variants. Update all references in CI
workflows, e2e tests, upload scripts, and testing Makefile.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
- Add pattern rule for building cozypkg binaries per platform
(assets-cozypkg-<os>-<arch>) with checksums generation
- Add Version variable to cozypkg CLI injected via ldflags
- Split manifests into separate cozystack-crds.yaml and
cozystack-operator.yaml files
- Update CI workflow to handle CRDs and operator as separate artifacts
- Update e2e tests and upload script for new manifest structure
- Suppress git describe stderr when no tags exist
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Adds a `k=kubectl` alias and bash completion for kubectl to the
e2e-testing sandbox container to maintainers have an easier time
exec'ing into the CI container when something needs to be debugged.
### Release note
```release-note
[testing] Add k=kubectl alias and enable kubectl completion in the CI
container.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Myasnikov Daniil