Chainsaw v0.2.15 ends an assert or error operation on the first API
error other than NotFound. kube-apiserver answers a GET with 500 when
etcd changes its leader, so a leader change that lands inside a poll
fails the step long before its own timeout, on whichever suite happens
to be polling.
The sandbox image now builds v0.2.15 from source with the fix proposed
upstream in kyverno/chainsaw#2761, applied unchanged, until a chainsaw
release carries it. The build runs the tests of the patched packages
and a test that drives chainsaw built from the patched tree against a
fake apiserver: a run of 500s must not fail an assert, and a
persistent one must fail it at its timeout rather than at the first
error.
Closes: #4720
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The flux install piped `curl -sSL` into bash, so on a 4xx/5xx the error
page was piped into the shell and curl's exit code was lost. Download the
installer to a file with `curl -fsSL` first, then run it, so an HTTP error
fails the build step instead of feeding bash a bad script.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The e2e-sandbox image fetched flux through `install.sh` with no version,
so the build resolved the installer's floating "latest". Add an
ARG FLUX_VERSION, matching the neighbouring COZYHR_VERSION and
CRUST_GATHER_VERSION pins, and pass it to install.sh so the sandbox's
flux CLI is fixed and reproducible like the rest of the image's tools.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
dl.min.io now answers 410 Gone for every community mc build, since
MinIO archived the project, so the sandbox image no longer builds.
The final mc release is still published as a GitHub release asset.
Pin it and verify the per-arch checksum, because the old install
trusted whatever the URL served.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Migrates the E2E **app** test suite from BATS to [Kyverno
Chainsaw](https://github.com/kyverno/chainsaw) — a declarative,
Kubernetes-native E2E framework (CNCF, part of the Kyverno project;
successor to KUTTL) — and wires it into CI as the replacement for the
per-app BATS loop. This grows the original pilot (postgres + bucket)
into the full suite.
All 22 app suites are ported under `hack/e2e-chainsaw/` and the
`hack/e2e-apps/*.bats` files are removed:
`postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `qdrant`,
`clickhouse`, `kafka`, `etcd`, `openbao`, `harbor`, `foundationdb`,
`external-dns`, `kuberture`, `vminstance`, `gateway`,
`kubernetes-latest`, `kubernetes-previous`, `kubernetes-oidc-system`,
`kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure`.
**Approach**
- **Declarative** suites assert on `status.conditions` and concrete
fields. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait`
pair that appeared ~100 times collapses into a single `assert` that
polls existence and state together, with a structured diff on failure
and automatic `events`/`describe`/`podLogs` capture via `catch`
(previously only `harbor.bats` did this, by hand).
- **Imperative** suites keep their logic in `script` steps: `openbao`
init/unseal, `kuberture` external-dns split-horizon probes,
`vminstance`, the `gateway` admission/impersonation cases, and
`kubernetes-latest`/`previous`, which wrap the relocated
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` verbatim (Kamaji bring-up,
LB/NFS/ouroboros checks).
- `gateway` tests derive the tenant apex from the namespace
`namespace.cozystack.io/host` label at runtime, so they are
host-independent.
**CI**
- The `e2e` job now runs `chainsaw test hack/e2e-chainsaw/` via a new
`test-chainsaw` target and uploads the JUnit `chainsaw-report.xml`.
- The `chainsaw` binary is added to the e2e-sandbox image.
- `install-cozystack` and `test-openapi` stay BATS (cluster bootstrap +
OpenAPI checks).
- The per-app 3-retry loop is dropped — assertion polling replaces the
fixed-timeout flakiness it papered over.
**Validation**
Ran against a development cluster: the DB/app, storage, and VM suites
pass; the three suites that depend on platform features not present on
that cluster (`gateway`, `kuberture`, `external-dns`) and the two
heavyweight `kubernetes-*` suites are exercised by this PR's CI run on a
freshly installed platform.
Note for reviewers: service-port asserts use the `(ports[*].port)`
projection form rather than a number-literal filter (`` ports[?port ==
`N`] ``), because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal
comparisons.
**2026-07-09: reconciled with main (~520 commits of drift)**
- etcd suite re-ported to the v1alpha2 operator contracts (#2859):
`readyMembers=3` gate, pod-label/`/scale`/WorkloadMonitor/metrics/defrag
contracts, plus the `ETCD_E2E_S3_ROUNDTRIP`-gated backup round-trip
driving `examples/backups/etcd`
- 12 new `ingress-hostname-policy` gateway cases ported (apex derived at
runtime from the tenant-root namespace label)
- post-branch bats drift folded in: `mariadb-single` webhook guard,
kafka `c1.small` presets, bucket/harbor 2m BucketClaim fail-fast, bucket
readonly-denial promoted to a hard fail (cosi-driver v0.3.1), qdrant
PVC-reclaim guard (#3059), kuberture fail-loud negations,
SC-fallback-default test (#2872 B1)
- 4 suites that only exist on main since the branch was cut are ported:
`kubernetes-oidc-system`, `kubernetes-oidc-customconfig`,
`securitygroup`, `serviceexposure`
- CI seams: `nightly.yaml` converted to the chainsaw invocation,
`run-kubernetes.sh` reconciled (Talos/CABPT waits, tenant drain, LINSTOR
pool wait, talos-image-cache under `_lib/`), `e2e-capture-dataplane.sh`
replicated in the Chainsaw global catch
### Release note
```release-note
fix(kubernetes): tenant Kubernetes teardown no longer hangs when the cluster has no working nodes — the pre-delete hook now bounds its wait for the in-tenant HelmReleases and force-clears their Flux finalizers on timeout (#3271)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* End-to-end testing is now driven by a unified Chainsaw suite (platform
apps, Kubernetes, networking, security groups, service exposure).
* Test Impact Analysis selects affected suites, with an option to run
the full suite.
* CI produces a JUnit report artifact and collects suite-scoped
diagnostics on failure.
* **Bug Fixes**
* CI E2E now fails fast and emits clearer Kubernetes/HelmRelease and
sandbox bucket state diagnostics.
* Improved teardown reliability with bounded wait + finalizer handling
to avoid hangs.
* **Documentation**
* Updated E2E testing docs and local guidance to reflect the new
Chainsaw-based workflow and suite paths.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.
Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.
The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Rebuilds the e2e test sandbox on current upstream to clear the published
advisories (containerd, helm, go-git, moby/spdystream, Go stdlib) carried by the
stale tool binaries and base layers:
- base: ubuntu 22.04 -> 24.04; netcat -> netcat-openbsd, since `netcat` is a
candidate-less virtual package on 24.04
- kubectl: 1.33.2 -> 1.33.13, staying on the 1.33 line that the e2e control
plane pins at v1.33.12, so client/server skew is unchanged
- talosctl: 1.10.4 -> 1.13.5, aligning the client with the v1.13.x Talos nodes
the sandbox provisions
- helm: 3.18.3 -> 3.21.2 (latest helm 3); the e2e flow drives the v3 CLI, so it
stays on the 3.x line rather than moving to the breaking 4.x
- yq: 4.44.3 -> 4.53.3
- values.yaml: re-pinned to the rebuilt multi-arch digest
crust-gather and cozyhr already track their latest releases; flux and mc install
the current release at build time.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The crust-gather install.sh substitutes the version verbatim into both
the release-tag path and the asset filename:
releases/download/<ver>/kubectl-crust-gather_<ver>_<target>.tar.gz
The release tag is 'v0.15.2', so passing '0.15.2' 404s on the tag path
(curl exit 22) and fails the image build. The 'v0.15.2' asset variant
exists too, so the leading v satisfies both. Verified HTTP 200.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
cozytest.sh runs each .bats as one invocation, exits on the first failing @test, and never invokes bats setup/teardown. Add a single EXIT-trap hook: on failure, snapshot the host AND every nested tenant cluster (tenantkubeconfig-*) with crust-gather into per-app dirs BEFORE cleanup — so each failure is independently inspectable (crust-gather serve) instead of one end-of-suite host-only dump; then ALWAYS run the file's cozy_cleanup() if defined.
run-kubernetes.sh gains cozy_cleanup() (delete tenant clusters, wait for PVCs to free): a failed kubernetes test otherwise leaves tens of GiB of worker-VM PVCs in tenant-test, exhausting the shared tenant-quota and cascade-failing every storage-heavy app after it. The kubernetes .bats now source run-kubernetes.sh at file scope so the parent-shell trap can reach the hook.
Ship pinned crust-gather 0.15.2 in the e2e-sandbox image; fold per-test snapshots into cozyreport.tgz.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The minio client was downloaded without -L, so when dl.min.io served
a 302 redirect, curl wrote the HTML body ("<a href=...>Found</a>") to
/usr/local/bin/mc, chmod +x'd it, and shipped that as the binary. When
the bucket E2E test ran "mc alias set ...", the shell tried to interpret
the HTML as a script and errored with:
/usr/local/bin/mc: 1: cannot open a: No such file
(dash reading "<a" as an input redirect from a file named "a").
The image is rebuilt per CI run from cached layers; the broken mc layer
was cached and reused. Adding -fsSL fixes the download and the textual
change busts the cache so the next build pulls a real ELF binary.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds a `k=kubectl` alias and bash completion for kubectl to the
e2e-testing sandbox container to maintainers have an easier time
exec'ing into the CI container when something needs to be debugged.
### Release note
```release-note
[testing] Add k=kubectl alias and enable kubectl completion in the CI
container.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced a new script for managing Helm releases in Kubernetes,
including installation and status monitoring.
- Added a configuration file for tenant settings, enabling monitoring
and SeaweedFS.
- Enhanced PostgreSQL initialization script to manage database roles and
privileges dynamically.
- Added a new local pre-commit hook for version map checks.
- **Bug Fixes**
- Updated pre-commit hooks for consistent formatting.
- **Tests**
- Improved testing capabilities for applications in a Kubernetes
environment with new Makefile targets.
- Enhanced Docker image with tools for YAML and JSON processing.
- Updated testing image to the latest version for improved performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Co-authored-by: Andrei Kvapil <kvapss@gmail.com>
This PR introduces new functionality for running e2e-tests in
k8s-cluster.
`make test` from a root invokes deploying of new sandbox for testing
cozystack.
from `packages/core/testing`:
`make test` - runs the end-to-end tests.
`make exec` - opens an interactive shell in the sandbox container.
`make login` - downloads the kubeconfig into a temporary directory and
runs a shell with the sandbox environment; mirrord must be installed.
`make proxy` - enables a SOCKS5 proxy; mirrord and gost must be
installed.
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>