The metrics contract runs a one-shot curl Pod and reads a `code=<status>`
marker back through `kubectl run --attach`. Attach connects only after the
Pod reaches Running, so a container that exits before the connection is up
writes its stdout into the void: the step then sees only kubectl's own
"If you don't see a command prompt, try pressing enter." and reports the
metrics endpoint as broken. curl against a Pod IP finishes in milliseconds,
so losing that race is routine, and `--rm` compounded it by deleting the Pod
the instant it exited, taking the log that still held the real output.
The result was a false negative on a healthy cluster, non-deterministic in
the way that reads as a flake because whether attach wins depends on Pod
startup timing rather than on anything under test.
Drop `--rm`, keep the Pod until its result is in hand, and re-read the log
when the attach stream carried no marker. The recovery cannot mask a real
failure: the marker encodes the outcome, so a recovered `code=503` stays a
failure, and a stream that already carried a marker is never second-guessed.
The pre-run delete stays blocking so the re-read cannot find a previous
incarnation's marker, and the post-run delete is guarded so a transient API
error cannot abort the step with the result in hand but unreported.
Pin `--pod-running-timeout` at 90s rather than leaning on kubectl's 1m
default. curlimages/curl is not otherwise used on these nodes, so the probe
pays a cold pull, and a pull that outruns the budget produces the same
markerless output this change exists to explain.
The resolver is split into a sourced helper so the recovery is unit-tested
without a cluster; the kubectl orchestration around it stays covered by the
e2e run. The image-cache probe carries the same recovery inline for the same
race.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The suite was ported from the pre-#2859 etcd.bats and asserted
etcd.aenix.io/v1alpha1 objects that no longer exist on main — every
Test hard-failed at the first assert on an unknown GVK.
- drop the etcd-empty-backup and etcd-backup-schedule scenarios with
their fixtures: the chart no longer renders EtcdBackupSchedule or
etcd-s3-creds (backups moved to the generic backups API), and
legacy-values tolerance is pinned by helm-unittest, not e2e
- rewrite the remaining scenario into etcd-1-contracts, mirroring
main's contracts test: gate on status.readyMembers=3 (Available=True
flips at seed-member discovery and races replica-count contracts),
assert the operator-owned member Pod shape (strict selector,
container name, named metrics port), WorkloadMonitor
operational/availableReplicas, and script the three contracts that
need imperative probes: the /scale subresource, the plaintext-HTTP
metrics probe against the Pod IP, and the on-demand defrag Job
- add etcd-2-backup-roundtrip driving examples/backups/etcd/run-all.sh,
gated on ETCD_E2E_S3_ROUNDTRIP=1 exactly like main (kind CI has no
publicly-trusted S3 endpoint the snapshot agent could verify)
- upgrade catch blocks to the v1alpha2 GVKs plus EtcdMember/
EtcdSnapshot/BackupJob describes and the snapshot-Job Pod logs
- strengthen _lib/etcd-cleanup.sh with main's etcd_drain deltas: probe
the Etcd CR itself (it can linger Terminating and block the next
singleton apply) and use the err-sentinel pattern so a transient API
error is never misread as "deleted"; add etcd_backup_cleanup wrapping
examples/backups/etcd/cleanup.sh for the round-trip's teardown
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd scenarios share the singleton 'etcd' release and relied on the
next scenario's ensure-clean to drain the previous one. Chainsaw runs Tests in
name order (not file order), and v0.2.15 has no test-level finally, so the
last-run scenario leaked its etcd plus the wedged Kamaji DataStore into every
later suite on full/release runs (lexfrei review B3).
Make each scenario self-contained instead: a single run step pre-cleans any
leftover etcd, installs and asserts, then drains in a finally that always runs
(the step is always reached, so teardown fires on success and failure alike).
Extract the shared drain to _lib/etcd-cleanup.sh, removing the ~25-line block
that was copied across all three scenarios.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd Chainsaw scenarios share the singleton release name 'etcd' and
the cluster-scoped DataStore/<namespace> the etcd chart provisions for Kamaji.
On teardown Kamaji leaks its kamaji.clastix.io/TenantControlPlane finalizer on
that DataStore — once the backing etcd is gone the reconcile that would drop it
cannot complete — so the DataStore wedges in Terminating and the etcd
HelmRelease uninstall hangs. In CI this errored every etcd case: the first
scenario's stuck teardown blocked the next scenario's install (HelmRelease
never Ready, "DataStore status: Terminating", etcd pods refusing :2381 because
the CA issuer had no key material).
Make teardown explicit instead of relying on Chainsaw auto-cleanup (which was
waiting on the wedged uninstall): set skipDelete on each Test and have every
scenario's ensure-clean drain the previous one — delete the etcd and
force-clear the orphaned finalizer (no TenantControlPlane uses this
per-namespace test DataStore) in a loop until the HelmRelease and DataStore are
both gone. Each scenario then installs from a clean slate.
Verified on a dev cluster: all three scenarios pass back-to-back where they
previously cascaded.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Aleksei Sviridkin