cozystack-api gives every release it generates a 600s install timeout,
and these asserts gave up at 5-6m. A failed install was therefore
reported as a bare assert timeout while the release was still
installing and had not yet recorded why it failed.
Raise them to 11m, above that floor, so the release's own reason is on
record before Chainsaw gives up. The cost is up to six more minutes on
a red run of each suite; a green run is unchanged. vm-instance-test
stays: vm-instance-rd disables the install wait, so its release goes
Ready as soon as helm has applied.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The etcd suite justified per-scenario teardown with a claim that Chainsaw
runs Tests in name order, not file order. That was wrong, but restating it
as document order is wrong too: at the pinned v0.2.15 Chainsaw guarantees
no order between Tests at all.
The runner registers every Test through testing.T.Run and the body calls
t.Parallel() whenever spec.concurrent is nil or true — the case in both
suites, since neither sets it. Go pauses those subtests and releases them
together once the parent returns, and execution.parallel maps only to
-test.parallel, a concurrency cap rather than an order. So parallel: 1
serialises the Tests without ordering them: not file, not document, not
name. The E2E run on this branch's own head shows it — clickhouse-2 ran
before clickhouse-1 with both etcd Tests interleaved between the clickhouse
pair.
The teardown decision the false claim justified still holds, for a reason
that does not touch order: each etcd scenario is self-contained under
skipDelete: true with a step-level finally, and v0.2.15 has no test-level
finally, so a scenario is reclaimed only if it reclaims itself — true of
whichever Test runs last regardless of order. Drop the ordering clause from
the etcd teardown rationale and the ordering sentence from the clickhouse
header rather than restating either. Comment-only; no behaviour changes.
Verified against Chainsaw v0.2.15 source: pkg/runner/runner.go (t.Run plus
t.Parallel gated on spec.concurrent) and pkg/runner/flags/flags.go
(execution.parallel maps to test.parallel).
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The metrics contract runs a one-shot curl Pod and reads a `code=<status>`
marker back through `kubectl run --attach`. Attach connects only after the
Pod reaches Running, so a container that exits before the connection is up
writes its stdout into the void: the step then sees only kubectl's own
"If you don't see a command prompt, try pressing enter." and reports the
metrics endpoint as broken. curl against a Pod IP finishes in milliseconds,
so losing that race is routine, and `--rm` compounded it by deleting the Pod
the instant it exited, taking the log that still held the real output.
The result was a false negative on a healthy cluster, non-deterministic in
the way that reads as a flake because whether attach wins depends on Pod
startup timing rather than on anything under test.
Drop `--rm`, keep the Pod until its result is in hand, and re-read the log
when the attach stream carried no marker. The recovery cannot mask a real
failure: the marker encodes the outcome, so a recovered `code=503` stays a
failure, and a stream that already carried a marker is never second-guessed.
The pre-run delete stays blocking so the re-read cannot find a previous
incarnation's marker, and the post-run delete is guarded so a transient API
error cannot abort the step with the result in hand but unreported.
Pin `--pod-running-timeout` at 90s rather than leaning on kubectl's 1m
default. curlimages/curl is not otherwise used on these nodes, so the probe
pays a cold pull, and a pull that outruns the budget produces the same
markerless output this change exists to explain.
The resolver is split into a sourced helper so the recovery is unit-tested
without a cluster; the kubectl orchestration around it stays covered by the
e2e run. The image-cache probe carries the same recovery inline for the same
race.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The suite was ported from the pre-#2859 etcd.bats and asserted
etcd.aenix.io/v1alpha1 objects that no longer exist on main — every
Test hard-failed at the first assert on an unknown GVK.
- drop the etcd-empty-backup and etcd-backup-schedule scenarios with
their fixtures: the chart no longer renders EtcdBackupSchedule or
etcd-s3-creds (backups moved to the generic backups API), and
legacy-values tolerance is pinned by helm-unittest, not e2e
- rewrite the remaining scenario into etcd-1-contracts, mirroring
main's contracts test: gate on status.readyMembers=3 (Available=True
flips at seed-member discovery and races replica-count contracts),
assert the operator-owned member Pod shape (strict selector,
container name, named metrics port), WorkloadMonitor
operational/availableReplicas, and script the three contracts that
need imperative probes: the /scale subresource, the plaintext-HTTP
metrics probe against the Pod IP, and the on-demand defrag Job
- add etcd-2-backup-roundtrip driving examples/backups/etcd/run-all.sh,
gated on ETCD_E2E_S3_ROUNDTRIP=1 exactly like main (kind CI has no
publicly-trusted S3 endpoint the snapshot agent could verify)
- upgrade catch blocks to the v1alpha2 GVKs plus EtcdMember/
EtcdSnapshot/BackupJob describes and the snapshot-Job Pod logs
- strengthen _lib/etcd-cleanup.sh with main's etcd_drain deltas: probe
the Etcd CR itself (it can linger Terminating and block the next
singleton apply) and use the err-sentinel pattern so a transient API
error is never misread as "deleted"; add etcd_backup_cleanup wrapping
examples/backups/etcd/cleanup.sh for the round-trip's teardown
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd scenarios share the singleton 'etcd' release and relied on the
next scenario's ensure-clean to drain the previous one. Chainsaw runs Tests in
name order (not file order), and v0.2.15 has no test-level finally, so the
last-run scenario leaked its etcd plus the wedged Kamaji DataStore into every
later suite on full/release runs (lexfrei review B3).
Make each scenario self-contained instead: a single run step pre-cleans any
leftover etcd, installs and asserts, then drains in a finally that always runs
(the step is always reached, so teardown fires on success and failure alike).
Extract the shared drain to _lib/etcd-cleanup.sh, removing the ~25-line block
that was copied across all three scenarios.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd Chainsaw scenarios share the singleton release name 'etcd' and
the cluster-scoped DataStore/<namespace> the etcd chart provisions for Kamaji.
On teardown Kamaji leaks its kamaji.clastix.io/TenantControlPlane finalizer on
that DataStore — once the backing etcd is gone the reconcile that would drop it
cannot complete — so the DataStore wedges in Terminating and the etcd
HelmRelease uninstall hangs. In CI this errored every etcd case: the first
scenario's stuck teardown blocked the next scenario's install (HelmRelease
never Ready, "DataStore status: Terminating", etcd pods refusing :2381 because
the CA issuer had no key material).
Make teardown explicit instead of relying on Chainsaw auto-cleanup (which was
waiting on the wedged uninstall): set skipDelete on each Test and have every
scenario's ensure-clean drain the previous one — delete the etcd and
force-clear the orphaned finalizer (no TenantControlPlane uses this
per-namespace test DataStore) in a loop until the HelmRelease and DataStore are
both gone. Each scenario then installs from a clean slate.
Verified on a dev cluster: all three scenarios pass back-to-back where they
previously cascaded.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Aleksei Sviridkin