## What this PR does
The etcd suite's metrics contract failed in CI with `metrics endpoint
did not return 200 over http on the 'metrics' port; got: If you don't
see a command prompt, try pressing enter.`
The probe runs a one-shot curl Pod and reads a `code=<status>` marker
back off its output via `kubectl run --rm --restart=Never --attach`.
`--attach` only establishes its connection once the Pod reaches Running,
so a container that exits in milliseconds — which is what curl against a
Pod IP does — writes its stdout into the void, and `--rm` then destroys
the Pod carrying the log that still held it. The step reports the
metrics endpoint as broken on a cluster that was serving it correctly
throughout.
The evidence that this was output loss rather than an endpoint failure
is that the captured output contained no `code=` at all. A genuine
connection failure would still have printed `code=000` from `-w`, plus
curl's own error text from `-fsS`, and both were being captured.
Meanwhile the probe Pod's events show a clean run, all three etcd
members were up without restarts, and every preceding assert in the
suite passed.
The fix drops `--rm`, re-reads the Pod log when the attach stream
carried no marker, and deletes the Pod explicitly afterwards. This is
the same treatment `hack/e2e-chainsaw/_lib/talos-image-cache.sh` already
applies to the identical race — that probe documents it inline; the etcd
one was simply never given it. A `--pod-running-timeout` is set for the
same reason it is there: the probe image is used nowhere else in the
tree, so it pays a cold pull on every run and kubectl's one-minute
default was a second way to end up with no marker.
The recovery cannot turn a real failure green. The marker encodes the
outcome, so a `code=503` recovered from the log still fails the step; an
attach stream that already carried a marker is never second-guessed; and
the pre-run delete blocks, so a stale Pod's log cannot be read in place
of this run's. Verified by running the probe's exact curl invocation
against a 503 and against a dead port: the first yields `code=503`, the
second `code=000` plus curl's error, and both fail. The probe still runs
exactly once — this is output recovery, not a retry.
Helper extracted to `hack/e2e-chainsaw/_lib/etcd-probe.sh` with unit
coverage in `hack/etcd-probe_test.bats`.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved etcd metrics probe reliability by recovering probe results
when attached output is incomplete.
* Preserved meaningful probe and Kubernetes error details, including
non-success status codes.
* Improved probe pod timing and cleanup to reduce intermittent
end-to-end test failures.
* **Tests**
* Added coverage for successful recovery, failed probes, missing output,
and log retrieval errors.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The tenant Kubernetes e2e pinned controlPlane.apiServer.resourcesPreset to
the legacy "small" preset, capping the apiserver at 1 CPU / 512Mi -- a
quarter of the memory the chart itself ships as the default (c1.medium,
2 CPU / 2Gi).
512Mi does not hold a two-node tenant cluster running the full addon set.
Once the workers join, cilium, coredns, metrics-server, csi, ingress-nginx
and VPA all establish watches at roughly the same moment, and the apiserver
crosses the limit: one replica is killed for failing its liveness probe as
reclaim stalls it, the other is OOMKilled outright. Both replicas restart
within about two minutes of each other, so any tenant HelmRelease that is
mid-install and waiting on a DaemonSet rollout observes 'InProgress' for
its entire timeout and fails, then re-enters a fresh install on remediation.
Which release loses is a matter of timing rather than of the release, which
is why the failure moved between the csi, ingress-nginx and cilium releases
and looked like an unrelated dependency problem each time.
Use the chart default so the test exercises a control plane sized the way a
real one is, instead of one held at a quarter of the shipped default.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The metrics contract runs a one-shot curl Pod and reads a `code=<status>`
marker back through `kubectl run --attach`. Attach connects only after the
Pod reaches Running, so a container that exits before the connection is up
writes its stdout into the void: the step then sees only kubectl's own
"If you don't see a command prompt, try pressing enter." and reports the
metrics endpoint as broken. curl against a Pod IP finishes in milliseconds,
so losing that race is routine, and `--rm` compounded it by deleting the Pod
the instant it exited, taking the log that still held the real output.
The result was a false negative on a healthy cluster, non-deterministic in
the way that reads as a flake because whether attach wins depends on Pod
startup timing rather than on anything under test.
Drop `--rm`, keep the Pod until its result is in hand, and re-read the log
when the attach stream carried no marker. The recovery cannot mask a real
failure: the marker encodes the outcome, so a recovered `code=503` stays a
failure, and a stream that already carried a marker is never second-guessed.
The pre-run delete stays blocking so the re-read cannot find a previous
incarnation's marker, and the post-run delete is guarded so a transient API
error cannot abort the step with the result in hand but unreported.
Pin `--pod-running-timeout` at 90s rather than leaning on kubectl's 1m
default. curlimages/curl is not otherwise used on these nodes, so the probe
pays a cold pull, and a pull that outruns the budget produces the same
markerless output this change exists to explain.
The resolver is split into a sourced helper so the recovery is unit-tested
without a cluster; the kubectl orchestration around it stays covered by the
e2e run. The image-cache probe carries the same recovery inline for the same
race.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Non-blocking review notes from #2826:
- Add `# shellcheck shell=bash` to the four sourced `_lib/*.sh` helpers
(etcd-cleanup, remediation-guard, run-kubernetes, talos-image-cache) so
shellcheck stops flagging SC2148 (no shell directive) — they are only ever
`. sourced`, so there is no runtime interpreter to infer from a shebang.
- Add a `# TODO(chainsaw>0.2.15)` marker at the postgres Service asserts
explaining the `(ports[0].port)`/`(ports[*].port)` projection is a v0.2.15
number-literal-filter workaround (and that `ports[0]` checks only the first
port), so it gets reverted on a chainsaw bump; cross-references the same
workaround in kafka/mariadb/harbor and the e2e-testing.md gotchas section.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The in-sandbox Talos image mirror (#3244) is unreachable by the tenant
workers it serves, so the kubernetes-* e2e tests fail deterministically at
the 12m node-join wait instead of relieving the public-factory flake (#3231).
Root cause (from #3254 by @lexfrei, verified against the failing run's
crust-gather snapshot): tenant namespaces run a default-deny Cilium egress
(allow-internal/external-communication + the <tenant>-egress CCNP) that
permits the world entity and the tenant tree but NOT an arbitrary kube-system
Service, so a worker's CDI importer resolves talos-image-cache.kube-system.svc
yet its TCP connect to the ClusterIP is silently dropped and no disk imports.
#3244's gate never caught this because it probed the server over localhost
inside the serve pod, which passes regardless of cross-pod reachability.
Port of #3254 to the Chainsaw layout (hack/e2e-chainsaw/_lib/):
- ship a tightly-scoped CiliumClusterwideNetworkPolicy (importer pods in
tenant-test -> mirror pod in kube-system), applied at point-of-use once
Cilium CRDs exist; excluded from the pre-Cilium install apply via yq.
- replace the readiness probe with a throwaway pod in tenant-test labelled
cdi.kubevirt.io=importer that curls the ClusterIP with a range request and
must get 206 - the exact egress path a real importer faces; tenants use the
mirror only when that end-to-end check passes, else fall back to the public
factory, so it can only help, never make CI worse.
- unit test pinning the manifest split, probe/policy label agreement, the
strict 206 gate, and the --overrides JSON builder.
Also adds (a2) DataVolume import-stage diagnostics to the node-join failure
block in run-kubernetes.sh (importer logs + a ClusterIP re-probe via
talos_image_cache_diagnose) so the import-stage failure mode is legible.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Fold the in-sandbox Talos image cache work into the Chainsaw layout:
the new shared helper relocates to hack/e2e-chainsaw/_lib/
talos-image-cache.sh and run-kubernetes.sh sources it from there (the
ouroboros DNS-check no-retry rework merges clean). The OIDC bats are
resolved as deleted again; their Chainsaw fixtures deliberately skip
the talos.imageFactoryURL injection — the helper only emits it when
the cache is up, and the zero-worker render-side tests instantiate no
DataVolume imports for it to redirect.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The suite was ported from the pre-#2859 etcd.bats and asserted
etcd.aenix.io/v1alpha1 objects that no longer exist on main — every
Test hard-failed at the first assert on an unknown GVK.
- drop the etcd-empty-backup and etcd-backup-schedule scenarios with
their fixtures: the chart no longer renders EtcdBackupSchedule or
etcd-s3-creds (backups moved to the generic backups API), and
legacy-values tolerance is pinned by helm-unittest, not e2e
- rewrite the remaining scenario into etcd-1-contracts, mirroring
main's contracts test: gate on status.readyMembers=3 (Available=True
flips at seed-member discovery and races replica-count contracts),
assert the operator-owned member Pod shape (strict selector,
container name, named metrics port), WorkloadMonitor
operational/availableReplicas, and script the three contracts that
need imperative probes: the /scale subresource, the plaintext-HTTP
metrics probe against the Pod IP, and the on-demand defrag Job
- add etcd-2-backup-roundtrip driving examples/backups/etcd/run-all.sh,
gated on ETCD_E2E_S3_ROUNDTRIP=1 exactly like main (kind CI has no
publicly-trusted S3 endpoint the snapshot agent could verify)
- upgrade catch blocks to the v1alpha2 GVKs plus EtcdMember/
EtcdSnapshot/BackupJob describes and the snapshot-Job Pod logs
- strengthen _lib/etcd-cleanup.sh with main's etcd_drain deltas: probe
the Etcd CR itself (it can linger Terminating and block the next
singleton apply) and use the err-sentinel pattern so a transient API
error is never misread as "deleted"; add etcd_backup_cleanup wrapping
examples/backups/etcd/cleanup.sh for the round-trip's teardown
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The etcd suite still failed every run after the clean-after-each rework:
etcd-empty-backup (first to run) passed, but etcd-basic and etcd-backup-schedule
wedged at EtcdCluster WaitingForFirstQuorum until the chart's helm --wait timed
out (InstallFailed).
Root cause, from the failed run's crust-gather snapshot: the StatefulSet's
data-etcd-{0,1,2} PVCs are retained on uninstall and the drain never reclaimed
them, so each later scenario rebound the first scenario's volumes (identical PV
UIDs and a single creationTimestamp across all three scenarios) and booted the
previous etcd's data dir and member identity — never forming a fresh first
quorum. The pods were individually Ready (StatefulSet readyReplicas: 3) with all
certs valid, which is why it looked like an operator/network issue rather than
stale storage.
Delete the data PVCs (app.kubernetes.io/managed-by=etcd-operator) in
etcd_cleanup and wait for them to clear alongside the HR and DataStore, so each
scenario starts from empty storage.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd scenarios share the singleton 'etcd' release and relied on the
next scenario's ensure-clean to drain the previous one. Chainsaw runs Tests in
name order (not file order), and v0.2.15 has no test-level finally, so the
last-run scenario leaked its etcd plus the wedged Kamaji DataStore into every
later suite on full/release runs (lexfrei review B3).
Make each scenario self-contained instead: a single run step pre-cleans any
leftover etcd, installs and asserts, then drains in a finally that always runs
(the step is always reached, so teardown fires on success and failure alike).
Extract the shared drain to _lib/etcd-cleanup.sh, removing the ~25-line block
that was copied across all three scenarios.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
run_kubernetes_test applies the tenant Kubernetes CR imperatively (kubectl), so
Chainsaw's auto-cleanup — which only tracks resources it apply/create-d — never
reclaims it. On success the script's tail deleted it, but on failure nothing did:
cozy_cleanup() used to be invoked by hack/cozytest.sh's trap, and the Chainsaw
harness has no equivalent caller, so a failed kubernetes-latest/previous run
leaked the tenant cluster (tens of GiB of worker PVCs) into the shared
tenant-test namespace and starved later suites.
Add a `finally` block to both kubernetes suites that runs cozy_cleanup. `finally`
always executes, after the `catch` phase, so the ordering matches the BATS
harness: pass -> cleanup; fail -> crust-gather (tenant trap + host catch) ->
cleanup. cozy_cleanup stays best-effort (|| true) so a slow tenant teardown never
flips a passing test red.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.
Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.
The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
myasnikovdaniil