A nested <name>-system release takes its install timeout from the
template that renders it, not from cozystack-api, and a Package release
takes it from cozystack-operator. The external-dns, openbao and qdrant
templates set 10m and harbor's sets 15m; the operator gives 10m. Their
asserts sat at 5m or 10m, so a failed install reported as a bare
timeout. Raise each above its own floor: 11m, and 16m for
harbor-test-system. harbor-test was on its 600s floor exactly, which
expires first because its clock starts before the install's; it goes
to 11m with the generated parents here.
kafka keeps its 6m: kafka-rd sets a 7h install timeout that no budget
inside the E2E job cap can clear, and the assert now says so.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The checklist entry had grown into a derivation and buried the rule it
states. The rule stays in the checklist and the derivation moves to a
section of its own.
The derivation under-budgeted in two ways. It stated the 10m install
timeout as universal, while releases an app chart renders itself take
their template's spec.install.timeout, then its spec.timeout, or
helm-controller's 5m where neither is set. And it restated which kinds
kstatus gates on, which is easy to get wrong, so it now points at the
upstream table instead.
The harbor comments now name the floor each of their asserts actually
sits against, and the opensearch comment stops citing a 5-6m standard
the checklist no longer states.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
helm-controller skips CRD upgrades by default, so an upgraded cluster would
keep the old spotahome schema and prune the fork's spec.engine field on
admission. The schema diff is additive, so CreateReplace is safe. Also refresh
the harbor e2e wait comment for the freshworks operator.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
The qdrant and harbor post-delete hooks guarded every delete with
`|| echo "WARNING: ..." >&2`, set no `-e`, and ended on an
unconditional completion message, so the container exited 0 on
every path. Helm read that as a successful hook and the
hook-succeeded delete policy removed the Job together with the pod
that carried the warning, leaving an uninstall that reclaimed
nothing indistinguishable from one that had nothing to reclaim.
Count the failures instead and set the exit code at the end, so a
failed delete does not skip the deletes that follow it. An object
that is already gone stays a success: every delete keeps
--ignore-not-found, which exits 0 in that case. This is the shape
packages/apps/mariadb already uses for the same problem.
Both hooks also consumed the output of `kubectl get` directly --
harbor by iterating it, qdrant by piping it into grep, whose status
the pipeline then reports. A listing that failed therefore arrived
as an empty list and the sweep was skipped rather than reported.
Each listing is now captured before it is used, so a listing that
failed and one that returned nothing are different events.
A hook that exits non-zero keeps its Job and pods, which is the
point: they hold the log. Give the Job one retry rather than the
Kubernetes default of six or none at all. Every delete is
idempotent, so a second attempt re-issues an API call rather than
re-running any logic, and one short backoff is what separates a
transient apiserver error from a real one -- these failures become
visible for the first time here, and a signal that fires on noise
stops being read, including on the day it is right. Six is the far
end of the same mistake: ten minutes of backoff across seven pods
outlasts the hook timeout, and Helm reports that timeout instead of
the hook's own error.
The qdrant Chainsaw suite explained its post-delete pod collector
by quoting the guard string removed here and concluding the pod is
always collected away. That is now backwards: a failed cleanup is
exactly when the pod survives and the collector is the evidence.
Rewrite the note, and the sentence in the harbor suite pointing at
it.
hack/app-cleanup-hooks_test.bats renders both scripts and runs them
against a fake kubectl. There are eight failure paths across the two
hooks -- six in harbor, two in qdrant -- and each has a case that
goes red when that one path alone is reverted to a swallowed
warning. Message assertions are anchored on the hook's own `ERROR: `
prefix, because the fake kubectl names the object in its own
diagnostic too and an unanchored match is satisfied by the fixture.
The helm-unittest suites pin the retry limit and the hook delete
policy: adding hook-failed to that annotation hands the Job back to
Helm to delete on failure and destroys the evidence this change
exists to keep, and until these asserts that reversal left both
suites green.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The harbor, openbao and qdrant app charts deploy their workload through
a nested HelmRelease named `<release>-system`, and pass
`fullnameOverride: <release>` down to it. The inner chart therefore
stamps app.kubernetes.io/instance from its own release name, with the
suffix, while object names come from the override, without it. The
catch blocks selected on the outer release name, so podLogs matched no
pods and every failed run collected an empty result.
A pod name says nothing about its instance label; the two come from
different sources on purpose. In qdrant the unsuffixed value is
therefore not retired but kept as a collector of its own: the parent
chart's PVC-cleanup Job labels its pod template with the parent release
name, and that suite deletes the parent inside a `try:`, so the Job is
in scope while the test still runs. The comment there is explicit about
how narrow that window is -- the hook always exits 0, so Helm calls it
succeeded and `hook-succeeded` takes the pod away; what is left is an
uninstall that hangs with Helm still waiting on it.
Harbor renders the same hook and does not get the same collector. It
has no `delete` op, so its parent release is uninstalled only in
Chainsaw's cleanup phase, which no catch can observe: catch is a defer
inside the step, while the cleaner runs from t.Cleanup, after the test
function returns. That collector would be the same permanently-empty
capture this commit removes. A comment records the asymmetry so the
next reader does not restore it for symmetry. openbao renders no such
hook at all.
Each workload value is now also pinned on a resource in an assert the
suite already runs, so the chart renaming a label fails a step instead
of quietly emptying a capture. The comments state what that does and
does not cover: it does not catch a wrong value typed into a selector,
and it catches nothing when the run fails before the assert.
Harbor's database was reached by no collector at all. It is selected on
`cnpg.io/cluster`, CNPG's own primary label, rather than on the
instance label CNPG merely derives from the Cluster name -- the primary
label has nothing to drift from, and it is what the postgres suite and
the WorkloadMonitors already use.
Each lookup keeps stderr apart from the value it captures. kubectl exits
0 while logging a klog line about an API group it could not reach, and
an aggregated APIService in that state is what a failed run usually
looks like; folded into the value, that line makes the "nothing found"
branch unreachable and prints as though it were a result.
Telling those outcomes apart is one helper in hack/e2e-chainsaw/_lib/
rather than a copy per call site, separated from the kubectl call so it
can be pinned without a cluster, the way etcd-probe.sh already is. Both
the census and the qdrant cleanup listing report through it, differing
only in wording, so the discrimination is implemented and tested once.
The tests cover each outcome for both wordings, including the zero exit
with a klog line on stderr -- the case a folded stream turned into a
row.
A collector that matches nothing cannot say so: podLogs is silent, and
`kubectl get -l` exits zero on an empty match. Each test-wide catch
prints the instance label of every pod in the namespace, and the qdrant
cleanup catch reports its three outcomes apart from a single call --
printing what that call returned rather than running it again, which
could put a healthy listing under a failure heading.
That census runs on a degraded cluster, where kubectl otherwise retries
discovery against an unreachable apiserver until something kills the
script, so every lookup carries `--request-timeout`. Measured against an
unreachable apiserver, a bounded call costs about five times the flag,
since kubectl retries discovery five times before it gives up. Each
script then carries an explicit op timeout sized above its own calls --
2m for the single-call censuses, 5m for the qdrant cleanup listing,
which makes three. An inner budget is what should fire: a SIGKILL at the
op boundary yields nothing, where a kubectl that gives up leaves an
error to print.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When harbor's redis sentinels never converge, the suite times out on
the HelmRelease assert with no evidence of why: the catch pod-log
selector matches neither the redis-operator (it runs in another
namespace) nor the rfr-/rfs- pods, and the RedisFailover CRD ships no
status subresource to describe-assert on. Add a RedisFailover describe
plus a best-effort script that dumps the operator log and the
rfr-/rfs- pod states and logs, so the stalled reconcile gate is
visible in the failure output.
Also widen the harbor-test HR wait from 6m to 10m: readiness is gated
on the operator electing a redis master, which normally takes 2-3m but
runs long on a CPU-committed runner. This matches the
harbor-test-system budget in the same suite and the justified longer
waits allowed by the e2e conventions.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Fold main's post-branch bucket/harbor bats drift into the Chainsaw
suites:
- BucketClaim bucketReady budgets tightened to 2m (bucket had the 5m
default, harbor a stale 10m with the reversed 'can lag on a loaded
runner' rationale): the objectstorage-controller requeues the claim
until the backend Bucket is ready (0f1cc548b), so readiness converges
in tens of seconds and a claim not ready after 2m is frozen — the
tight bound surfaces the propagation regression instead of masking
it (63d14e748)
- the bucket viewer-upload soft-xfail is promoted back to a hard
failure: accessPolicy=readonly enforcement landed with
seaweedfs-cosi-driver v0.3.1 (#1921, eab9b034f), so a successful
readonly upload is now a regression, exactly what the suite's own
TODO called for
- both catch blocks gain the backend view (cluster-scoped Buckets +
cozy-objectstorage-controller pods) where a frozen claim actually
surfaces
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw port capped the harbor-test-registry BucketClaim bucketReady wait
at the default 5m, half the 600s the BATS original used. seaweedfs/COSI
provisioning can lag several minutes on a loaded runner, so the dependent
harbor-test-system HelmRelease keeps its 10m budget while the BucketClaim it
depends on was capped at 5m — a latent false-flake. Restore 10m
(docs/agents/e2e-testing.md section 6).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Aleksei Sviridkin