Files
Aleksei Sviridkin 3fc8e524b2 fix(e2e): select catch pod logs by the nested release instance label
The harbor, openbao and qdrant app charts deploy their workload through
a nested HelmRelease named `<release>-system`, and pass
`fullnameOverride: <release>` down to it. The inner chart therefore
stamps app.kubernetes.io/instance from its own release name, with the
suffix, while object names come from the override, without it. The
catch blocks selected on the outer release name, so podLogs matched no
pods and every failed run collected an empty result.

A pod name says nothing about its instance label; the two come from
different sources on purpose. In qdrant the unsuffixed value is
therefore not retired but kept as a collector of its own: the parent
chart's PVC-cleanup Job labels its pod template with the parent release
name, and that suite deletes the parent inside a `try:`, so the Job is
in scope while the test still runs. The comment there is explicit about
how narrow that window is -- the hook always exits 0, so Helm calls it
succeeded and `hook-succeeded` takes the pod away; what is left is an
uninstall that hangs with Helm still waiting on it.

Harbor renders the same hook and does not get the same collector. It
has no `delete` op, so its parent release is uninstalled only in
Chainsaw's cleanup phase, which no catch can observe: catch is a defer
inside the step, while the cleaner runs from t.Cleanup, after the test
function returns. That collector would be the same permanently-empty
capture this commit removes. A comment records the asymmetry so the
next reader does not restore it for symmetry. openbao renders no such
hook at all.

Each workload value is now also pinned on a resource in an assert the
suite already runs, so the chart renaming a label fails a step instead
of quietly emptying a capture. The comments state what that does and
does not cover: it does not catch a wrong value typed into a selector,
and it catches nothing when the run fails before the assert.

Harbor's database was reached by no collector at all. It is selected on
`cnpg.io/cluster`, CNPG's own primary label, rather than on the
instance label CNPG merely derives from the Cluster name -- the primary
label has nothing to drift from, and it is what the postgres suite and
the WorkloadMonitors already use.

Each lookup keeps stderr apart from the value it captures. kubectl exits
0 while logging a klog line about an API group it could not reach, and
an aggregated APIService in that state is what a failed run usually
looks like; folded into the value, that line makes the "nothing found"
branch unreachable and prints as though it were a result.

Telling those outcomes apart is one helper in hack/e2e-chainsaw/_lib/
rather than a copy per call site, separated from the kubectl call so it
can be pinned without a cluster, the way etcd-probe.sh already is. Both
the census and the qdrant cleanup listing report through it, differing
only in wording, so the discrimination is implemented and tested once.
The tests cover each outcome for both wordings, including the zero exit
with a klog line on stderr -- the case a folded stream turned into a
row.

A collector that matches nothing cannot say so: podLogs is silent, and
`kubectl get -l` exits zero on an empty match. Each test-wide catch
prints the instance label of every pod in the namespace, and the qdrant
cleanup catch reports its three outcomes apart from a single call --
printing what that call returned rather than running it again, which
could put a healthy listing under a failure heading.

That census runs on a degraded cluster, where kubectl otherwise retries
discovery against an unreachable apiserver until something kills the
script, so every lookup carries `--request-timeout`. Measured against an
unreachable apiserver, a bounded call costs about five times the flag,
since kubectl retries discovery five times before it gives up. Each
script then carries an explicit op timeout sized above its own calls --
2m for the single-call censuses, 5m for the qdrant cleanup listing,
which makes three. An inner budget is what should fire: a SIGKILL at the
op boundary yields nothing, where a kubectl that gives up leaves an
error to print.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-05 16:46:50 +03:00
..