Commit Graph
5 Commits
Author SHA1 Message Date
Myasnikov Daniil cc942fed68 test(tests): run unit suites with Bats and retain POSIX coverage
The shell translator rewrites test bodies and loses Bats reporting.
The unit lane needs complete TAP results and JUnit reports without
changing how live-cluster suites run.

Sourced production helpers still need a dash pass, and unit test bodies
need nounset without exporting shell options into subprocesses.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 11:04:00 +03:00
Aleksei Sviridkin 7e2db4da7e fix(e2e): bound every remaining read in the report collector
Four sections of hack/cozyreport.sh read with a wall-clock ceiling and
the rest -- nodes, namespaces, events, HelmReleases, packages,
packagesources, PVCs, cert-manager, COSI, kamaji, the Talos node reads
and every module gate -- read with none. The tarball is written on the
collector's last line, so a single read that never returns does not
truncate the report, it loses it, on the run where it is the only
evidence left.

Each read now goes through the shape that fits it: the existing bounded
reader for a read whose output is a file, the existing selector for one
that feeds a loop, and a new probe for the `if kubectl get deploy ...`
gates. A gate is a read like any other, and an unbounded one is worse
than the reads it guards, since the module below it never starts. Its
cut-off is recorded, because skipping the module leaves the same empty
tree as a cluster that never had the component; a gate that answers "not
installed" is recorded nowhere, since that is the cluster answering
correctly and a marker firing on every install without KubeVirt is one a
reader learns to skip. The reads that stay on the bare wrapper say so
where they sit; what they share is that the output is not the reader's
to write, going into a variable or through a pipeline that transforms
it, so there is no moment at which a marker could attach.

The header claimed a per-section list of what was bounded, and the tests
froze that list plus a per-section count of the unbounded reads. Both are
gone. The list had already gone stale, naming four sections while the
file had ten, and the counts were an undercount by construction -- they
matched a redirect to `$VAR/`, which no gate, selector or command
substitution has. One test replaces both: no read below the library guard
runs without a ceiling, checked by looking for a counterexample.

A ceiling per read is not a ceiling on the report, and the header now
says so where it used to imply the opposite. The pod, VM, VMI and
services walks carry a count cap and a time budget; the walks over nodes,
namespaces, HelmReleases, packages, PVCs and certificates carry neither,
so their cost is the cluster's to set. What this removes is the read that
never returns.

The bounded reader now names the tool it ran and describes what that tool
returned. It served kubectl alone until this change gave it talosctl and
the four host commands, so a `df` that hit a stale mount left a marker
reading "kubectl exited 1 ... not because the object ended" -- two false
statements in one line, in an artifact whose premise is that its notes
can be believed.

The exec reader keeps the remote command's whole message in the listing
it belongs to. `kubectl exec` appends its own `command terminated with
exit code N` after whatever the command said, so quoting the last line
there drops the cause and keeps the trailer, and sending the message
away from the file leaves a zero-byte listing -- the "returned nothing,
or never ran" question this collector exists to answer. Its cut-off note
also stops asserting that the archive is beside it: a stream that got
members across is kept and truncated, one killed before its first byte
leaves zero bytes and the call site's emptiness check removes it, and
the reader runs before that check, so it names both.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 14:38:29 +03:00
Aleksei Sviridkin e80b8e370f fix(e2e): collect logs for unready pods and record truncated captures
The failure snapshot selected pods by their STATUS column, so a pod
whose container runs while its readiness probe never passes -- STATUS
Running, READY 0/1 -- was dropped from the report entirely. That is a
common way for a suite to time out: the replica never joins, its owning
HelmRelease sits in InProgress, and the artifact carries no trace of the
pod the run died on. Select on the READY column instead, which also
covers the partially ready multi-container case, and keep every phase
that was collected before. On one degraded cluster that is four of the
seven broken pods the old filter reported nothing about. The VMI and VM
sections had the same blind spot and now share one selector that reads
READY as the last field, because a Pending object prints fewer columns
than a Running one and the two kinds differ only in how many columns
precede READY. READY counts ready containers rather than the pod's Ready
condition, so a pod held back by an unsatisfied readiness gate is out of
scope; a test pins that rather than leaving it to be found in an empty
directory.

Logs are requested for every container of a selected pod rather than
skipped for Pending ones, and the per-pod collection moved into a helper
so it can be driven against a stub kubectl. kubectl's refusal is the
finding, and an empty result is ambiguous on its own: a read against an
unscheduled pod exits 0 having printed nothing, which is what a
container that started and stayed quiet also leaves. Every site that
classifies a failed read now orders its branches the same way, naming
the timeout first where both a cutoff and an uncapturable stderr apply.
The CRD gate in the summary had no such branch at all: it returned in
silence, which is byte-identical to the CRD being absent, so every
section behind that gate vanished from a report whose only real problem
was that the machine writing it had nowhere to put a temp file.

Collection used to stop silently. A walk that hit its cap said so in the
job log and nowhere in the artifact. Now the pod walk leaves
COLLECTION-TRUNCATED.txt when it stops at COZY_REPORT_PODS_MAX or
COZY_REPORT_PODS_BUDGET, CONTAINER-LOGS-TRUNCATED.txt on a pod cut off
part way through, COLLECTION-FAILED.txt when the pod list never came
back rather than the empty tree a healthy cluster produces, and
COLLECTION-UNBOUNDED.txt when timeout is missing locally, which is a
fact about the machine writing the report and not about the cluster.
Tenant namespaces are collected first, so a cap that only bites on a
broadly degraded cluster is not spent on platform pods while the pod the
suite died on falls off the end.

Container logs are read with --tail=2000 where they were previously read
whole, and every bounded file states the bound it was written under,
because --tail drops the oldest lines and nothing else in the output
would say so. COZY_REPORT_POD_TAIL=-1 restores the complete log.

The bounded whole-object read no longer merges stderr into pod.yaml,
vm.yaml and vmi.yaml, which are read by yq rather than by eye: a
warning on an otherwise complete read left an unparseable file. stderr
goes to a scratch file and is copied in only when kubectl returned no
object, and the truncation markers those files carry are written as YAML
comments, since a bare marker line after a complete mapping is a second
top-level node.

Six latent faults surfaced while making these reads report. The kamaji
module was gated on the LINSTOR controller, so a cluster without LINSTOR
got no kamaji evidence at all, which is what a tenant-kubernetes
bootstrap timeout needs most. The Flux sources section nested one
jsonpath filter inside another, which kubectl rejects outright, so it
printed nothing on every report ever taken and read as "all sources
Ready". The HelmRelease listing took a single field as STATUS where
STATUS is the Ready condition's whole message, so every failed release
rendered as one word. The services section filtered for a pending
LoadBalancer address on the CLUSTER-IP column, which never carries that
value. The cozystack module read its image from a Deployment the
installer stopped shipping when it moved to the operator, putting
kubectl's NotFound in the first file of the report. And the cozystack
apps section gated all three of its kinds on a CRD in an aggregated API
group, so it never ran once; two of those names were wrong, and both
are collected again.

The tests avoid EXIT-trap cleanup, and e2e-testing.md says why: a bats
test that installs one and then fails prints no TAP line at all, so
output grepped for `not ok` reads as green, while hack/cozytest.sh
reports the same failure correctly. A guard enforces that over a named
list of files rather than a glob with exceptions, because a list of
exceptions is the same claim written so that forgetting to shorten it
makes it false.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-05 17:37:04 +03:00
Myasnikov Daniil 6a41d65f16 test(e2e): avoid ripgrep dependency in diagnostics tests
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-22 17:27:02 +05:00
Myasnikov Daniil ad1adbf785 test(e2e): fix diagnostic collector blind spots
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-22 17:27:02 +05:00