Commit Graph
9 Commits
Author SHA1 Message Date
Myasnikov Daniil e949f39800 fix(e2e): preserve previous-log cap slots
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-28 15:29:29 +05:00
Myasnikov Daniil a5c90d984e fix(tests): guard the finishedAt leaf and make three guards actually bite
Findings from two independent adversarial reviews of the preceding two
commits. Each was reproduced locally before being fixed.

The ordering inverted itself on a shape I had not probed. `finishedAt`
is a metav1.Time behind `json:",omitempty"`, which cannot omit a
struct, and its zero value marshals to `null`; the kubelet does
synthesize a terminated state with no time (ContainerStatusUnknown on a
pod deleted mid-run, which also bumps restartCount and so clears the
restart filter). Two guards deep left the leaf exposed, and kubectl
renders it as the literal `<no value>` -- which sorts ABOVE every digit,
so that row took the front of the queue from the most recent real
restart, and the header printed `previous instance ended <no value>`, a
claim about a time nothing observed. Verified with a text/template probe
over seven lastState shapes: `null` and a missing key both produced it.

Fixed at both layers, because they fail independently. The template now
guards the leaf, so the string cannot be produced. And the sort
normalises any field-6 value not shaped like a timestamp to empty, so
the ordering stays correct and the header stays quiet even if some other
producer emits one. Earlier verification against a live cluster showed
no `<no value>` and I read that as the guard being right; it only meant
no container on that cluster was in that state.

Three guards did not bite:

  - both stability fixtures were pre-sorted ascending, which is exactly
    what sort's last-resort whole-line compare reproduces with no -s at
    all, so the tests written for -s passed without it. Reversed; they
    now fail when -s is dropped.
  - no test fed a six-field row through the whole script, so the
    timestamp reached the dump header uncovered -- deleting the read
    variable left the suite green at 39/39. One integration test now
    asserts the header names the time.
  - the e2e guard matched any mention of COZYTEST_TRACE, so documenting
    the design over there would have turned it red. It now matches an
    assignment, and tolerating a comment is verified too.

Also: a failed sort no longer discards a whole namespace group. The
caller compares row counts to report the cap, so losing a group silently
became "0 of N captured, N more not captured" and sent the reader after
a cap that never fired. It falls back to discovery order and says so.
LC_ALL=C on the sort, since the key is fixed-format ASCII.

And the claim itself was too strong. "Nothing diagnostic is lost" holds
only for a test that RETURNS: the fail handler runs after the pipeline,
and there is no INT/TERM trap, so a test killed rather than failed -- a
hang reaped by the job timeout -- never dumps its log, and under quiet
mode its trace is gone rather than already printed. The comments in
hack/cozytest.sh and the Makefile now say so and name the escape hatch.
A trap that flushed the tail of the log would close it; left out
deliberately rather than bundled in here.

Every fix is mutation-proven: reverting the leaf guard, removing the
key normalisation, dropping the read variable, dropping -s, and
silencing an e2e recipe are each caught. Unit lane 1513 passed, 1 failed
(hack/ghcr-mirror_test.bats, which reads a live cluster and fails the
same way under main's runner).

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-27 22:06:53 +05:00
Myasnikov Daniil 7741843178 test(e2e): order previous-instance log capture by restart recency
The container cap (COZY_PREVLOG_MAX=12) is spent in namespace order, and
"unrelated" has meant only "in another namespace" - time was never a
key. So a container that restarted once during cluster bring-up
competed on equal footing with one that crash-looped during the test,
and won whenever the pod list reached it first. Two measured e2e jobs
spent 1034 and 1545 lines on virt-api and cdi-apiserver, restarted once
at install time by `Unable to create certwatcher` hours before any app
suite ran; on a development cluster 178 containers carried a restart, their
timestamps spanning 24 days, so the cap binds hard enough for the order to
decide everything.

Ordering, not filtering, and deliberately: an install-time restart IS
the explanation when the install is what broke, and this capture has no
reference point for "when the test began" that would make a cutoff
honest. Namespace stays the primary key - a stale restart in the
namespace under test beats a fresh one three namespaces away. So this
fixes cap displacement, not the raw volume of any single run.

lastState.terminated.finishedAt joins the row, guarded so a missing
lastState cannot error the list call and fail the capture open (verified
against a live cluster: no <no value>, empty where the kubelet has
already dropped it). Reading it also fixes a latent bug the sixth field
would otherwise have caused - `read -r ... restarts` folds a trailing
field into the last variable, so the dump header would have printed
`restarts=1|2026-08-22T09:06:00Z`. It now names the time instead, which
is the first thing that decides whether the dump is worth reading.

Sorting is lexicographic over fixed-width RFC3339 UTC, so no date
parsing: reverse puts newest first and leaves undated rows last, and -s
keeps a node reboot's simultaneous restarts in input order so the
selection is comparable between runs. Rows with no timestamp pass
through untouched, which is what makes this a no-op for the existing
five-field fixtures.

Five tests added; both mutations are caught (a passthrough
sort_recent, and dropping -s - the latter by the pre-existing namespace
test). capture-previous-logs 39/39, cozyreport 184/184,
bats-no-exit-trap 22/22.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-27 22:06:33 +05:00
Aleksei Sviridkin e80b8e370f fix(e2e): collect logs for unready pods and record truncated captures
The failure snapshot selected pods by their STATUS column, so a pod
whose container runs while its readiness probe never passes -- STATUS
Running, READY 0/1 -- was dropped from the report entirely. That is a
common way for a suite to time out: the replica never joins, its owning
HelmRelease sits in InProgress, and the artifact carries no trace of the
pod the run died on. Select on the READY column instead, which also
covers the partially ready multi-container case, and keep every phase
that was collected before. On one degraded cluster that is four of the
seven broken pods the old filter reported nothing about. The VMI and VM
sections had the same blind spot and now share one selector that reads
READY as the last field, because a Pending object prints fewer columns
than a Running one and the two kinds differ only in how many columns
precede READY. READY counts ready containers rather than the pod's Ready
condition, so a pod held back by an unsatisfied readiness gate is out of
scope; a test pins that rather than leaving it to be found in an empty
directory.

Logs are requested for every container of a selected pod rather than
skipped for Pending ones, and the per-pod collection moved into a helper
so it can be driven against a stub kubectl. kubectl's refusal is the
finding, and an empty result is ambiguous on its own: a read against an
unscheduled pod exits 0 having printed nothing, which is what a
container that started and stayed quiet also leaves. Every site that
classifies a failed read now orders its branches the same way, naming
the timeout first where both a cutoff and an uncapturable stderr apply.
The CRD gate in the summary had no such branch at all: it returned in
silence, which is byte-identical to the CRD being absent, so every
section behind that gate vanished from a report whose only real problem
was that the machine writing it had nowhere to put a temp file.

Collection used to stop silently. A walk that hit its cap said so in the
job log and nowhere in the artifact. Now the pod walk leaves
COLLECTION-TRUNCATED.txt when it stops at COZY_REPORT_PODS_MAX or
COZY_REPORT_PODS_BUDGET, CONTAINER-LOGS-TRUNCATED.txt on a pod cut off
part way through, COLLECTION-FAILED.txt when the pod list never came
back rather than the empty tree a healthy cluster produces, and
COLLECTION-UNBOUNDED.txt when timeout is missing locally, which is a
fact about the machine writing the report and not about the cluster.
Tenant namespaces are collected first, so a cap that only bites on a
broadly degraded cluster is not spent on platform pods while the pod the
suite died on falls off the end.

Container logs are read with --tail=2000 where they were previously read
whole, and every bounded file states the bound it was written under,
because --tail drops the oldest lines and nothing else in the output
would say so. COZY_REPORT_POD_TAIL=-1 restores the complete log.

The bounded whole-object read no longer merges stderr into pod.yaml,
vm.yaml and vmi.yaml, which are read by yq rather than by eye: a
warning on an otherwise complete read left an unparseable file. stderr
goes to a scratch file and is copied in only when kubectl returned no
object, and the truncation markers those files carry are written as YAML
comments, since a bare marker line after a complete mapping is a second
top-level node.

Six latent faults surfaced while making these reads report. The kamaji
module was gated on the LINSTOR controller, so a cluster without LINSTOR
got no kamaji evidence at all, which is what a tenant-kubernetes
bootstrap timeout needs most. The Flux sources section nested one
jsonpath filter inside another, which kubectl rejects outright, so it
printed nothing on every report ever taken and read as "all sources
Ready". The HelmRelease listing took a single field as STATUS where
STATUS is the Ready condition's whole message, so every failed release
rendered as one word. The services section filtered for a pending
LoadBalancer address on the CLUSTER-IP column, which never carries that
value. The cozystack module read its image from a Deployment the
installer stopped shipping when it moved to the operator, putting
kubectl's NotFound in the first file of the report. And the cozystack
apps section gated all three of its kinds on a CRD in an aggregated API
group, so it never ran once; two of those names were wrong, and both
are collected again.

The tests avoid EXIT-trap cleanup, and e2e-testing.md says why: a bats
test that installs one and then fails prints no TAP line at all, so
output grepped for `not ok` reads as green, while hack/cozytest.sh
reports the same failure correctly. A guard enforces that over a named
list of files rather than a glob with exceptions, because a list of
exceptions is the same claim written so that forgetting to shorten it
makes it false.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-05 17:37:04 +03:00
Aleksei Sviridkin d3ae21b7f0 fix(e2e): name every way the previous-instance capture falls short
The capture wrote its reasons to the job log only. A reader holding the
uploaded artifact saw N logs in a directory with no way to tell N from
all of them: the container cap that dropped the rest, a pod list that
never returned, a previous instance that produced nothing, and a read
the outer backstop cut short all looked alike from outside. Every one
of those lines now goes to capture-notes.txt as well, and the directory
is created before the first of them, so a capture that found nothing is
distinguishable from one that never ran.

A pod list that failed is no longer reported as nothing having
restarted. That is a claim about the cluster which the script never
observed, and it is the claim most likely to talk a triager out of the
crash-loop hypothesis on exactly the run where it is right.

Exit 137 no longer asserts whose signal ended the read. 124 comes from
timeout and from nothing else, but 137 is 128+SIGKILL, which the kill
grace produces and so does the OOM killer or a teardown signalling the
process group. Both belong in the cut-off branch, since the capture
ends early either way, but reported flatly as the script's own timeout
a read killed at second two reads as one that waited the full eighteen.

COZY_PREVLOG_TAIL rejects a value kubectl cannot use and says so. A
zero tail asks for no lines while kubectl still exits 0, which the
empty-output branch would otherwise report as the container having
logged nothing, discarding the only remaining copy of a log the kubelet
has already collected. A leading zero is rejected because `$(( ))`
reads it as octal while `[` reads it as decimal, so one string means
two numbers. Each kept log names the bound it was written under, since
--tail drops the oldest lines and nothing else in the output says so,
and -1 suppresses that note because no bound was applied.

An output directory that cannot be written is reported as the local
condition it is. It arrives as an argument, so it can be a file, or
missing, or unwritable; the redirection then fails before kubectl runs
and hands the shell's own status to the branch that names kubectl,
which reported "no previous instance retrieved (kubectl exit 2)" once
per container.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-05 16:03:57 +03:00
Aleksei Sviridkin a7ce4371c2 docs(agents): correct the chainsaw podLogs capability note
The e2e conventions, the previous-logs capture script, and the global
chainsaw catch all claimed the v0.2.15 podLogs collector exposes only
container/tail/selector. It also accepts name, namespace, timeout and
the cluster fields; only the previous option is genuinely missing. The
wrong list steers catch authors toward imperative scripts for
cross-namespace log capture that podLogs handles declaratively.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-27 22:48:25 +03:00
Aleksei Sviridkin f3a0f2d323 test(e2e): avoid head -n 0 in prevlog_cap for BSD portability
A zero container cap reached head -n 0, which GNU head treats as an empty
result but BSD/macOS head rejects ("illegal line count -- 0"), aborting a
local run. Special-case the zero cap to consume stdin instead, so the
helper behaves the same on both coreutils flavours.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-21 19:32:42 +03:00
Aleksei Sviridkin 76f7267ff2 test(e2e): describe --previous as the preceding instance, not the first
kubectl logs --previous returns the immediately preceding terminated
container instance, not the original first one — after multiple restarts
the first instance is already gone. Reword the helper, the BATS and
Chainsaw failure handlers, and the diagnostics doc to match the actual
--previous semantics.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-21 18:57:01 +03:00
Aleksei Sviridkin 07c67a8913 test(e2e): capture previous container logs and stop killing the snapshot
Root-causing a crash-loop failure was blocked twice over by the
diagnostics collected on failure.

`kubectl logs` shows only the current container instance, so for a
crash-looping pod the suites' `podLogs` collector captured the Nth
restart -- a fresh replay of the same startup path -- while the
evidence that explained the failure sat in the first, interrupted
instance. Chainsaw's podLogs collector has no `previous` option, so
add e2e-capture-previous-logs.sh and drive it from the global catch
and from the BATS failure hook. It discovers containers with
restartCount > 0 at runtime (init containers included, where an
interrupted bootstrap lives), orders the app suites' namespace first,
and dumps each to the job log and the snapshot tree. Containers that
never restarted are skipped, so a healthy run emits one line rather
than a page of "previous terminated container not found".

Every way that capture can come up short is named rather than
inferred, because kubectl exits 1 for a garbage-collected container,
an RBAC denial and an unreachable node alike: the container cap, a
pod list that never returned (which is not the same as nothing having
restarted), a read that timed out versus one kubectl refused and the
reason it gave, and a previous instance that produced no output. A
partial capture is kept and marked truncated rather than deleted --
it is the only copy of a log already gone from the cluster.

The crust-gather snapshot was killed by the Chainsaw op containing it.
A script op is bounded by timeouts.exec (5m), and the catch's own
wall-clock wrappers summed to 960s, so the script was SIGKILLed
partway through -- taking out the process group mid-write and skipping
the data-plane capture entirely. Give each catch op an explicit
timeout above the budgets it contains, so an inner timeout fires
instead and leaves a partial, still-servable snapshot behind. The
data-plane leg drops from 600s to 300s at that call site so both legs
fit one envelope; the BATS hook keeps the wider budget, having no
containing op.

Set crust-gather --duration explicitly as well. It is the collection
phase's own budget, it defaults to 60s, and on elapse it abandons the
collection and skips its finish step, leaving a truncated tree with no
indication anything is missing. The outer wall-clock has to exceed it
because the API discovery that runs first is not covered by that
budget at all -- with an unhealthy aggregated APIService, the state a
failed run is usually in, discovery alone can consume the whole
timeout before a single object is written. Send each collector's
output to a log beside its snapshot instead of /dev/null, with a
complete/truncated/failed verdict, so that question is answerable
from the artifact.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-21 18:37:07 +03:00