Commit Graph
3 Commits
Author SHA1 Message Date
Myasnikov Daniil cc942fed68 test(tests): run unit suites with Bats and retain POSIX coverage
The shell translator rewrites test bodies and loses Bats reporting.
The unit lane needs complete TAP results and JUnit reports without
changing how live-cluster suites run.

Sourced production helpers still need a dash pass, and unit test bodies
need nounset without exporting shell options into subprocesses.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 11:04:00 +03:00
Aleksei Sviridkin 527d4ea00f test(e2e): measure tenant worker CPU as a rate, and the sandbox nodes under it
Two instruments for the node-join failure the kubernetes-* suites keep
hitting. Neither changes what the suites assert.

The worker CPU capture now reads container_cpu_usage_seconds_total
beside the CFS counters, and reads them twice a fixed interval apart
instead of once. The throttled counters say a container met its ceiling
and cannot say how much CPU it got, which are the two candidate
explanations for a stalled worker: a guest held at a one-core ceiling,
and a guest whose vCPU thread is rarely scheduled at all. Both leave
throttled periods behind. Every counter involved accumulates from
container start, so a single reading is an average over the whole uptime
and no rate at all, and the profile under investigation is a guest that
freezes for tens of seconds and then runs flat out: the average comes
out low, the instantaneous figure comes out at the cap, and only the
second separates it from a guest that is simply capped.

The sandbox nodes are a new subject. The worker captures describe a
cgroup, so a guest that is runnable and never scheduled looks from there
like a guest that asked for nothing. Whether the sandbox node itself was
given its turn is recorded only in the steal column of its own
/proc/stat: no kubelet endpoint carries it, and crust-gather cannot
reach it because PodSecurity refuses its node-shell Pod here, so the
read goes over the Talos API, which is already how the report reaches
those nodes for dmesg. The rows are captured verbatim with their column
legend beside them rather than reduced to a percentage, because
/proc/stat has no header, steal and guest sit next to each other, and
guest is large on a node running VMs and already counted inside user.

Both subjects are read on either side of one shared wait, so the pair
pays it once. The wait is not the interval either subject's counters
span, though: each pass walks its own nodes and the other subject's pass
falls between a subject's two readings, and on the run this exists for
those passes are the slow part. Each capture therefore stamps both ends
of its read, so a rate comes from subtracting stamps rather than from
assuming the knob, and the width of those brackets is the uncertainty.
That wait is a measurement interval rather than a stand-in for a
condition nobody wrote a wait for, so the e2e conventions carve it out
by name instead of marking it as a fixed timeout to be replaced.

The pair is gated below the serial console rather than above it. Its
ceiling is sixteen bounded reads plus the interval, which is most of the
phase budget, and admission gates only when a collector may start, so
above the console a run that hit those bounds would leave the console
never started rather than merely cut short. The console is the only
capture that survives a worker which never reached apid, which is the
shape this failure usually takes. That overrun is inside the term the
budget derivation already allows for one late admission, and a guard
holds the pair's ceiling against it.

No collector here may leave an empty directory, since an empty directory
reads as a healthy node: a missing binary, a missing talosconfig, a
listing that never answered, a node with no address and a read that was
cut off each land as a named note or a COLLECTION-FAILED.txt.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-15 10:19:19 +03:00
Aleksei Sviridkin 61edf33ef4 fix(e2e): capture both ends of the worker byte path on node-join failure
A tenant worker that misses the node-Ready deadline while its kubelet
image pull crawls leaves two mechanisms behind one symptom, and the
artifact could not tell them apart. Either the link is slow, or the pull
keeps restarting and discarding what it fetched; from inside the guest
both are a progress line that barely moves.

Per-worker network counters, read from the kubelet's cAdvisor endpoint
the CPU throttling capture already uses, count every byte that crossed
into the Pod including the ones a restart threw away, so the gap
between that and the progress the guest reported separates the two.

Which row carries it is not obvious and getting it wrong inverts the
reading, so the collector states it. cAdvisor publishes one row per
interface in the Pod netns, and a bridge-bound worker has four: the one
still named eth0 is a dummy holding the Pod IP, carries no traffic, and
reports zero -- the row a reader reaches for first, where zero reads as
a worker that received nothing. The renamed NIC's receive is the number
that answers; the bridge counts the same bytes again and the tap is
named from the host's end. No interface predicate is applied, so all
four reach the artifact and can be told apart.

Two further properties of the family fail by capturing nothing rather
than loudly: the dropped-packet series are spelled packets_dropped, and
every row carries an empty container label because cAdvisor publishes
them on the Pod sandbox, so a filter borrowed from the CPU counters
beside it matches nothing at all. Both are stated where the filter is.
The fixture's rows are real kubelet output relabelled; its interface
names are read from the KubeVirt version this repo ships rather than
sampled, since no virt-launcher Pod was available.

The guest side gains the service state machine and the link table. Talos
creates a service's log buffer when its runner opens the log writer, as
it starts the process, so a kubelet still waiting on its volumes, on
time sync, or on the container runtime reporting healthy never reached
its runner, has no log at all, and the capture recorded an rpc error
naming a log that was never registered: true about the log, silent about
the service, while the state and the awaited condition were the
finding. The link table is read as yaml because MTU lives in the spec
rather than among the printed columns, and it is the value that decides
whether a path-MTU reading of a stalled transfer survives. Both carry a
tighter bound than the two reads beside them, being single tabular RPCs
rather than streams, and the walk they sit in is the term the phase
budget sizes itself against.

A worker whose reads all failed now says so once for the worker. Each
log already carried its own error and exit code, which is accurate per
file and adds up to a directory indistinguishable from a worker that had
little to report.

The phase budget drops from 8m to 7m to pay for the two added guest
reads, because the inequality keeping the tenant snapshot reachable had
ten seconds of slack. A shorter budget declines more, and what it
declines first is what is gated last: the mirror capture, whose subject
is partly recoverable from its own request counts and the host snapshot,
and the image-cache re-probe, which has no substitute and was already
the first to go.

The second cost lands on the guest captures, which are gated after the
new counters and reach their gate with both the shortened budget and a
listing plus a read per node already spent -- up to 100s at the read
bound and the three-node cap, a ceiling those numbers impose rather than
a duration measured on a live cluster. Sharing that read would mean the
two captures becoming one walk, which is left out for scope rather than
merit: the phase gate is a deadline rather than a per-collector
allowance, so one walk admitted at the first of the two gates would cost
less and collect more. The cost is stated rather than netted away, and
the source says which way the argument runs.

Both captures run one shared body, which also settles what the two
copies had already broken once: the audit that catches an unbounded read
pins each collector by something only it produces, and two collectors
issuing byte-identical reads left neither with anything of its own, so
stubbing one out of the audit went unnoticed. It is pinned by the report
directory each writes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-14 14:34:37 +03:00