The Ready-conditions and events reads share one block and added a line
of their own when they failed, on top of what their stderr put into
the block. Once dp_run marked every cut and refusal there, a cut read
carried two lines for one event, and before the budget status existed
the second named a timeout the first said the deadline had chosen.
The two reads now send stderr to the scratch file and, after their
output, write kubectl's stderr into the block whole, without dp_run's
marker, plus one line naming the read when it failed. kubectl's
message is kept as it came, since an error can follow lines of
throttling notices on a degraded apiserver. Without a scratch file
their stderr goes into the block directly. The probe comments also say
what the "spent" outcome now covers.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Once dp_run marked a deadline cut inside an exec block, the reads that
report their own outcome still read the 124 their shortened bound
produced and called it the read's own timeout, quoting a bound the
deadline had chosen. A Ready-conditions block then gave two reasons
for one cut, and every list and lookup note did the same beside the
section line that blamed the budget.
dp_run now returns a status of its own for a deadline cut, which the
read notes, the node and interface lookups and the LB probe all report
as the budget. pod_on_node takes it from dp_run instead of deciding it
a second time.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
An exec block keeps whatever it printed before its bound fired, and
nothing in the block said the bound had fired. The service map every
affected node gets is the largest such block, and a
truncated one answers the question it is there for, whether a backend
is still in the map, wrongly while looking complete.
dp_run now marks a cutoff where the call's output goes, as it already
does a refusal: the deadline's own cut names the budget, any other
names the call's bound. The section note also stops asserting that a
call was cut when the budget only refused the next one.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
dp_run credited a cutoff to the budget whenever the deadline had passed
by the time the call returned. A call under its own, unshortened bound
can fire, or come back from its kill grace, just past a deadline it
never reached; the section note then said the budget cut a call, next
to the read's own note naming its own timeout, and the walk closed as
cut short by the budget. The bound alone is not enough either: a call
killed early, as an OOM-killed remote process is, returns 137 under a
bound cut to the deadline long before the deadline arrives.
dp_run and pod_on_node now share one rule: a cutoff is the budget's
only when the call's bound reached the deadline and the deadline has
passed, and never when no timeout binary bounded the call at all.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A walk the budget cut short closed on "see the section line above",
but that line goes only to capture-notes.txt, while the closing line
also reaches the job log, where nothing sits above it to see. It now
names the file.
The budget knob replacement and the run without a timeout binary had no
test of their own; both are now pinned.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The LoadBalancer walk can stop on the budget, as the pod walk can, and
still closed on "capture complete" right under the line naming what it
did not probe. Both walks also called themselves complete when they
ended on their own after the deadline had cut a call inside their last
item.
Each walk now closes as stopped early when it broke out of the loop,
and as cut short by the budget when this run's section note says a
call was cut or refused; the note's own text is the evidence, since the
clock read after the loop can pass the deadline without any call having
been cut. The LB cap break, which shares that closing line, closes the
same way.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The pod walk can now also stop when the pod section's share of the
budget runs out, as it already could at the pod cap. Both exits name
what they left behind, and the line after them still announced the
capture as complete. It now says the walk stopped early.
The helper comment about the run separator also named every capture
file, while the OVN dump and the speaker logs are still replaced on
each run; it now lists the files it covers.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Once bounds are clipped to the budget, a cutoff has two meanings: the
call's own bound fired, or the deadline shortened it. Node lookups cut
the second way were still written up as lookups that did not answer, a
cause about the cluster that nothing observed. And the section note was
written only when a call was refused, so when the deadline cut a
section's last call the truncated capture carried no word about the
budget at all.
A lookup cut under a bound the deadline shortened now names the budget
in the node capture, the endpoint interface lookup and the LB probe,
decided from the bound it ran under rather than from the clock when its
status is read. A call the deadline cuts also writes the section
note, as a refusal does. The heavy-capture line also stops claiming
every probe failed when some of them were stopped by the budget.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A few reads that go through dp_run still failed or fell back without a
trace. A speaker log read that failed looked like a
speaker that never announced the address; it now writes a note. The
endpoint tcpdump's fallback to the geneve interface is stated in the LB
capture. The node capture now carries the run separator the pod and LB
files use, through one helper.
Two budget interactions named a cause that never happened. A per-read
timeout knob of zero made every read look refused by the budget, and a
value timeout cannot parse failed every read with 125, reported as
kubectl failing; both knobs are now validated like the budget ones. An
LB probe whose cni-server lookup the deadline cut was reported as a
second, unanswered lookup; it now counts as stopped by the budget.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The budget can run out inside one capture rather than between them. A
node lookup refused by dp_run was then written up as a lookup that did
not answer, an LB probe whose cni-server lookup was refused as the same,
and a probe exec refused, or cut by a bound the deadline had shortened,
as a node with no cni-server or no probe client. Each names a cause that
never happened.
The lookup's own status now reaches its callers, the node capture says
the lookup was not asked, and the probe reports a "spent" outcome that
the decision weighs like an unrun probe and the reason line attributes
to the section's budget. A reserve at least as large as the budget is
also named, since the pod section then starts no call at all.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The collector's default budget is sized under the 300s Chainsaw
backstop. The cozytest.sh caller allows 600s, and on the default it
would give up more than half of that window, so it passes a budget
matching its own backstop. The budget test reads each caller's
override and fails when a budget leaves more of its backstop unused
than the backstop's kill grace.
The Chainsaw call-site comment still described the fan-out caps as the
bound, and the e2e testing guide described an EndpointSlice exception
to the notes contract that the code does not have; both now describe
what the collector does.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The pod section stops a reserve short of the end of the budget and the
LoadBalancer section keeps starting calls after it, but a refusal was
reported as the whole run's budget running out, which the LB capture
written below it contradicts. The note was also deduplicated once per
run, so the LB section running out later left no line of its own.
Refusals now name their section, and the note is written once per
section. Both budget knobs are also read as decimal: shell arithmetic
took a leading zero for octal, so 08 aborted the run before it wrote
a note and 010 meant 8.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The per-LB capture and its two tcpdump files joined namespace and
Service name with a hyphen, which both may contain, so a-b/c and a/b-c
wrote the same files, and each run truncated them while the notes
beside them accumulate. They now use the separator and the run marker
the per-pod files use.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The per-node lookup memo keeps a cutoff so a hung lookup is not paid
for twice. Once bounds are clipped to the budget, a lookup can be cut
after a second or two because the pod section's deadline arrived, and
caching that made the LoadBalancer section report the component as
unknown for the node while its reserve still had time to ask.
A cutoff is now cached only when the lookup ran under its full bound.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A call dp_run refuses for a spent budget returned silently, and every
exec block swallows its status, so a capture the budget ran out in the
middle of kept only section headers. A header with nothing under it
reads as a command that found nothing, and the notes said nothing
either, because the loops only decline whole pods and LBs.
The refusal now goes to stderr, which lands in the block it emptied,
and capture-notes.txt gets one line per run. The OVS interface lookup
also stops calling a failed, cut or refused lookup an absent interface.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The fan-out caps counted pods and LoadBalancers while both callers'
backstops count seconds, and a single affected pod's worst case already
ran past the 300s Chainsaw one. No count could be made to fit, and a
cut-off run lost whatever came last, which was always the LoadBalancer
capture.
Every kubectl call now goes through dp_run, which clips the call's
bound to what is left of a run-wide budget sized under the smaller
backstop and starts nothing once it is spent. The pod section stops a
reserve short of the end so the LoadBalancer section always gets time
to start. A read cut by a clipped bound quotes the bound it ran under,
a read never started says so instead of naming a timeout, and a section
that runs out of budget names what it did not capture.
Tests pin the budget against both callers' backstops, the behaviour on
a cluster where every read hangs, and that no kubectl call bypasses
dp_run, listing any that does by line.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
`cilium-dbg bpf lb list` ran only inside the LoadBalancer path, which is
entered only when a LoadBalancer IP fails its probe. The map describes
every service translation on the node, so a pod that cannot reach a
ClusterIP left no record of which backend the VIP resolved to or
whether a backend that stopped answering was still in the map.
It now runs unfiltered in the per-node capture, which runs for the node
of each affected pod the collector reaches.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
POSIX sh has no local scope, so every assignment in a function lands in
the caller. capture_node wrote `node`, the variable of the loop that
calls it, and capture_lb_node wrote the LB loop's _lbip, _np, _epip and
_of. Both stayed harmless only because each caller passed what it
already held; any other argument would have made the rest of the
iteration read another node's pods and file them under the right name.
Both functions now use their own prefix, like the newer helpers in the
file, and a static test fails when a function assigns a name that a
`read -r` loop or another scope also owns.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The per-pod capture file joined namespace and pod name with a hyphen,
which both may contain, so a-b/c and a/b-c wrote the same file and the
second capture silently replaced the first. An underscore can appear in
neither, so it keeps every pod apart.
The file was also truncated on each run while capture-notes.txt beside
it accumulates, leaving notes from two runs next to evidence from one.
It now appends with the same run separator the notes use.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The collector characterises the host->local-pod route for an affected pod
through ovn0 alone, so when a pod IP turns out to be a local address of the
host netns instead, nothing records which interface actually holds it. The
host netns capture now also takes `ip -o addr show` across every interface
and `ip route show table local`, so the owner can be named on the next
occurrence rather than only ruled out on ovn0.
The collector also inspects NotReady pods only, so a run has no same-window
baseline for what a normal route and conntrack state look like on the same
node. The per-pod capture becomes a reusable function and runs a second time
against one Ready, Running, non-hostNetwork pod on that node, into the same
file, so the comparison is in front of whoever opens it. hostNetwork is
excluded because such a pod's podIP IS the node's own address and would show
the same local-route fingerprint as the fault for an entirely ordinary
reason.
That baseline depends on the node and nothing else -- the candidates are the
rows of the one snapshot the affected set was computed from, and a pod
reporting Ready there is by definition not one this run is investigating --
so it is captured once per node and folded into each affected pod's file
from a cache. Several affected pods on one node is what a wedged
host->local-pod datapath looks like rather than an edge case, and capturing
the same healthy pod once per affected pod would spend the caller's envelope
re-asking a question already answered -- the cost the node-global dedup and
the lookup memo beside it exist to avoid.
This adds no read of its own. The cluster-wide pod list gains one column
instead of being asked a second time, and that single read's status is what
lets the artifact separate a node with no eligible pod from a list that
never answered: one line for both would make an unchecked baseline read as
an established absence. The two host-netns additions are best-effort execs,
like the captures they sit beside. The new row filter carries the field
count its jsonpath emits, so a list cut mid-record cannot hand over a
fragment as the pod every reading of the affected one is compared against.
The unit suite pins these on what they are rather than on what they leave
behind. Both host-netns captures are asserted through the kubectl argv the
collector issues, container included, because the section headers are
separate echoes that outlive the exec being removed. The baseline is pinned
from both sides: several affected pods on one node share a single capture,
and an affected pod is served the healthy pod from its own node rather than
from whichever node the walk reached first. The pod list's column order is
asserted against the positions its readers index, because every runtime
test answers that list from a stub written by hand and none of them can
observe the jsonpath at all.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Five inputs walked into the LB probe gate's skip branch having run zero
probes, and all five came out as "probe: LB reachable or not probeable
... -- reachable, skipped": no node to probe from, a Service with no
port, a probe node the lookup says runs no kube-ovn-cni pod, a host netns
with no nc/curl/wget, and an exec that could not run (no timeout on PATH,
exec denied, the pod gone mid-probe). None of them observed the address.
An e2e image without a probe client would stamp every LoadBalancer in the
cluster reachable and skip the whole heavy capture, which reads as a
healthy datapath. A sixth way in, the cni-server lookup not answering at
all, already reported unknown and still does.
Zero probes now routes to that same explicit unknown outcome, and each
way of getting there carries a reason the run can actually support. The
two structural inputs are named where they are detected. The rest are
told apart by the shape of the outcome set, which is the only place the
difference survives: the probe helper emits an explicit unknown token
only when the lookup itself did not answer, so a set that is present and
wholly unknown names the lookup, while an empty set names the three ways
a probe dies after the lookup answered. Those three stay together --
the exec's status is swallowed by design, since this collector never
fails a job -- but none of them is reported as a lookup that went
unanswered, because it did not. The reachable line now says which
evidence produced it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two EndpointSlice reads in the LB section discarded stderr and were
interpreted by emptiness alone, so a refused or cut-off read wrote the
same `<none>` into the artifact that a Service with no endpoints does.
That is the assertion without the refutation: a reader holding the
uploaded report, and not the run, has no way to tell which happened, and
the section is at its weakest exactly when the apiserver is struggling.
Both reads now carry their status beside the value, in the vocabulary the
per-node captures already use.
Reporting the status arms a landmine the silence used to hide: the target
port was read with a jsonpath indexing `items[0]`, and client-go's
evalArray has no allowMissingKeys escape, so an empty list is a hard
error and kubectl exits 1. A Service with no endpointslices is an
ordinary answer here and would have started reporting as a failed read,
so the read now asks with `items[*]` -- empty output, exit 0 -- and takes
the first value, the way the pod lookups already do.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
pod_on_node turns its node argument into `--field-selector
spec.nodeName=<node>`, and an empty node asks for pods whose nodeName is
empty -- the unscheduled ones. So an unidentified node did not yield "no
pod found": it yielded the first pending kube-ovn-cni pod in the
namespace, and the tcpdump then ran against a pod that is not on that
node and may not be running at all.
Both halves of the cross-node capture had this, and each gap opens in
the very state its own leg exists to characterise. The announcer is
unknown exactly when MetalLB is misbehaving; a Service with no ready
endpoint is the ordinary shape of an LB outage, and the endpoint side
carried two such lookups, the cni-server pod and the backend's OVS
interface. In either state the artifact came out holding a tcpdump from
an unrelated pending pod under an ANNOUNCER or ENDPOINT label, or a
confusing exec failure -- both pointing away from the actual problem.
Each leg is now guarded the way the static capture above it already was,
and records in the artifact that it was skipped rather than leaving a
silent gap. The guards stay at the call sites rather than moving into
pod_on_node: that helper serves five callers, and an early return there
would have to invent a status for a lookup nobody made.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The record filters selected rows by field value and never checked the
field count, so a list cut mid-record parsed as a valid record: an IP cut
mid-octet still looks like an IP, and a cut inside the nodeName column
still looks like a scheduled pod. The capture then probed an address the
cluster never had and opened a per-node file for a node that does not
exist -- confidently labelled data about objects that are artifacts of
the cut.
The kubectl jsonpath behind each filter emits a fixed field count per
record, so a shorter row is a fragment by construction: seven for the
Service list, six for the pod list, five for the endpoint list. Dropping
it costs the reader nothing -- the read's own note already says the list
did not finish. The endpoint filter is the one that matters most, because
it stops at the first row it accepts: a fragment at its head is not one
bad row among many, it is the backend, and its half-parsed node is the
node every endpoint-side capture then aims at.
Where the class lives in this file, so the map does not have to be
rebuilt: four sites parse records, and the two loops walking pods and
LoadBalancers are not among them -- they consume what these filters
already returned, so a cut is caught upstream of them. Three of the four
are the filters guarded here. The fourth, the MetalLB speaker loop, is at
the limit of what a field count can do: its records carry two fields, a
cut in the first leaves no separator and its existing emptiness guard
already rejects that, and a cut in the second delivers every separator,
which nothing in-band tells apart from a genuinely short node name.
That limit is not confined to the speaker loop, and the comment on the
first filter now says so rather than implying a guard that holds
everywhere: a cut inside any record's LAST field passes a field count. It
lands on a display-only column for Services, and on a decision column for
the other two, where a truncated value matches neither state being
excluded and the row is kept -- over-capturing rather than dropping
evidence.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The header credited cozytest.sh alone, while the Chainsaw global catch
invokes the same script under a different wall-clock backstop -- 300s
against cozytest.sh's 600s. A reader sizing this collector's budget
against the caller the header names can still overrun the one it omits,
and the arithmetic further down the file already assumes both exist.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Five places turned an empty or missing result into a fact about the
cluster: no affected pods, no ovn-central, no LoadBalancer service, no
cilium-agent or ovs on a node, and a LoadBalancer recorded as reachable.
Each is also what the script produces when the read behind it never
answered, and with a note now saying the read failed, lines sat next to
each other and disagreed.
Each claim has a counterpart chosen by the status of the read that
produced it, and each says what is left unknown rather than what the
read did:
no scheduled NotReady pods / whether any pod is affected
no ovn-central pod in <ns> / whether <ns> runs ovn-central
no Service type=LoadBalancer / whether any LoadBalancer needs capturing
(no cilium-agent pod found) / could not determine whether one runs there
LB reachable, skipped / whether the LB is reachable is unknown
The last of those is the subtlest, because nothing about it looks like a
claim of absence. host_http_probe resolves a cni-server before probing,
and a lookup that never answered produced no probe outcome at all, which
the decision helper read as "nothing failed" and the artifact stamped as
reachable -- a verdict about an address the script never touched. The
probe now emits an explicit unknown, and a set of outcomes that is
entirely unknown stays unknown rather than collapsing into either
answer. Only a wholly unknown set: one unrun probe beside real failures
leaves the failures standing and still captures, because they are the
evidence the capture exists to characterise.
Wording that describes the read instead would be wrong in a case these
bounds create. A list can fail after emitting rows, since a bound firing
mid-stream leaves output on stdout and 124 in $?, so an empty result
with a non-zero status can mean the read said nothing or that it said
only uninteresting things. Before the bounds a read either answered or
hung forever, and a partial answer was not a state this script could
reach. For the same reason a failure note is unconditional on the status
while the consequence -- skipped, unresolved -- is stated only by the
branch that takes it, which is why the note inside pod_on_node names
none at all: five call sites share it and their consequences differ.
Two of these five are in the artifact rather than the log, and they are
there because of the bounds. Unbounded, the per-node lookup blocked
until the caller's backstop killed the script and no capture file was
written; bounded, the script survives and writes the file, so a read
that never answered would have been recorded as a pod that is not there
or an address that answered. pod_on_node returns its status with the
name and the memo stores it, since a hit that dropped it would launder a
cutoff into a confident absence. The status travels packed with the
value through stdout because every caller reads the helper through a
command substitution, where a variable dies with the subshell -- the
same property that makes the memo itself a file. Only answers and
cutoffs are memoised: an instant failure costs nothing to ask again, and
caching it would make one transient permanent for the whole run.
Every note also ships beside the capture it explains, not only in the
job log, because the reader who has the uploaded report and not the run
is exactly the reader who cannot otherwise tell a capture that found
nothing from one that never ran. Both sibling collectors hold that
contract and this script's notes are modelled on them.
The two phrasing helpers move above the sourcing guard so the unit suite
can assert their branches directly. The stderr sink stays below it,
because anything created above is created again by every test that
sources this file and by every host that exits early for want of
kubectl.
What is deliberately left is the same distinction for the two
EndpointSlice reads, which still discard stderr and report neither
outcome.
Each branch is pinned by a stub that answers one read and fails another,
each partial-answer case by a stub that emits a row and then fails, the
artifact cases in the artifact rather than the log, and the memo's
status round-trip by consuming one lookup from both a miss and a hit.
Every one of them carries a positive anchor as well, so a stub that
stops short of the branch fails rather than passing quietly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The same (namespace, label, node) is asked up to three times per
affected pod across the sections of this script. That was free while the
first call hung forever and the script died inside it; with a per-read
bound each repeat costs its own timeout, and against a wedged apiserver
the repetition becomes the largest consumer of the caller's envelope --
enough that a 300s budget buys two pods where it could buy five.
Answers are memoised, empty ones included: "there is no cilium-agent on
this node" is exactly the lookup worth not repeating.
The memo is a file rather than a variable, which is not a style choice.
The walk over affected pods runs on the right-hand side of a pipeline,
so it executes in a subshell and a variable memo is discarded at the end
of it, leaving every lookup to be paid for again in the sections that
follow. With the variable form the suite still recorded each of the
three lookups twice. That is also why this is not the shape of the node
memo above it, which is a space-delimited string matched with a case
glob and is correct for a dedup living entirely inside the walk.
Keys are compared as whole tab-separated fields, so the '=' inside a
label selector is ordinary payload; what the scheme rests on is that no
namespace, label selector or node name contains a tab, which Kubernetes
names and selectors cannot.
An empty pod list is also separated from one that never answered, so the
branch below it stops reporting a failed read as a cluster with nothing
wrong.
Pinned by a stub that records every per-node lookup it is given: the
test fails if any is issued twice.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Six `kubectl get` calls ran with no bound of their own, while all 27
`kubectl exec` calls in the same script went through one. Against an
apiserver that hangs rather than refuses, the first of them -- the
cluster-wide pod list -- held the script until the caller's backstop
killed it, so the capture wrote nothing at all. From the Chainsaw catch
that is worse than empty: the run is cut off by the outer `timeout -k`
and the script says nothing about having been cut short, so a truncated
capture reads exactly like a complete one that found no affected pods.
Measured against a kubectl stub that never returns: before, the script
was still blocked on its first call when a 60s backstop killed it;
after, it reaches its own end and says which read was cut off and what
that leaves missing. Roughly 4s at a 1s bound, 40s at the 20s default.
A per-read bound buys forward progress, not a smaller total. The sum of
the bounds here is far past any caller's envelope and deliberately so,
because the MAX_PODS and MAX_LBS caps bound work while the outer
`timeout -k` bounds wall clock. What changes is that no single hung read
can consume that envelope before anything is written.
The reporting around them is built to survive being read by someone who
was not there. Only 124 and 137 may name a timeout, since those are the
only statuses a bound produces and every other one is kubectl answering
-- refused, denied, a kind the cluster does not serve -- so the reads
keep their stderr instead of discarding it and quote it back. 137 is
described as what it is, a SIGKILL that cannot be told apart from the
bound's own kill grace. Notes print through `printf` rather than `echo`,
because under /bin/sh echo expands backslash escapes and kubectl's
stderr now flows through the logger: a message holding a literal \n
would otherwise split the note and forge a second "[capture-dataplane]"
line that reads as this script's own verdict, which needs no malice
given the jsonpath in these reads contains {"\n"} and kubectl quotes the
expression back when it fails to parse it.
The per-node lookups ask with `{.items[*].metadata.name}` rather than
`{.items[0]...}`. client-go's evalArray has no allowMissingKeys escape
the way evalField does, so indexing into an empty list is a hard error
and kubectl exits 1 -- which meant nothing while the status was
discarded, and would report "there is no cilium-agent on this node", the
ordinary answer, as a read that failed.
The tests drive the script as a subprocess against a stub kubectl that
hangs, fails instantly, or returns a forged line, and read what it
wrote. The stub that proves forward progress answers the pod list and
hangs afterwards, so the walk over affected pods is actually entered.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Root-causing a crash-loop failure was blocked twice over by the
diagnostics collected on failure.
`kubectl logs` shows only the current container instance, so for a
crash-looping pod the suites' `podLogs` collector captured the Nth
restart -- a fresh replay of the same startup path -- while the
evidence that explained the failure sat in the first, interrupted
instance. Chainsaw's podLogs collector has no `previous` option, so
add e2e-capture-previous-logs.sh and drive it from the global catch
and from the BATS failure hook. It discovers containers with
restartCount > 0 at runtime (init containers included, where an
interrupted bootstrap lives), orders the app suites' namespace first,
and dumps each to the job log and the snapshot tree. Containers that
never restarted are skipped, so a healthy run emits one line rather
than a page of "previous terminated container not found".
Every way that capture can come up short is named rather than
inferred, because kubectl exits 1 for a garbage-collected container,
an RBAC denial and an unreachable node alike: the container cap, a
pod list that never returned (which is not the same as nothing having
restarted), a read that timed out versus one kubectl refused and the
reason it gave, and a previous instance that produced no output. A
partial capture is kept and marked truncated rather than deleted --
it is the only copy of a log already gone from the cluster.
The crust-gather snapshot was killed by the Chainsaw op containing it.
A script op is bounded by timeouts.exec (5m), and the catch's own
wall-clock wrappers summed to 960s, so the script was SIGKILLed
partway through -- taking out the process group mid-write and skipping
the data-plane capture entirely. Give each catch op an explicit
timeout above the budgets it contains, so an inner timeout fires
instead and leaves a partial, still-servable snapshot behind. The
data-plane leg drops from 600s to 300s at that call site so both legs
fit one envelope; the BATS hook keeps the wider budget, having no
containing op.
Set crust-gather --duration explicitly as well. It is the collection
phase's own budget, it defaults to 60s, and on elapse it abandons the
collection and skips its finish step, leaving a truncated tree with no
indication anything is missing. The outer wall-clock has to exceed it
because the API discovery that runs first is not covered by that
budget at all -- with an unhealthy aggregated APIService, the state a
failed run is usually in, discovery alone can consume the whole
timeout before a single object is written. Send each collector's
output to a log beside its snapshot instead of /dev/null, with a
complete/truncated/failed verdict, so that question is answerable
from the artifact.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The dataplane diagnostic's announcer-node detection reported the wrong
node in two failover situations and could match the wrong IP:
- A node that announced an IP and then withdrew it (with no
re-announce) was still reported as the announcer, because a withdraw
did not retract a prior announce.
- Speaker logs are concatenated per-pod, not globally time-sorted, so
a stale former owner's announce could outrank the true current
owner purely by its position in the input.
- The IP was matched as a substring, so a query for 192.0.2.5 also
matched a 192.0.2.50 line.
Reimplement the detection to report a node only when that node's LAST
own announce/withdraw event for the exact IP is an announce, which is
robust to both concat order and L2 failover, and anchor the IP match to
a whole IP-literal token instead of a substring.
Also fix the heavy-capture budget: the MAX_LBS cap counted every
enumerated LB, so a broken LB enumerated after a cap's worth of
reachable LBs was never captured. Count only LBs actually captured, via
a small pure budget helper, so reachable/skipped LBs do not consume the
budget. Document that this cap bounds work, not wall-clock -- at two or
more unreachable LBs the real wall-clock bound is the outer backstop at
the call site.
Add bats coverage pinning the fixed announcer behavior (withdraw,
concat-order, exact-IP) and the captured-budget accounting.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The on-failure dataplane diagnostic only captured host->pod CNI state for
NotReady pods. A recurrent flake presents differently: a tenant Service
type=LoadBalancer IP goes unreachable while its backend pod/VMI stays Ready
and Running, so there is no NotReady pod and the hook captured nothing --
leaving the host->cross-node->bridged-worker-VMI->nodePort path
uncharacterised, which is exactly where root-causing has stalled.
Add a gated LoadBalancer-datapath section. It enumerates LoadBalancer
services with an ingress IP, derives the EndpointSlice backend, the
nodePort/externalTrafficPolicy, and the announcer node from the hostNetwork
MetalLB speaker logs, then probes the LB IP from a host netns. Only an LB
whose every probe fails gets the heavy capture, and that probe doubles as
the traffic for the tcpdumps. For a failing LB it captures, on both the
announcer and endpoint nodes, the host cilium LB map, kernel
conntrack/neigh, br-int flows, ovn-controller timing, and a live tcpdump on
the geneve tunnel and the backend tap -- localising the failing hop to
host-cilium programming vs kube-ovn delivery. In-guest VMI capture is
documented as a deferred stretch that needs tenant-cluster access.
The existing NotReady-pod capture is unchanged. The new section is
best-effort and time-boxed like the rest of the script, and a unit test
pins the pure enumeration, announcer-node, and capture-or-skip logic against
mock input.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Extend the host->pod data-plane capture with the three datums that make an
OVN incremental-processing lag dispositive on the next recurrence, all
diagnostic-only and gated on the existing on-failure path:
- ovn-controller.log from the ovs-ovn pod (grep for physical_flow_output,
if_status_mgr, took <N>ms, recompute, long poll interval; plus a bounded
tail) -- shows whether ovn-installed was set before the LSP local-delivery
flow was programmed.
- per-interface ovn-installed flag + ovn-installed-ts on the pod's OVS
interface, looked up by iface-id <pod>.<ns>.
- ovs-ovn cgroup cpu.stat (nr_throttled / throttled_usec, cgroup v2 with v1
fallback) -- shows whether ovn-controller was CPU-starved.
Node-global captures land in node-<node>.txt, the per-interface flag in the
per-pod file. Every command stays timeboxed and || true, no behavior or
exit-code change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The E2E install intermittently fails on a CNI host->local-pod data-plane
transient: kubelet on a node reaches a local pod's readiness/startup probe
with "connection refused" for several minutes while overlay pod->pod
traffic works, then it self-heals. This is rooted in the cilium+kube-ovn
chaining config (forced enable-host-legacy-routing, CNI
InstallEndpointRoute:false), which delegates host->local-pod routing to
kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object
state but not the node's L3 forwarding state, so the mechanism cannot be
root-caused after the fact.
On test failure, for each NotReady pod that is scheduled, has a podIP, and is
not terminal (Succeeded/Failed excluded), collect the host->pod data-plane
state from the pod's node into the crust-gather snapshot dir (so it ships in
the uploaded report):
- host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>,
ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for
the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under
enable-host-legacy-routing the host->local-pod path traverses the kernel
netfilter stack, not cilium BPF, so this is the authoritative conntrack
table and route/rule state for the transient;
- cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a
bounded monitor --type drop, and hubble dropped verdicts when present
(complementary: confirms the path is not handled in BPF);
- OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding /
Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list.
Diagnostic-only: it runs after the existing crust-gather collect and only on
an already-failed test, never mutates the cluster, and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
|| true, the whole run has a wall-clock backstop, node-global captures are
deduped per node, and the pod count is capped. It no-ops cleanly when no
pods are affected or when a tool/pod is absent.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin