Root-causing a crash-loop failure was blocked twice over by the
diagnostics collected on failure.
`kubectl logs` shows only the current container instance, so for a
crash-looping pod the suites' `podLogs` collector captured the Nth
restart -- a fresh replay of the same startup path -- while the
evidence that explained the failure sat in the first, interrupted
instance. Chainsaw's podLogs collector has no `previous` option, so
add e2e-capture-previous-logs.sh and drive it from the global catch
and from the BATS failure hook. It discovers containers with
restartCount > 0 at runtime (init containers included, where an
interrupted bootstrap lives), orders the app suites' namespace first,
and dumps each to the job log and the snapshot tree. Containers that
never restarted are skipped, so a healthy run emits one line rather
than a page of "previous terminated container not found".
Every way that capture can come up short is named rather than
inferred, because kubectl exits 1 for a garbage-collected container,
an RBAC denial and an unreachable node alike: the container cap, a
pod list that never returned (which is not the same as nothing having
restarted), a read that timed out versus one kubectl refused and the
reason it gave, and a previous instance that produced no output. A
partial capture is kept and marked truncated rather than deleted --
it is the only copy of a log already gone from the cluster.
The crust-gather snapshot was killed by the Chainsaw op containing it.
A script op is bounded by timeouts.exec (5m), and the catch's own
wall-clock wrappers summed to 960s, so the script was SIGKILLed
partway through -- taking out the process group mid-write and skipping
the data-plane capture entirely. Give each catch op an explicit
timeout above the budgets it contains, so an inner timeout fires
instead and leaves a partial, still-servable snapshot behind. The
data-plane leg drops from 600s to 300s at that call site so both legs
fit one envelope; the BATS hook keeps the wider budget, having no
containing op.
Set crust-gather --duration explicitly as well. It is the collection
phase's own budget, it defaults to 60s, and on elapse it abandons the
collection and skips its finish step, leaving a truncated tree with no
indication anything is missing. The outer wall-clock has to exceed it
because the API discovery that runs first is not covered by that
budget at all -- with an unhealthy aggregated APIService, the state a
failed run is usually in, discovery alone can consume the whole
timeout before a single object is written. Send each collector's
output to a log beside its snapshot instead of /dev/null, with a
complete/truncated/failed verdict, so that question is answerable
from the artifact.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The dataplane diagnostic's announcer-node detection reported the wrong
node in two failover situations and could match the wrong IP:
- A node that announced an IP and then withdrew it (with no
re-announce) was still reported as the announcer, because a withdraw
did not retract a prior announce.
- Speaker logs are concatenated per-pod, not globally time-sorted, so
a stale former owner's announce could outrank the true current
owner purely by its position in the input.
- The IP was matched as a substring, so a query for 192.0.2.5 also
matched a 192.0.2.50 line.
Reimplement the detection to report a node only when that node's LAST
own announce/withdraw event for the exact IP is an announce, which is
robust to both concat order and L2 failover, and anchor the IP match to
a whole IP-literal token instead of a substring.
Also fix the heavy-capture budget: the MAX_LBS cap counted every
enumerated LB, so a broken LB enumerated after a cap's worth of
reachable LBs was never captured. Count only LBs actually captured, via
a small pure budget helper, so reachable/skipped LBs do not consume the
budget. Document that this cap bounds work, not wall-clock -- at two or
more unreachable LBs the real wall-clock bound is the outer backstop at
the call site.
Add bats coverage pinning the fixed announcer behavior (withdraw,
concat-order, exact-IP) and the captured-budget accounting.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The on-failure dataplane diagnostic only captured host->pod CNI state for
NotReady pods. A recurrent flake presents differently: a tenant Service
type=LoadBalancer IP goes unreachable while its backend pod/VMI stays Ready
and Running, so there is no NotReady pod and the hook captured nothing --
leaving the host->cross-node->bridged-worker-VMI->nodePort path
uncharacterised, which is exactly where root-causing has stalled.
Add a gated LoadBalancer-datapath section. It enumerates LoadBalancer
services with an ingress IP, derives the EndpointSlice backend, the
nodePort/externalTrafficPolicy, and the announcer node from the hostNetwork
MetalLB speaker logs, then probes the LB IP from a host netns. Only an LB
whose every probe fails gets the heavy capture, and that probe doubles as
the traffic for the tcpdumps. For a failing LB it captures, on both the
announcer and endpoint nodes, the host cilium LB map, kernel
conntrack/neigh, br-int flows, ovn-controller timing, and a live tcpdump on
the geneve tunnel and the backend tap -- localising the failing hop to
host-cilium programming vs kube-ovn delivery. In-guest VMI capture is
documented as a deferred stretch that needs tenant-cluster access.
The existing NotReady-pod capture is unchanged. The new section is
best-effort and time-boxed like the rest of the script, and a unit test
pins the pure enumeration, announcer-node, and capture-or-skip logic against
mock input.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Extend the host->pod data-plane capture with the three datums that make an
OVN incremental-processing lag dispositive on the next recurrence, all
diagnostic-only and gated on the existing on-failure path:
- ovn-controller.log from the ovs-ovn pod (grep for physical_flow_output,
if_status_mgr, took <N>ms, recompute, long poll interval; plus a bounded
tail) -- shows whether ovn-installed was set before the LSP local-delivery
flow was programmed.
- per-interface ovn-installed flag + ovn-installed-ts on the pod's OVS
interface, looked up by iface-id <pod>.<ns>.
- ovs-ovn cgroup cpu.stat (nr_throttled / throttled_usec, cgroup v2 with v1
fallback) -- shows whether ovn-controller was CPU-starved.
Node-global captures land in node-<node>.txt, the per-interface flag in the
per-pod file. Every command stays timeboxed and || true, no behavior or
exit-code change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The E2E install intermittently fails on a CNI host->local-pod data-plane
transient: kubelet on a node reaches a local pod's readiness/startup probe
with "connection refused" for several minutes while overlay pod->pod
traffic works, then it self-heals. This is rooted in the cilium+kube-ovn
chaining config (forced enable-host-legacy-routing, CNI
InstallEndpointRoute:false), which delegates host->local-pod routing to
kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object
state but not the node's L3 forwarding state, so the mechanism cannot be
root-caused after the fact.
On test failure, for each NotReady pod that is scheduled, has a podIP, and is
not terminal (Succeeded/Failed excluded), collect the host->pod data-plane
state from the pod's node into the crust-gather snapshot dir (so it ships in
the uploaded report):
- host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>,
ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for
the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under
enable-host-legacy-routing the host->local-pod path traverses the kernel
netfilter stack, not cilium BPF, so this is the authoritative conntrack
table and route/rule state for the transient;
- cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a
bounded monitor --type drop, and hubble dropped verdicts when present
(complementary: confirms the path is not handled in BPF);
- OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding /
Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list.
Diagnostic-only: it runs after the existing crust-gather collect and only on
an already-failed test, never mutates the cluster, and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
|| true, the whole run has a wall-clock backstop, node-global captures are
deduped per node, and the pod count is capped. It no-ops cleanly when no
pods are affected or when a tool/pod is absent.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin