The dataplane diagnostic's announcer-node detection reported the wrong
node in two failover situations and could match the wrong IP:
- A node that announced an IP and then withdrew it (with no
re-announce) was still reported as the announcer, because a withdraw
did not retract a prior announce.
- Speaker logs are concatenated per-pod, not globally time-sorted, so
a stale former owner's announce could outrank the true current
owner purely by its position in the input.
- The IP was matched as a substring, so a query for 192.0.2.5 also
matched a 192.0.2.50 line.
Reimplement the detection to report a node only when that node's LAST
own announce/withdraw event for the exact IP is an announce, which is
robust to both concat order and L2 failover, and anchor the IP match to
a whole IP-literal token instead of a substring.
Also fix the heavy-capture budget: the MAX_LBS cap counted every
enumerated LB, so a broken LB enumerated after a cap's worth of
reachable LBs was never captured. Count only LBs actually captured, via
a small pure budget helper, so reachable/skipped LBs do not consume the
budget. Document that this cap bounds work, not wall-clock -- at two or
more unreachable LBs the real wall-clock bound is the outer backstop at
the call site.
Add bats coverage pinning the fixed announcer behavior (withdraw,
concat-order, exact-IP) and the captured-budget accounting.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The on-failure dataplane diagnostic only captured host->pod CNI state for
NotReady pods. A recurrent flake presents differently: a tenant Service
type=LoadBalancer IP goes unreachable while its backend pod/VMI stays Ready
and Running, so there is no NotReady pod and the hook captured nothing --
leaving the host->cross-node->bridged-worker-VMI->nodePort path
uncharacterised, which is exactly where root-causing has stalled.
Add a gated LoadBalancer-datapath section. It enumerates LoadBalancer
services with an ingress IP, derives the EndpointSlice backend, the
nodePort/externalTrafficPolicy, and the announcer node from the hostNetwork
MetalLB speaker logs, then probes the LB IP from a host netns. Only an LB
whose every probe fails gets the heavy capture, and that probe doubles as
the traffic for the tcpdumps. For a failing LB it captures, on both the
announcer and endpoint nodes, the host cilium LB map, kernel
conntrack/neigh, br-int flows, ovn-controller timing, and a live tcpdump on
the geneve tunnel and the backend tap -- localising the failing hop to
host-cilium programming vs kube-ovn delivery. In-guest VMI capture is
documented as a deferred stretch that needs tenant-cluster access.
The existing NotReady-pod capture is unchanged. The new section is
best-effort and time-boxed like the rest of the script, and a unit test
pins the pure enumeration, announcer-node, and capture-or-skip logic against
mock input.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Extend the host->pod data-plane capture with the three datums that make an
OVN incremental-processing lag dispositive on the next recurrence, all
diagnostic-only and gated on the existing on-failure path:
- ovn-controller.log from the ovs-ovn pod (grep for physical_flow_output,
if_status_mgr, took <N>ms, recompute, long poll interval; plus a bounded
tail) -- shows whether ovn-installed was set before the LSP local-delivery
flow was programmed.
- per-interface ovn-installed flag + ovn-installed-ts on the pod's OVS
interface, looked up by iface-id <pod>.<ns>.
- ovs-ovn cgroup cpu.stat (nr_throttled / throttled_usec, cgroup v2 with v1
fallback) -- shows whether ovn-controller was CPU-starved.
Node-global captures land in node-<node>.txt, the per-interface flag in the
per-pod file. Every command stays timeboxed and || true, no behavior or
exit-code change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The E2E install intermittently fails on a CNI host->local-pod data-plane
transient: kubelet on a node reaches a local pod's readiness/startup probe
with "connection refused" for several minutes while overlay pod->pod
traffic works, then it self-heals. This is rooted in the cilium+kube-ovn
chaining config (forced enable-host-legacy-routing, CNI
InstallEndpointRoute:false), which delegates host->local-pod routing to
kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object
state but not the node's L3 forwarding state, so the mechanism cannot be
root-caused after the fact.
On test failure, for each NotReady pod that is scheduled, has a podIP, and is
not terminal (Succeeded/Failed excluded), collect the host->pod data-plane
state from the pod's node into the crust-gather snapshot dir (so it ships in
the uploaded report):
- host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>,
ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for
the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under
enable-host-legacy-routing the host->local-pod path traverses the kernel
netfilter stack, not cilium BPF, so this is the authoritative conntrack
table and route/rule state for the transient;
- cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a
bounded monitor --type drop, and hubble dropped verdicts when present
(complementary: confirms the path is not handled in BPF);
- OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding /
Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list.
Diagnostic-only: it runs after the existing crust-gather collect and only on
an already-failed test, never mutates the cluster, and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
|| true, the whole run has a wall-clock backstop, node-global captures are
deduped per node, and the pod count is capped. It no-ops cleanly when no
pods are affected or when a tool/pod is absent.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin