mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-02 18:16:53 +09:00
Aleksei Sviridkin
079e4079d3
The E2E install intermittently fails on a CNI host->local-pod data-plane transient: kubelet on a node reaches a local pod's readiness/startup probe with "connection refused" for several minutes while overlay pod->pod traffic works, then it self-heals. This is rooted in the cilium+kube-ovn chaining config (forced enable-host-legacy-routing, CNI InstallEndpointRoute:false), which delegates host->local-pod routing to kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object state but not the node's L3 forwarding state, so the mechanism cannot be root-caused after the fact. On test failure, for each NotReady pod that is scheduled, has a podIP, and is not terminal (Succeeded/Failed excluded), collect the host->pod data-plane state from the pod's node into the crust-gather snapshot dir (so it ships in the uploaded report): - host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>, ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under enable-host-legacy-routing the host->local-pod path traverses the kernel netfilter stack, not cilium BPF, so this is the authoritative conntrack table and route/rule state for the transient; - cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a bounded monitor --type drop, and hubble dropped verdicts when present (complementary: confirms the path is not handled in BPF); - OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding / Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list. Diagnostic-only: it runs after the existing crust-gather collect and only on an already-failed test, never mutates the cluster, and cannot change the pass/fail outcome. Every live capture is time-boxed, every command is || true, the whole run has a wall-clock backstop, node-global captures are deduped per node, and the pod count is capped. It no-ops cleanly when no pods are affected or when a tool/pod is absent. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>