mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
Ivan Okhotnikov
8fb1e11f7f
Follows #3209 (LINSTOR ErrorReports) with the second half of the "cozyreport is blind to the host-networking path" gap surfaced by the PR #3044 vlselect flake investigation (run 28772259934): `vlselect-generic-*` on one satellite hit `dial tcp <podIP>:9471: connect: connection refused` on every livenessProbe for 49s and got CrashLoopBackOff, while the sibling replica on another satellite — same pod spec, same probe config, same image — ran clean. The cozyreport artefact carried the CiliumEndpoint `state: ready` line, the pod-side listener log confirming the HTTP server was bound at 0.0.0.0:9471 within 5 ms of start, and the kube-ovn CNI ADD trace completing for the pod IP well before the first probe — every one of which says the path SHOULD have worked. Attribution of the drop to a specific host-side cause (stale conntrack from a previous pod that shared the IP, a Cilium policy drop that only Hubble sees, or an OVS-flow race between the CNI ADD and the iptables NAT chain) was not possible from the artefact because none of the host-side networking state is captured. Talos-side kubelet logs come out as one-line stubs (`error constructing client: failed to determine endpoints` — a separate cozyreport bug), so kubelet-view evidence is unavailable too. Add a `net-diag` section that runs after the sandbox-host block and copies four artefacts per satellite node: - `<node>-iptables.txt` — `iptables-save` from `kube-ovn-cni` (all tables in one blob so grepping across chains does not need a per-table unpack step). - `<node>-conntrack.txt` — `conntrack -L` first 5000 lines, from the same pod; the head cap keeps the file small on nodes with heavy tenant traffic and preserves the failure-window entries on the first-in-first-out flush semantics of the kernel table. - `<node>-hubble-flows.txt` — `hubble observe --last 2000 --output=compact` from `cilium-agent`. Hubble flow log carries the verdict (FORWARDED / DROPPED / ERROR) with reason field per flow, which is the only source that tells apart a Cilium policy drop from a datapath race. - `<node>-cilium-endpoints.txt` — `cilium-dbg endpoint list` for the per-endpoint policy state (identity, enforcement, controllers, status) that resolves "was the drop a policy decision" once hubble flags one. Every collection call uses a DaemonSet pod that already exists on the node with hostNetwork and the required binary preinstalled: `iptables-save` + `conntrack` live in `kube-ovn-cni`'s cni-server container; `hubble observe` + `cilium-dbg` live in the `cilium-agent` container. Both label selectors were confirmed against a live production cluster; both tool paths were confirmed present in the current piraeus / cilium image tags. Every capture is guarded with `|| true` and stderr redirection so a broken pod / missing tool / transient exec failure never fails the surrounding cozyreport run, and the parent directory is pruned when neither CNI is present on the cluster (e.g. a cozyreport captured against a non-cozystack cluster). Shipped as diagnostic-only test infra; no product-side change, no schema change, no chart change. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>