Ivan Okhotnikov
8fb1e11f7f
Follows #3209 (LINSTOR ErrorReports) with the second half of the "cozyreport is blind to the host-networking path" gap surfaced by the PR #3044 vlselect flake investigation (run 28772259934): `vlselect-generic-*` on one satellite hit `dial tcp <podIP>:9471: connect: connection refused` on every livenessProbe for 49s and got CrashLoopBackOff, while the sibling replica on another satellite — same pod spec, same probe config, same image — ran clean. The cozyreport artefact carried the CiliumEndpoint `state: ready` line, the pod-side listener log confirming the HTTP server was bound at 0.0.0.0:9471 within 5 ms of start, and the kube-ovn CNI ADD trace completing for the pod IP well before the first probe — every one of which says the path SHOULD have worked. Attribution of the drop to a specific host-side cause (stale conntrack from a previous pod that shared the IP, a Cilium policy drop that only Hubble sees, or an OVS-flow race between the CNI ADD and the iptables NAT chain) was not possible from the artefact because none of the host-side networking state is captured. Talos-side kubelet logs come out as one-line stubs (`error constructing client: failed to determine endpoints` — a separate cozyreport bug), so kubelet-view evidence is unavailable too. Add a `net-diag` section that runs after the sandbox-host block and copies four artefacts per satellite node: - `<node>-iptables.txt` — `iptables-save` from `kube-ovn-cni` (all tables in one blob so grepping across chains does not need a per-table unpack step). - `<node>-conntrack.txt` — `conntrack -L` first 5000 lines, from the same pod; the head cap keeps the file small on nodes with heavy tenant traffic and preserves the failure-window entries on the first-in-first-out flush semantics of the kernel table. - `<node>-hubble-flows.txt` — `hubble observe --last 2000 --output=compact` from `cilium-agent`. Hubble flow log carries the verdict (FORWARDED / DROPPED / ERROR) with reason field per flow, which is the only source that tells apart a Cilium policy drop from a datapath race. - `<node>-cilium-endpoints.txt` — `cilium-dbg endpoint list` for the per-endpoint policy state (identity, enforcement, controllers, status) that resolves "was the drop a policy decision" once hubble flags one. Every collection call uses a DaemonSet pod that already exists on the node with hostNetwork and the required binary preinstalled: `iptables-save` + `conntrack` live in `kube-ovn-cni`'s cni-server container; `hubble observe` + `cilium-dbg` live in the `cilium-agent` container. Both label selectors were confirmed against a live production cluster; both tool paths were confirmed present in the current piraeus / cilium image tags. Every capture is guarded with `|| true` and stderr redirection so a broken pod / missing tool / transient exec failure never fails the surrounding cozyreport run, and the parent directory is pruned when neither CNI is present on the cluster (e.g. a cozyreport captured against a non-cozystack cluster). Shipped as diagnostic-only test infra; no product-side change, no schema change, no chart change. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.