Aleksei Sviridkin
079e4079d3
The E2E install intermittently fails on a CNI host->local-pod data-plane transient: kubelet on a node reaches a local pod's readiness/startup probe with "connection refused" for several minutes while overlay pod->pod traffic works, then it self-heals. This is rooted in the cilium+kube-ovn chaining config (forced enable-host-legacy-routing, CNI InstallEndpointRoute:false), which delegates host->local-pod routing to kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object state but not the node's L3 forwarding state, so the mechanism cannot be root-caused after the fact. On test failure, for each NotReady pod that is scheduled, has a podIP, and is not terminal (Succeeded/Failed excluded), collect the host->pod data-plane state from the pod's node into the crust-gather snapshot dir (so it ships in the uploaded report): - host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>, ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under enable-host-legacy-routing the host->local-pod path traverses the kernel netfilter stack, not cilium BPF, so this is the authoritative conntrack table and route/rule state for the transient; - cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a bounded monitor --type drop, and hubble dropped verdicts when present (complementary: confirms the path is not handled in BPF); - OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding / Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list. Diagnostic-only: it runs after the existing crust-gather collect and only on an already-failed test, never mutates the cluster, and cannot change the pass/fail outcome. Every live capture is time-boxed, every command is || true, the whole run has a wall-clock backstop, node-global captures are deduped per node, and the pod count is capped. It no-ops cleanly when no pods are affected or when a tool/pod is absent. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have Feature Requests please use the Discussion's Feature Request section.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.