mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
Aleksei Sviridkin
4ac02f59d5
## What this PR does The sandbox Talos config and the platform install step both used `10.244.0.0/16`, one as the Kubernetes cluster CIDR and the other as the kube-ovn `POD_CIDR`, so two allocators drew from one range. kube-ovn owns pod IPAM and allocates flat from `POD_CIDR`. Cilium is chained behind it (`generic-veth`, target `kube-ovn`) and allocates no pod addresses at all, but it still carves its own per-node router IP and Ingress IP out of `Node.spec.podCIDR`, which kube-controller-manager slices from the cluster CIDR. Sooner or later kube-ovn hands a pod an address cilium already holds. When that address is the Ingress IP the agent refuses the sandbox with `IP ipv4:<addr> is already in use`, and cilium never relents on it, so the pod proceeds only if something reschedules it onto a different address; in the run this was filed from, the same address was refused seven times across six pod UIDs before the suite ran out of budget. When it is the router IP nothing complains and only host-to-pod traffic breaks. This moves the sandbox cluster CIDR to `100.65.0.0/16`, which no other range in the tree uses, and adds a guard so the two cannot be brought back together by accident. Pod addresses do not change. They still come from kube-ovn out of `10.244.0.0/16`, and cilium does not require a chained endpoint's address to sit inside its own allocation CIDR, so only cilium's two per-node infrastructure addresses move. The quiet variant gets fixed twice over, which is worth spelling out because only half of it was obvious. Under `ipam.mode: kubernetes` cilium also installs a route for the node's own pod CIDR via `cilium_host`. On the current default that route covers `10.244.a.0/24` on a node whose pods actually live behind `ovn0`, so it shadows real pod addresses. Moving the cluster CIDR both stops the router IP from ever being handed to a pod and retargets that route at `100.65.a.0/24`, where it shadows nothing. Excluding the addresses on the kube-ovn side instead is not an option: cilium picks its offsets by random scan over the node's range, so there is no static set to exclude. The collision shows up in two shapes, and both have been seen. Loudly, on the Ingress IP, where the agent names the address in the event, which is the failure https://github.com/cozystack/cozystack/issues/3736 was filed for. Quietly, on the router IP, where the pod comes up and the only symptom is `connection refused` on a kubelet probe, with `ip route get <podIP>` on the node resolving to `local ... dev lo`; that variant is documented in https://github.com/cozystack/cozystack/issues/3445 and https://github.com/cozystack/cozystack/issues/3615. They were being tracked as unrelated flakes in different suites before the shared cause was identified. ## What the guard covers, and what it does not `hack/sandbox-cidr-disjoint.bats` pins the configuration, not the behaviour. It reads the values out of the tree and fails when they overlap. It does not reproduce the collision, which needs a running cluster. It checks the invariant rather than the literals, so either value may move as long as they stay disjoint, and it checks the cluster CIDR against the service and join ranges too, because landing on one of those instead is the next way to get this wrong. An extraction that comes up empty, duplicated or misshapen fails the test instead of shrinking the comparison, and comments and blank lines inside the list are skipped rather than read as the end of it. It is not a YAML parser, though: anything it does not read as a bare dotted quad fails the shape validation rather than being understood, which is the safe direction but not the same promise. Those three are the allocator ranges and nothing more. The sandbox node network, the load-balancer pool and the transit range of a component that is off by default are not read, so the guard is not a complete map of what the cluster CIDR has to avoid. An IPv6 entry fails the shape validation rather than getting an overlap check, so dual-stack means extending the guard rather than passing through it. ## Residual risk I did not prove from source that cilium is the only consumer of `Node.spec.podCIDR` in this topology. What I did check is that kube-proxy is disabled and that `local-ccm` runs the node-lifecycle controller only, with no route controller. This diff selects the full e2e suite, 21 Chainsaw suites with `gateway` and `mongodb` among them, so a consumer I missed surfaces as a red run on this PR rather than staying an assumption. Split of what covers what: the static checks and the unit guard cover the configuration, and the e2e run covers the behaviour. Nothing short of a green suite answers whether cilium's router and Ingress addresses behave once they sit outside the range kube-ovn allocates from. ## A check that is not "the suite went green" `hack/e2e-cilium-endpoint-leak-healer.sh` already prints a line of its own for this exact collision, so there is a signal to read that does not depend on the overall verdict. In the `cozyreport` artifact it lands at `snapshots/<suite>/host/namespaces/kube-system/v1/pod/cilium-leak-healer-*/healer/current.log`, and the lines to grep for are `IP is this node's Cilium ingress IP`. In the run this was filed from there are 42 of them, all naming the same address. After this change that line cannot be produced by an address inside the kube-ovn range, because there is no longer an address that is in both. So the expectation is zero from the first run, not zero on average: one occurrence contradicts the change rather than being noise, and is worth stopping on. A green suite can be green for many reasons; this line disappearing has only one. ## Scope This fixes the sandbox and nothing else. For a real installation the cluster CIDR lives in the operator's own machine configuration, outside this repository, so the same collision is still reachable there. https://github.com/cozystack/cozystack/issues/3740 tracks the product side, including the finding that the install documentation does not merely allow the overlap but instructs operators to create it, by requiring `podCIDR` to match the cluster's pod CIDR. There is a tradeoff in this worth naming rather than leaving to be discovered. Talos defaults `podSubnets` to `10.244.0.0/16` (`DefaultIPv4PodNet`, `pkg/machinery/constants/constants.go:888` at v1.13.6, the version this tree ships), and the platform defaults `networking.podCIDR` to the same value, so an operator who takes both defaults gets the overlap. Until now the sandbox shared that topology and was the one place the bug surfaced where anyone would see it. After this change it no longer reproduces the operator-facing form, and nothing else does either. That is the right call for CI, which should not be wedged by a defect it did not introduce, but it does mean the production exposure loses its only accidental detector. Closes https://github.com/cozystack/cozystack/issues/3736 To be explicit about what closing that issue does and does not mean: this removes the reproduction in the sandbox and leaves the product configuration untouched. The collision is still reachable for users, and https://github.com/cozystack/cozystack/issues/3740 is where that lives. The issue text describes the root mechanism, which is wider than its title, so the closure covers the e2e failure it was filed for and not the platform default. ### Screenshots Not a UI change. ### Downstream repositories No box is ticked deliberately, and no follow-up PR is opened, so a human decides. Walking the trigger map against this diff: both `hack/e2e-prepare-cluster.bats` entries in it are scoped to node prerequisites, the kernel modules a node must load, the containerd settings, the LVM `global_filter` and the cluster domain. This diff changes none of those, it changes a network range, so no entry fires literally. The underlying constraint does reach several of them all the same. `cozystack/talm` sets `podSubnets: [10.244.0.0/16]` in `charts/cozystack/values.yaml`, the same range the platform uses for `networking.podCIDR`, so the documented Talos bootstrap produces the overlap this PR removes from the sandbox. `cozystack/ansible-cozystack` reaches it from the other side: `roles/cozystack/defaults/main.yml` sets `cozystack_pod_cidr: "10.42.0.0/16"` and renders it into `networking.podCIDR`, while its inventories set `cluster-cidr: 10.42.0.0/16`, so those two are identical as well. And `cozystack/website` prescribes the equality outright, `install/kubernetes/generic.md` stating that `podCIDR` and `serviceCIDR` must match the Kubernetes cluster configuration. No issue is filed there yet on purpose: which side moves is not decided, and asking talm to change `podSubnets` would be the wrong request if the platform ends up moving instead. That decision belongs to https://github.com/cozystack/cozystack/issues/3740. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Updated cluster networking ranges to avoid conflicts with Kubernetes, kube-ovn, kilo, and Cilium networks. - Added validation to detect missing, invalid, or overlapping cluster and platform CIDR ranges. - Preserved safeguards for environments where network ranges overlap, with clearer diagnostic logging. - **Documentation** - Documented network separation requirements and clarified expected sandbox networking behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.