mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 18:47:04 +09:00
Aleksei Sviridkin
f61e13fdef
## What this PR does This raises the cilium-agent memory limit from 4Gi to 24Gi. The 4Gi limit from #3495 has no room for the agent's BPF maps. On nodes with roughly a terabyte of RAM or more, the agent is OOM-killed on every start after a reboot, and the node loses pod networking. The maps grow with node RAM. Cilium sizes its CT, NAT, neighbor and socket rev-NAT maps as a share of total RAM (`bpf-map-dynamic-size-ratio`, 0.0025 by default), and caps each map at 16Mi entries. These are LRU hash maps, so the kernel preallocates them in full. Since Linux 5.11 that memory is charged to the memcg of the process that creates the maps, which is the agent container. This is also why the upgrade itself works. A map reused from bpffs stays charged to the cgroup that created it. The failure shows up on the first reboot after the upgrade, when the maps are recreated inside the limited cgroup. Policy maps don't belong on that list. The issue and the upstream chart's configmap comment both say the ratio sizes them, but at this version they are LPM tries sized by `bpf.policyMapMax`, and they allocate per entry. One DaemonSet serves every node, so the limit can't follow node RAM. The map sizes are capped, though, so a fixed limit can cover any node. With every map at 16Mi entries for both IPv4 and IPv6, the maps take just under 20Gi of kernel memory. Add the 4Gi the agent already had and you get 24Gi. The request stays at 256Mi. The limit doesn't reserve memory, and the pod cgroup still gets `memory.max`, which is what #3495 was after. What it costs is the leak backstop. The agent is now stopped at 24Gi minus its own map memory, so on most nodes a leak in its heap is no longer caught before the node feels it. If someone prefers a tighter backstop over IPv6 headroom, they can lower `cilium.resources.limits.memory` in the networking component values. About 13Gi still covers the IPv4-only map ceiling. I didn't pin the map sizes (`bpf.ctTcpMax`, `bpf.natMax` and so on). No single value works for every node. A value small enough for a 16Gi node cuts the conntrack capacity of a 1.5TiB node many times over. A pinned size also differs from what almost every node above ~14Gi of RAM runs today, so the agent would recreate those maps on upgrade and drop their conntrack and NAT state. A lower ratio has the same problem and doesn't lower the ceiling. The new test keeps the ratio at its default and the sizes unpinned, so changing either has to be deliberate. The figures are computed, not measured. I read the sizing code at cilium 1.19.5 (`pkg/option/config.go`), and 1.20.2, which #3343 moves to, has the same code. Memory comes from that formula plus the kernel hash map layout: a 48-byte element header, the key and value rounded up to 8 bytes, and a 16-byte bucket per power of two of `max_entries`. By that math the IPv4 maps alone pass 4Gi at roughly a terabyte of RAM. Bucket rounding moves the exact point, so I wouldn't read more precision into it. For the 1.5TiB node in the report, the issue's 3.9GiB is the agent's own budget (0.25% of RAM), and the kernel's actual charge comes out higher. Follow-ups, not in this PR: - Operators who want smaller maps on large nodes can size them per node with a `CiliumNodeConfig` selected by node label. This needs a documentation note. - The reporter's `bpftool` measurement of the real map memory is still pending. That is the measured check for the numbers above. The cilium helm-unittest suite passes on this branch, and also with this change applied over the 1.20.2 chart from #3343. Each new assertion fails when its value changes: the limit, the CT, NAT and neighbor sizes, the socket rev-NAT size set through `extraConfig`, and the ratio. Closes #4800 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff changes only `packages/system/cilium/values.yaml` and its test. None of the trigger-map entries apply: no app, schema, platform value, variant, release asset or tooling changes, and no website page documents the agent limit. ### Release note ```release-note fix(cilium): raise the cilium-agent memory limit from 4Gi to 24Gi so it covers the agent's BPF maps, which cilium sizes from node RAM. On nodes with around a terabyte of RAM or more, the 4Gi limit got the agent OOM-killed on every start after a reboot, leaving the node without pod networking. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased the Cilium agent memory limit to 24 GiB. * Updated validation for BPF map sizing: checks the dynamic size ratio and confirms that specific map-size settings are absent. <!-- end of auto-generated comment: release notes by coderabbit.ai -->