Files
Aleksei Sviridkin f61e13fdef fix(cilium): size the agent memory limit for its BPF maps (#4803)
## What this PR does

This raises the cilium-agent memory limit from 4Gi to 24Gi. The 4Gi
limit from #3495 has no room for the agent's BPF maps. On nodes with
roughly a terabyte of RAM or more, the agent is OOM-killed on every
start after a reboot, and the node loses pod networking.

The maps grow with node RAM. Cilium sizes its CT, NAT, neighbor and
socket rev-NAT maps as a share of total RAM
(`bpf-map-dynamic-size-ratio`, 0.0025 by default), and caps each map at
16Mi entries. These are LRU hash maps, so the kernel preallocates them
in full. Since Linux 5.11 that memory is charged to the memcg of the
process that creates the maps, which is the agent container.

This is also why the upgrade itself works. A map reused from bpffs stays
charged to the cgroup that created it. The failure shows up on the first
reboot after the upgrade, when the maps are recreated inside the limited
cgroup.

Policy maps don't belong on that list. The issue and the upstream
chart's configmap comment both say the ratio sizes them, but at this
version they are LPM tries sized by `bpf.policyMapMax`, and they
allocate per entry.

One DaemonSet serves every node, so the limit can't follow node RAM. The
map sizes are capped, though, so a fixed limit can cover any node. With
every map at 16Mi entries for both IPv4 and IPv6, the maps take just
under 20Gi of kernel memory. Add the 4Gi the agent already had and you
get 24Gi. The request stays at 256Mi.

The limit doesn't reserve memory, and the pod cgroup still gets
`memory.max`, which is what #3495 was after. What it costs is the leak
backstop. The agent is now stopped at 24Gi minus its own map memory, so
on most nodes a leak in its heap is no longer caught before the node
feels it. If someone prefers a tighter backstop over IPv6 headroom, they
can lower `cilium.resources.limits.memory` in the networking component
values. About 13Gi still covers the IPv4-only map ceiling.

I didn't pin the map sizes (`bpf.ctTcpMax`, `bpf.natMax` and so on). No
single value works for every node. A value small enough for a 16Gi node
cuts the conntrack capacity of a 1.5TiB node many times over. A pinned
size also differs from what almost every node above ~14Gi of RAM runs
today, so the agent would recreate those maps on upgrade and drop their
conntrack and NAT state. A lower ratio has the same problem and doesn't
lower the ceiling. The new test keeps the ratio at its default and the
sizes unpinned, so changing either has to be deliberate.

The figures are computed, not measured. I read the sizing code at cilium
1.19.5 (`pkg/option/config.go`), and 1.20.2, which #3343 moves to, has
the same code. Memory comes from that formula plus the kernel hash map
layout: a 48-byte element header, the key and value rounded up to 8
bytes, and a 16-byte bucket per power of two of `max_entries`. By that
math the IPv4 maps alone pass 4Gi at roughly a terabyte of RAM. Bucket
rounding moves the exact point, so I wouldn't read more precision into
it. For the 1.5TiB node in the report, the issue's 3.9GiB is the agent's
own budget (0.25% of RAM), and the kernel's actual charge comes out
higher.

Follow-ups, not in this PR:

- Operators who want smaller maps on large nodes can size them per node
with a `CiliumNodeConfig` selected by node label. This needs a
documentation note.
- The reporter's `bpftool` measurement of the real map memory is still
pending. That is the measured check for the numbers above.

The cilium helm-unittest suite passes on this branch, and also with this
change applied over the 1.20.2 chart from #3343. Each new assertion
fails when its value changes: the limit, the CT, NAT and neighbor sizes,
the socket rev-NAT size set through `extraConfig`, and the ratio.

Closes #4800

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff changes only `packages/system/cilium/values.yaml` and its test.
None of the trigger-map entries apply: no app, schema, platform value,
variant, release asset or tooling changes, and no website page documents
the agent limit.

### Release note

```release-note
fix(cilium): raise the cilium-agent memory limit from 4Gi to 24Gi so it covers the agent's BPF maps, which cilium sizes from node RAM. On nodes with around a terabyte of RAM or more, the 4Gi limit got the agent OOM-killed on every start after a reboot, leaving the node without pod networking.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Increased the Cilium agent memory limit to 24 GiB.
* Updated validation for BPF map sizing: checks the dynamic size ratio
and confirms that specific map-size settings are absent.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-09 20:50:44 +03:00

135 lines
6.5 KiB
YAML

cilium:
# Required for the supported service-routing topology. Cozystack does not
# provide a supported alternative owner for ClusterIP translation when this
# is false; on kube-ovn, kube-proxy is absent and the OVN service load balancer
# is explicitly disabled. A false value can therefore break Service
# connectivity. tests/kube_proxy_replacement_test.yaml asserts the rendered
# value across the defaults and every variant valuesFiles combination.
#
# The source-side admission-webhook denies in packages/system/ingress-nginx
# and packages/system/kubeovn-webhook also rely on Cilium translating the
# Service before egress policy. Both policies retain a target-side ingressDeny,
# so even a second unsupported change that introduced another translator would
# not make the webhooks fail open.
#
# No test here can observe a live cluster's values, so an operator override in
# the field is caught only by templates/kube-proxy-replacement-beacon.yaml,
# which renders a status ConfigMap whenever this is not true. The beacon keeps
# a live unsupported override visible; it is not an alternative supported
# mode, and the required remediation is to restore true.
kubeProxyReplacement: true
# Upstream cilium with hostFirewall enabled and the IPv6 datapath disabled
# (ipv6.enabled=false, the default here) drops ALL node IPv6 before any
# policy evaluation ("Unsupported L3 protocol", DROP_UNKNOWN_L3 in
# bpf/bpf_host.c), breaking ICMPv6 Neighbor Discovery and BGP unnumbered on
# IPv6/L3 fabrics. The cozystack cilium image carries a BPF patch
# (images/cilium/patches/) that passes IPv6 to the kernel stack instead.
# Consequence: Cilium host policies apply to IPv4 only; node IPv6 is not
# filtered by Cilium.
hostFirewall:
enabled: true
hubble:
enabled: false
externalIPs:
enabled: true
nodePort:
enabled: true
loadBalancer:
algorithm: maglev
serviceTopology: true
ipam:
mode: "kubernetes"
image:
repository: ghcr.io/cozystack/cozystack/cilium
tag: v1.6.0
digest: "sha256:136a7dfff4d6adcd448424c2354144160028a8e96eb6558b917fd3ee6d7b39fe"
envoy:
enabled: true
# Upstream default is {} — no requests/limits. The embedded envoy DaemonSet
# idles at ~100 MB RAM/node and grows under L7 traffic; without requests
# the DaemonSet schedules onto any node with no reserved headroom, and
# without a memory limit a runaway listener could OOM the node.
# The baseline matches the commented-out example in the upstream Helm values.
resources:
requests:
cpu: 100m
memory: 512Mi
limits:
memory: 1Gi
# The cilium-agent DaemonSet had the same gap the envoy DaemonSet above
# already closed. Beyond the node-headroom argument, a memory limit is what
# sets memory.max on the pod cgroup, and the Talos userspace OOM handler
# (v1.12+) only ever selects cgroups that lack it — so an unlimited agent is
# an eviction candidate on every node regardless of what caused the pressure.
#
# The limit has to hold the agent's BPF maps as well as its heap. Read at
# cilium 1.19.5: the CT, NAT, neighbor and socket rev-NAT maps are sized as a
# share of total node RAM (bpf-map-dynamic-size-ratio, 0.0025 by default),
# each capped at 16Mi entries (LimitTableMax). They are LRU hash maps, which
# the kernel always preallocates in full, and since Linux 5.11 that memory is
# charged to the memcg of the process that creates them, i.e. this container.
# A map reused from bpffs stays charged to whoever created it, so a limit too
# small for the maps first fails on the start that recreates them: after a
# reboot, or when an upgrade changes a map's size or layout. Policy maps do
# not scale with RAM: they are LPM tries sized by bpf.policyMapMax that
# allocate per entry.
#
# One DaemonSet serves every node, so the limit cannot follow node RAM and
# covers the map ceiling instead: just under 20Gi of kernel memory with every
# map at 16Mi entries for both IPv4 and IPv6 (IPv6 is off by default, but
# turning it on must not bring the OOM back), plus 4Gi for the agent itself,
# the figure from the upstream values' commented example. A limit is a
# ceiling, not a reservation, so it costs no schedulable memory. What it does
# cost is the backstop against a leak in the agent itself: on every node the
# agent is stopped at 24Gi minus its own map memory, which on most nodes is
# no practical backstop. memory.max still keeps the pod out of the Talos
# handler's victim set, which is what the limit is for. Pinning the map sizes
# instead would cap connection capacity on large nodes and, on upgrade,
# recreate the maps and drop their conntrack and NAT state.
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 24Gi
# Shared by the agent's init containers (config, mount-cgroup, mount-bpf-fs,
# apply-sysctl-overwrites, clean-cilium-state). All are short-lived setup
# steps, but each one still has to carry a limit for the pod cgroup to get
# memory.max at all.
initResources:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 256Mi
gatewayAPI:
enabled: true
# Upstream default is false — Envoy then advertises no ALPN protocols on
# TLS-terminating Gateway listeners, and browsers (which negotiate HTTP/2
# exclusively through TLS ALPN "h2") silently fall back to HTTP/1.1 on
# every request. ingress-nginx, which the Gateway API path replaces,
# advertised h2 out of the box, so false here is a client-visible
# regression, not a neutral default. With true, ALPN offers h2 then
# http/1.1 — clients without HTTP/2 keep negotiating http/1.1 unchanged.
# This only affects the client↔gateway hop: gateway↔backend stays
# HTTP/1.1 unless a Service opts in per GEP-1911 via appProtocol
# kubernetes.io/h2c (enableAlpn also switches that support on; no
# cozystack-shipped Service declares such a value today).
# Platform-managed clusters can flip this through the platform chart's
# gateway.http2 value, which overrides this default either way.
enableAlpn: true
rollOutCiliumPods: true
operator:
rollOutPods: true
replicas: 1
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 100m
memory: 128Mi
# kubectl and jq for the hook Jobs in templates/ipam-migration.yaml.
ipamMigration:
image: docker.io/alpine/k8s:1.36.2@sha256:44ef4942e171939b9c665a4a84beb80e2dcdb9a24330d4651cfdfd2e9deecc47