mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
## What this PR does This raises the cilium-agent memory limit from 4Gi to 24Gi. The 4Gi limit from #3495 has no room for the agent's BPF maps. On nodes with roughly a terabyte of RAM or more, the agent is OOM-killed on every start after a reboot, and the node loses pod networking. The maps grow with node RAM. Cilium sizes its CT, NAT, neighbor and socket rev-NAT maps as a share of total RAM (`bpf-map-dynamic-size-ratio`, 0.0025 by default), and caps each map at 16Mi entries. These are LRU hash maps, so the kernel preallocates them in full. Since Linux 5.11 that memory is charged to the memcg of the process that creates the maps, which is the agent container. This is also why the upgrade itself works. A map reused from bpffs stays charged to the cgroup that created it. The failure shows up on the first reboot after the upgrade, when the maps are recreated inside the limited cgroup. Policy maps don't belong on that list. The issue and the upstream chart's configmap comment both say the ratio sizes them, but at this version they are LPM tries sized by `bpf.policyMapMax`, and they allocate per entry. One DaemonSet serves every node, so the limit can't follow node RAM. The map sizes are capped, though, so a fixed limit can cover any node. With every map at 16Mi entries for both IPv4 and IPv6, the maps take just under 20Gi of kernel memory. Add the 4Gi the agent already had and you get 24Gi. The request stays at 256Mi. The limit doesn't reserve memory, and the pod cgroup still gets `memory.max`, which is what #3495 was after. What it costs is the leak backstop. The agent is now stopped at 24Gi minus its own map memory, so on most nodes a leak in its heap is no longer caught before the node feels it. If someone prefers a tighter backstop over IPv6 headroom, they can lower `cilium.resources.limits.memory` in the networking component values. About 13Gi still covers the IPv4-only map ceiling. I didn't pin the map sizes (`bpf.ctTcpMax`, `bpf.natMax` and so on). No single value works for every node. A value small enough for a 16Gi node cuts the conntrack capacity of a 1.5TiB node many times over. A pinned size also differs from what almost every node above ~14Gi of RAM runs today, so the agent would recreate those maps on upgrade and drop their conntrack and NAT state. A lower ratio has the same problem and doesn't lower the ceiling. The new test keeps the ratio at its default and the sizes unpinned, so changing either has to be deliberate. The figures are computed, not measured. I read the sizing code at cilium 1.19.5 (`pkg/option/config.go`), and 1.20.2, which #3343 moves to, has the same code. Memory comes from that formula plus the kernel hash map layout: a 48-byte element header, the key and value rounded up to 8 bytes, and a 16-byte bucket per power of two of `max_entries`. By that math the IPv4 maps alone pass 4Gi at roughly a terabyte of RAM. Bucket rounding moves the exact point, so I wouldn't read more precision into it. For the 1.5TiB node in the report, the issue's 3.9GiB is the agent's own budget (0.25% of RAM), and the kernel's actual charge comes out higher. Follow-ups, not in this PR: - Operators who want smaller maps on large nodes can size them per node with a `CiliumNodeConfig` selected by node label. This needs a documentation note. - The reporter's `bpftool` measurement of the real map memory is still pending. That is the measured check for the numbers above. The cilium helm-unittest suite passes on this branch, and also with this change applied over the 1.20.2 chart from #3343. Each new assertion fails when its value changes: the limit, the CT, NAT and neighbor sizes, the socket rev-NAT size set through `extraConfig`, and the ratio. Closes #4800 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff changes only `packages/system/cilium/values.yaml` and its test. None of the trigger-map entries apply: no app, schema, platform value, variant, release asset or tooling changes, and no website page documents the agent limit. ### Release note ```release-note fix(cilium): raise the cilium-agent memory limit from 4Gi to 24Gi so it covers the agent's BPF maps, which cilium sizes from node RAM. On nodes with around a terabyte of RAM or more, the 4Gi limit got the agent OOM-killed on every start after a reboot, leaving the node without pod networking. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased the Cilium agent memory limit to 24 GiB. * Updated validation for BPF map sizing: checks the dynamic size ratio and confirms that specific map-size settings are absent. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
135 lines
6.5 KiB
YAML
135 lines
6.5 KiB
YAML
cilium:
|
|
# Required for the supported service-routing topology. Cozystack does not
|
|
# provide a supported alternative owner for ClusterIP translation when this
|
|
# is false; on kube-ovn, kube-proxy is absent and the OVN service load balancer
|
|
# is explicitly disabled. A false value can therefore break Service
|
|
# connectivity. tests/kube_proxy_replacement_test.yaml asserts the rendered
|
|
# value across the defaults and every variant valuesFiles combination.
|
|
#
|
|
# The source-side admission-webhook denies in packages/system/ingress-nginx
|
|
# and packages/system/kubeovn-webhook also rely on Cilium translating the
|
|
# Service before egress policy. Both policies retain a target-side ingressDeny,
|
|
# so even a second unsupported change that introduced another translator would
|
|
# not make the webhooks fail open.
|
|
#
|
|
# No test here can observe a live cluster's values, so an operator override in
|
|
# the field is caught only by templates/kube-proxy-replacement-beacon.yaml,
|
|
# which renders a status ConfigMap whenever this is not true. The beacon keeps
|
|
# a live unsupported override visible; it is not an alternative supported
|
|
# mode, and the required remediation is to restore true.
|
|
kubeProxyReplacement: true
|
|
# Upstream cilium with hostFirewall enabled and the IPv6 datapath disabled
|
|
# (ipv6.enabled=false, the default here) drops ALL node IPv6 before any
|
|
# policy evaluation ("Unsupported L3 protocol", DROP_UNKNOWN_L3 in
|
|
# bpf/bpf_host.c), breaking ICMPv6 Neighbor Discovery and BGP unnumbered on
|
|
# IPv6/L3 fabrics. The cozystack cilium image carries a BPF patch
|
|
# (images/cilium/patches/) that passes IPv6 to the kernel stack instead.
|
|
# Consequence: Cilium host policies apply to IPv4 only; node IPv6 is not
|
|
# filtered by Cilium.
|
|
hostFirewall:
|
|
enabled: true
|
|
hubble:
|
|
enabled: false
|
|
externalIPs:
|
|
enabled: true
|
|
nodePort:
|
|
enabled: true
|
|
loadBalancer:
|
|
algorithm: maglev
|
|
serviceTopology: true
|
|
ipam:
|
|
mode: "kubernetes"
|
|
image:
|
|
repository: ghcr.io/cozystack/cozystack/cilium
|
|
tag: v1.6.0
|
|
digest: "sha256:136a7dfff4d6adcd448424c2354144160028a8e96eb6558b917fd3ee6d7b39fe"
|
|
envoy:
|
|
enabled: true
|
|
# Upstream default is {} — no requests/limits. The embedded envoy DaemonSet
|
|
# idles at ~100 MB RAM/node and grows under L7 traffic; without requests
|
|
# the DaemonSet schedules onto any node with no reserved headroom, and
|
|
# without a memory limit a runaway listener could OOM the node.
|
|
# The baseline matches the commented-out example in the upstream Helm values.
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 512Mi
|
|
limits:
|
|
memory: 1Gi
|
|
# The cilium-agent DaemonSet had the same gap the envoy DaemonSet above
|
|
# already closed. Beyond the node-headroom argument, a memory limit is what
|
|
# sets memory.max on the pod cgroup, and the Talos userspace OOM handler
|
|
# (v1.12+) only ever selects cgroups that lack it — so an unlimited agent is
|
|
# an eviction candidate on every node regardless of what caused the pressure.
|
|
#
|
|
# The limit has to hold the agent's BPF maps as well as its heap. Read at
|
|
# cilium 1.19.5: the CT, NAT, neighbor and socket rev-NAT maps are sized as a
|
|
# share of total node RAM (bpf-map-dynamic-size-ratio, 0.0025 by default),
|
|
# each capped at 16Mi entries (LimitTableMax). They are LRU hash maps, which
|
|
# the kernel always preallocates in full, and since Linux 5.11 that memory is
|
|
# charged to the memcg of the process that creates them, i.e. this container.
|
|
# A map reused from bpffs stays charged to whoever created it, so a limit too
|
|
# small for the maps first fails on the start that recreates them: after a
|
|
# reboot, or when an upgrade changes a map's size or layout. Policy maps do
|
|
# not scale with RAM: they are LPM tries sized by bpf.policyMapMax that
|
|
# allocate per entry.
|
|
#
|
|
# One DaemonSet serves every node, so the limit cannot follow node RAM and
|
|
# covers the map ceiling instead: just under 20Gi of kernel memory with every
|
|
# map at 16Mi entries for both IPv4 and IPv6 (IPv6 is off by default, but
|
|
# turning it on must not bring the OOM back), plus 4Gi for the agent itself,
|
|
# the figure from the upstream values' commented example. A limit is a
|
|
# ceiling, not a reservation, so it costs no schedulable memory. What it does
|
|
# cost is the backstop against a leak in the agent itself: on every node the
|
|
# agent is stopped at 24Gi minus its own map memory, which on most nodes is
|
|
# no practical backstop. memory.max still keeps the pod out of the Talos
|
|
# handler's victim set, which is what the limit is for. Pinning the map sizes
|
|
# instead would cap connection capacity on large nodes and, on upgrade,
|
|
# recreate the maps and drop their conntrack and NAT state.
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
limits:
|
|
memory: 24Gi
|
|
# Shared by the agent's init containers (config, mount-cgroup, mount-bpf-fs,
|
|
# apply-sysctl-overwrites, clean-cilium-state). All are short-lived setup
|
|
# steps, but each one still has to carry a limit for the pod cgroup to get
|
|
# memory.max at all.
|
|
initResources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 32Mi
|
|
limits:
|
|
memory: 256Mi
|
|
gatewayAPI:
|
|
enabled: true
|
|
# Upstream default is false — Envoy then advertises no ALPN protocols on
|
|
# TLS-terminating Gateway listeners, and browsers (which negotiate HTTP/2
|
|
# exclusively through TLS ALPN "h2") silently fall back to HTTP/1.1 on
|
|
# every request. ingress-nginx, which the Gateway API path replaces,
|
|
# advertised h2 out of the box, so false here is a client-visible
|
|
# regression, not a neutral default. With true, ALPN offers h2 then
|
|
# http/1.1 — clients without HTTP/2 keep negotiating http/1.1 unchanged.
|
|
# This only affects the client↔gateway hop: gateway↔backend stays
|
|
# HTTP/1.1 unless a Service opts in per GEP-1911 via appProtocol
|
|
# kubernetes.io/h2c (enableAlpn also switches that support on; no
|
|
# cozystack-shipped Service declares such a value today).
|
|
# Platform-managed clusters can flip this through the platform chart's
|
|
# gateway.http2 value, which overrides this default either way.
|
|
enableAlpn: true
|
|
rollOutCiliumPods: true
|
|
operator:
|
|
rollOutPods: true
|
|
replicas: 1
|
|
resources:
|
|
limits:
|
|
cpu: 1000m
|
|
memory: 1Gi
|
|
requests:
|
|
cpu: 100m
|
|
memory: 128Mi
|
|
# kubectl and jq for the hook Jobs in templates/ipam-migration.yaml.
|
|
ipamMigration:
|
|
image: docker.io/alpine/k8s:1.36.2@sha256:44ef4942e171939b9c665a4a84beb80e2dcdb9a24330d4651cfdfd2e9deecc47
|