kube-multus serves the CNI call for every pod sandbox on its node, and
its 100m CPU limit was hardcoded in the patched upstream template.
After a CNI outage a node had ~600 dead sandboxes and 220 pods waiting:
multus was throttled in every CFS period, calls outlived kubelet's
two-minute CRI deadline, each retry grew the queue, and the node never
recovered. With the CPU limit lifted the 1Gi memory limit OOM-looped;
2 CPU and 4Gi cleared the backlog in about three minutes.
Resources now come from values, defaulting to the same requests, no
CPU limit and a 4Gi memory limit. The memory limit stays so the pod
cgroup keeps memory.max and remains outside the Talos OOM handler's
candidate set.
Since an explicit null in a values override could drop either memory
key, the render now fails without limits.memory or requests.memory.
Without the request it defaults to the limit, and a node with less
allocatable memory than that is left without multus.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Talos userspace OOM handler (v1.12+) ranks cgroups with
memory_max.hasValue() ? 0.0 :
{Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current
and discards every cgroup scoring zero, so only a memory limit takes a
pod out of the victim set. Selection is decoupled from the trigger: a
tenant workload thrashing against its own limit drives node memory PSI,
and a limit-free system DaemonSet is killed for it. metallb-speaker and
linstor-satellite were the observed victims.
Set requests and limits on the node DaemonSets that had none. Requests
come from 14d peak working set measured on a 3-node dev cluster; limits
are deliberately loose, sized as a runaway backstop rather than a fitted
ceiling, because several of these scale with cluster size that a dev
cluster understates and because undersizing converts a rare
pressure-driven kill into a deterministic one. A limit costs nothing
until it is hit -- only the request reserves.
memory.max lands on the POD cgroup only when EVERY container in the pod
carries a limit, so sidecars and init containers are covered too; one
limit-free container returns the whole pod to the victim set.
Two DaemonSets are not chart-rendered and needed their own seam:
linstor-satellite via the LinstorSatelliteConfiguration podTemplate, and
virt-handler via KubeVirt customizeComponents.patches, since virt-operator
owns that DaemonSet. Note linstor-controller stays deliberately limit-free
-- it is a JVM that must not be OOM-killed mid-operation, and cluster_test
pins that.
Two things the multus manifest needed beyond the values change.
customize-deployment.patch is the source for that template, so the
limits are carried there rather than only in the generated file, which
`make update` would discard. And the reason kube-multus had no memory
limit was that this patch DELETED upstream's 50Mi rather than raising
it -- 50Mi being well under the 230Mi measured peak -- which is what
removed memory.max from the pod cgroup in the first place.
Not covered, because upstream hardcodes the containers with no resources
knob: the four frr-k8s cp-* init containers, the kube-ovn hostpath-init
and install-cni init containers, and cozy-proxy, whose vendored chart
exposes no resources value at all. hami and kilo are left alone as
optional components with no usage measurements behind them. All of these
still get a limit from the default LimitRange the operator maintains in
every system namespace.
Every limit is pinned by a helm-unittest assertion, and every assertion
was mutation-tested: changing the rendered value turns its suite red.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The check that multus is pointed at the cilium conflist matched its
JSON fragment anywhere in the rendered daemon config. The same text
nested inside any other key satisfies it, so a config whose effective
multusMasterCNI names a different conflist passes. Multus uses that
value as written when it is non-empty -- it delegates to whatever is
named, so a wrong-but-existing name silently routes through the other
CNI instead of cilium, which is the state this assertion exists to
prevent. Auto-discovery happens only when the value is empty, and a
name that does not exist fails manager startup rather than falling
back.
Anchor it to the top level, where the template writes its keys. That
pins the position rather than parsing the document; deciding which key
is effective needs a JSON parser, which helm-unittest does not have.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Multus-backed NetworkAttachmentDefinitions delegate to the upstream
containernetworking reference plugins (bridge, macvlan, ipvlan, ...),
and nothing guarantees a node has them in the directory Multus
searches. On k3s they live under /var/lib/rancher/k3s/data/cni while
Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails
there with `failed to find plugin "bridge" in path [/opt/cni/bin]`.
Bundle 14 of the release's plugins into the multus image and add an
install-cni-plugins init container that installs them into the host
/opt/cni/bin, mirroring the existing install-multus-binary container.
loopback, dhcp, tap and dummy are left out; the Dockerfile records why.
Gate it on a new chart value, stageCniPlugins, resolved per bundle and
on for both. Talos bundles only bridge, firewall, flannel, host-local,
loopback and portmap, so the other ten are missing there exactly as on
generic Linux; the four that overlap are rewritten with the same
upstream version, v1.9.1 on both sides, rather than an older one. A
marker in the image Dockerfile records the Talos release that pin was
compared against, and a test holds it to the release the platform
ships. multus therefore moves out of system.common-packages into the
two variant branches, the way linstor already does for talos.enabled.
networking.stageCniPlugins overrides the variant default in either
direction.
Where staging is on the install is unconditional: a plugin is replaced
on every pod recreation, not only when the destination is missing. A
destination that is already a directory is reported and left alone,
since replacing it is not something this can do safely. Turning the
value off afterwards does not undo it; nothing here removes or
restores a plugin.
The container is not privileged and takes no mount propagation. It
runs in the spc_t SELinux domain, without which an enforcing node
refuses the write. Each plugin is installed under a temporary name and
swapped in with a rename: /opt/cni/bin is shared with cilium and
kube-ovn. The plugin tarball is checksummed before extraction.
A plugin that cannot be installed is reported on stderr and to
/dev/termination-log, and the container still exits 0: Multus stays the
node's primary CNI and nothing removes its config, so an init container
that never completes would leave the shim with no daemon behind it.
The manifest change lives in customize-deployment.patch so it survives
make update.
BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI
plugins in /opt/cni/bin on existing nodes, Talos included -- four of the
names it writes are ones the Talos node image also carries. Opt out
before upgrading: declining afterwards does not restore what was
replaced.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Re-vendors the multus-cni thick-mode daemonset from upstream v4.2.3 to v4.3.0
and rebuilds the cozystack image (multi-arch, golang:1.26 builder; v4.3.0's
go.mod and the published security fixes warrant the stdlib bump from 1.24). The
fix-del-without-cache and customize-deployment patches both re-apply cleanly to
v4.3.0. The rebuilt image is pinned by digest in the daemonset.
Folds the previously hand-edited `multusMasterCNI: "05-cilium.conflist"`
CNI-ordering fix into customize-deployment.patch so it survives every make
update (it was a raw daemonset edit that the re-vendor would otherwise wipe,
re-introducing the kube-ovn-before-cilium startup race).
Adds helm-unittest coverage (the package shipped none): asserts the daemonset
runs the cozystack-built image and that the daemon-config pins multusMasterCNI
to the cilium conflist.
Adaptations:
- images/multus-cni/Dockerfile: builder golang:1.24 -> 1.26, ARG VERSION v4.2.3 -> v4.3.0
- templates/multus-daemonset-thick.yml: re-vendored v4.3.0 + customize patch; pin rebuilt multi-arch image
- patches/customize-deployment.patch: carry the multusMasterCNI CNI-ordering fix
- Makefile: add test target
- tests/multus_test.yaml: new helm-unittest coverage
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin