Files
cozystack/packages/system/multus/tests/multus_test.yaml
T
Myasnikov Daniil c9ba7a3cca fix(platform): give node DaemonSets memory requests and limits
The Talos userspace OOM handler (v1.12+) ranks cgroups with

  memory_max.hasValue() ? 0.0 :
    {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current

and discards every cgroup scoring zero, so only a memory limit takes a
pod out of the victim set. Selection is decoupled from the trigger: a
tenant workload thrashing against its own limit drives node memory PSI,
and a limit-free system DaemonSet is killed for it. metallb-speaker and
linstor-satellite were the observed victims.

Set requests and limits on the node DaemonSets that had none. Requests
come from 14d peak working set measured on a 3-node dev cluster; limits
are deliberately loose, sized as a runaway backstop rather than a fitted
ceiling, because several of these scale with cluster size that a dev
cluster understates and because undersizing converts a rare
pressure-driven kill into a deterministic one. A limit costs nothing
until it is hit -- only the request reserves.

memory.max lands on the POD cgroup only when EVERY container in the pod
carries a limit, so sidecars and init containers are covered too; one
limit-free container returns the whole pod to the victim set.

Two DaemonSets are not chart-rendered and needed their own seam:
linstor-satellite via the LinstorSatelliteConfiguration podTemplate, and
virt-handler via KubeVirt customizeComponents.patches, since virt-operator
owns that DaemonSet. Note linstor-controller stays deliberately limit-free
-- it is a JVM that must not be OOM-killed mid-operation, and cluster_test
pins that.

Two things the multus manifest needed beyond the values change.
customize-deployment.patch is the source for that template, so the
limits are carried there rather than only in the generated file, which
`make update` would discard. And the reason kube-multus had no memory
limit was that this patch DELETED upstream's 50Mi rather than raising
it -- 50Mi being well under the 230Mi measured peak -- which is what
removed memory.max from the pod cgroup in the first place.

Not covered, because upstream hardcodes the containers with no resources
knob: the four frr-k8s cp-* init containers, the kube-ovn hostpath-init
and install-cni init containers, and cozy-proxy, whose vendored chart
exposes no resources value at all. hami and kilo are left alone as
optional components with no usage measurements behind them. All of these
still get a limit from the default LimitRange the operator maintains in
every system namespace.

Every limit is pinned by a helm-unittest assertion, and every assertion
was mutation-tested: changing the rendered value turns its suite red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-13 16:47:52 +05:00

447 lines
22 KiB
YAML

suite: cozy-multus daemonset invariants
release:
name: multus
namespace: cozy-multus
tests:
# The daemonset image tag tracks the cozystack release version and is
# re-pinned by release-prep, so pin the repository (cozystack-built, not the
# upstream k8snetworkplumbingwg image) rather than the volatile tag.
- it: runs the cozystack-built multus image
template: templates/multus-daemonset-thick.yml
documentSelector:
path: kind
value: DaemonSet
asserts:
- matchRegex:
path: spec.template.spec.containers[0].image
pattern: '^ghcr\.io/cozystack/cozystack/multus-cni:'
# multus runs on every node and had a CPU limit but no memory one, which
# leaves the pod cgroup without memory.max. The Talos userspace OOM handler
# (v1.12+) only ever selects cgroups that lack it, and picks its victim by QoS
# class rather than by what caused the pressure, so multus was an eviction
# candidate on every node however healthy it was itself.
#
# memory.max lands on the POD cgroup only when EVERY container carries a
# limit. install-cni-plugins is asserted further down, where the rest of that
# container's contract lives; the two below complete the set for a default
# render. Keep all three: dropping any one silently returns the whole pod to
# the victim set while the other assertions still pass.
- it: kube-multus and its binary installer declare memory limits
template: templates/multus-daemonset-thick.yml
documentSelector:
path: kind
value: DaemonSet
asserts:
- equal:
path: spec.template.spec.containers[?(@.name=="kube-multus")].resources.limits.memory
value: 1Gi
- equal:
path: spec.template.spec.containers[?(@.name=="kube-multus")].resources.requests.memory
value: 100Mi
- equal:
path: spec.template.spec.initContainers[?(@.name=="install-multus-binary")].resources.limits.memory
value: 128Mi
# Guards the cozystack CNI-ordering fix (multusMasterCNI pinned to the cilium
# conflist) so multus does not auto-detect kube-ovn ahead of cilium at node
# startup and bypass the cilium CNI chain. This is a hand-edit that lives in
# patches/customize-deployment.patch so it survives every make update.
- it: pins the master CNI to the cilium conflist
template: templates/multus-daemonset-thick.yml
documentSelector:
path: kind
value: ConfigMap
asserts:
- matchRegex:
path: data["daemon-config.json"]
# Anchored to the top level of the JSON object, whose keys the template
# writes at four spaces: the same text nested inside another key
# satisfies an unanchored match while the effective value points
# somewhere else. This pins the position the template produces, which
# is as far as a pattern can go -- deciding which key is effective
# needs a JSON parser, and helm-unittest has none.
pattern: '(?m)^ "multusMasterCNI": "05-cilium\.conflist"'
# A quoted "false" is a non-empty string; a truthiness test would read it as
# on. The value is compared as text.
- it: treats a quoted "false" as off, not as a non-empty string
set:
stageCniPlugins: "false"
template: templates/multus-daemonset-thick.yml
documentSelector:
path: kind
value: DaemonSet
asserts:
- lengthEqual:
path: spec.template.spec.initContainers
count: 1
- equal:
path: spec.template.spec.initContainers[0].name
value: install-multus-binary
- it: stages when the value arrives as a quoted "true"
set:
stageCniPlugins: "true"
template: templates/multus-daemonset-thick.yml
documentSelector:
path: kind
value: DaemonSet
asserts:
- equal:
path: spec.template.spec.initContainers[1].name
value: install-cni-plugins
# The chart is reachable directly (helm install, the cozypkg dev loop), so it
# holds the same line as the platform helper: an unreadable spelling stops the
# render instead of resolving to off.
- it: refuses a value that is neither true nor false
set:
stageCniPlugins: maybe
asserts:
- failedTemplate:
errorMessage: 'stageCniPlugins: expected a boolean, got "maybe"'
# An explicit null is how a values file clears the key, and the platform helper
# folds it back onto its default rather than treating it as a spelling. The
# chart does the same with ITS default, off -- without this the render fails
# naming "<nil>", a spelling the operator never typed.
- it: reads an explicit null as the chart default rather than refusing it
set:
stageCniPlugins: null
template: templates/multus-daemonset-thick.yml
documentSelector:
path: kind
value: DaemonSet
asserts:
- lengthEqual:
path: spec.template.spec.initContainers
count: 1
- equal:
path: spec.template.spec.initContainers[0].name
value: install-multus-binary
- it: does not stage anything unless asked to
template: templates/multus-daemonset-thick.yml
documentSelector:
path: kind
value: DaemonSet
asserts:
- equal:
path: spec.template.spec.initContainers[0].name
value: install-multus-binary
- lengthEqual:
path: spec.template.spec.initContainers
count: 1
# The staging init container installs the plugins bundled in the image into
# /opt/cni/bin. Distros like k3s do not populate it with them. The container
# lives in patches/customize-deployment.patch so it survives every make update.
- it: stages the reference CNI plugins into the host cni bin dir
template: templates/multus-daemonset-thick.yml
set:
stageCniPlugins: true
documentSelector:
path: kind
value: DaemonSet
asserts:
- equal:
path: spec.template.spec.initContainers[1].name
value: install-cni-plugins
# What runs the script, not only what it says. The behavioural suite
# executes command[2] under sh and never reads command[0] and [1], so a
# container handed the same text to `echo` would print it, exit 0 and
# stage nothing while every suite stays green.
- equal:
path: spec.template.spec.initContainers[1].command[0]
value: sh
- equal:
path: spec.template.spec.initContainers[1].command[1]
value: -c
# restartPolicy on an init container makes it a native sidecar: the
# kubelet restarts it when it exits, so the staging loop would rewrite
# /opt/cni/bin continuously instead of once per pod recreation, which is
# what the bundle comment and the commit message both claim it does.
- notExists:
path: spec.template.spec.initContainers[1].restartPolicy
# args are appended after command[2], where they become the positional
# parameters of the script rather than more script.
- notExists:
path: spec.template.spec.initContainers[1].args
# The volume NAME, not only the mountPath: which hostPath sits behind it is
# decided here. Swapping cnibin for hostroot, mounted at /, keeps every
# other assertion true while the plugins land elsewhere.
- equal:
path: spec.template.spec.initContainers[1].volumeMounts[0].name
value: cnibin
- equal:
path: spec.template.spec.initContainers[1].volumeMounts[0].mountPath
value: /host/opt/cni/bin
# subPath moves what the mountPath actually resolves to: with
# `subPath: staging` the container writes into <hostPath>/staging and
# every step still succeeds, so the script reports a clean install while
# the live plugin directory is untouched. subPathExpr is the same field
# with variable expansion.
- notExists:
path: spec.template.spec.initContainers[1].volumeMounts[0].subPath
- notExists:
path: spec.template.spec.initContainers[1].volumeMounts[0].subPathExpr
# And exactly one mount: asserting [0] says nothing about a second one.
# A mount at /cni-plugins masks the image payload, so the loop would
# install whatever that volume holds instead of the fourteen plugins.
- lengthEqual:
path: spec.template.spec.initContainers[1].volumeMounts
count: 1
# The script writes its report to the literal /dev/termination-log, and
# this field decides where the kubelet reads it from. Point it elsewhere
# and `kubectl describe` shows nothing while the container still exits 0.
- notExists:
path: spec.template.spec.initContainers[1].terminationMessagePath
# FallbackToLogsOnError reads the container log into the status message
# only when the container exits with an ERROR and the termination file is
# empty (k8s.io/api core/v1: TerminationMessageFallbackToLogsOnError).
# This container exits 0 by design, so on the ordinary path the file is
# what `kubectl describe` shows and the policy changes nothing -- a failed
# write leaves the message empty either way, and only the container log
# keeps the report. It is pinned for the other path: if the script ever
# dies before reaching `exit 0`, the log is then the only place the reason
# exists, and `File` would report nothing at all.
- equal:
path: spec.template.spec.initContainers[1].terminationMessagePolicy
value: FallbackToLogsOnError
# The LSM side that is not SELinux, and the one place this container asks
# for nothing. Leaving the field unset means containerd's default profile
# applies, which permits ordinary file writes, so staging works under it.
# Unset is therefore the state to hold: setting it is how the container
# would gain LSM privilege it does not need, and cilium's nsenter escape
# hatch in packages/system/cilium/values-apparmor.yaml shows what asking
# looks like. Should a future default profile ever refuse these writes,
# the remedy is `appArmorProfile: {type: Unconfined}` -- and changing this
# assertion is then a deliberate act rather than an accident.
- notExists:
path: spec.template.spec.initContainers[1].securityContext.appArmorProfile
- notExists:
path: spec.template.spec.securityContext.appArmorProfile
# A read-only mount is refused by the kernel on every write, so the script
# reports each plugin as failed, exits 0 by design, and the pod goes Ready
# having staged nothing.
- notEqual:
path: spec.template.spec.initContainers[1].volumeMounts[0].readOnly
value: true
# The volume itself, name and hostPath together: the mount names cnibin,
# but which host directory sits behind that name is decided in volumes,
# and nothing above reads it. Asserted as a pair rather than by index.
- contains:
path: spec.template.spec.volumes
content:
name: cnibin
hostPath:
path: /opt/cni/bin
# mountPropagation governs mounts made INSIDE the container; this script
# makes none, and its only other effect is to force privileged.
- notExists:
path: spec.template.spec.initContainers[1].volumeMounts[0].mountPropagation
- notExists:
path: spec.template.spec.initContainers[1].securityContext.privileged
- equal:
path: spec.template.spec.initContainers[1].securityContext.allowPrivilegeEscalation
value: false
- contains:
path: spec.template.spec.initContainers[1].securityContext.capabilities.drop
content: ALL
# `drop: [ALL]` alone is not "every capability dropped": `add` is applied
# after it, so a later NET_ADMIN would be granted with this assertion
# still green.
- notExists:
path: spec.template.spec.initContainers[1].securityContext.capabilities.add
# On an enforcing node a container with every capability dropped cannot
# write a host-labelled /opt/cni/bin without a domain that allows it.
- equal:
path: spec.template.spec.initContainers[1].securityContext.seLinuxOptions.type
value: spc_t
# The level is part of the label, not decoration: an invalid or
# incompatible one stops the kubelet from creating the container at all,
# before the script has a chance to run.
- equal:
path: spec.template.spec.initContainers[1].securityContext.seLinuxOptions.level
value: s0
# env is the other half of "what runs": the command is pinned above, but a
# PATH pointing at nothing makes sh itself unresolvable, and the container
# then stages nothing while every assertion here still holds.
- notExists:
path: spec.template.spec.initContainers[1].env
# envFrom reaches the same variables by another route, and a referenced
# ConfigMap that is missing and not optional stops the container from
# starting at all.
- notExists:
path: spec.template.spec.initContainers[1].envFrom
# workingDir is resolved before the command runs: point it at a directory
# that does not exist and the container never starts, so the same 'nothing
# was staged' outcome arrives as a crashloop instead of a clean exit.
- notExists:
path: spec.template.spec.initContainers[1].workingDir
# What this costs on every node. Multus is a DaemonSet, so an init
# container's requests are charged once per node in the fleet, and an
# init container's requests also set the floor for the whole pod while it
# runs. Pinned so a change is a decision rather than a drift.
- equal:
path: spec.template.spec.initContainers[1].resources.requests.cpu
value: 10m
- equal:
path: spec.template.spec.initContainers[1].resources.requests.memory
value: 15Mi
# And a memory limit, which this assertion used to deny. The reasoning it
# carried -- nothing here is worth capping, it copies fourteen files and
# exits -- is about capping runtime, and that is not what the limit is
# for. A limit is what puts memory.max on the POD cgroup, and the Talos
# userspace OOM handler (v1.12+) only ever selects cgroups that lack it.
# The value is per-pod, not per-container: one limit-free container here
# leaves the whole multus pod in the victim set, and the system bundle
# defaults stageCniPlugins to true, so that is the default deployment.
#
# The original worry stands on its own terms and is answered by the size,
# not by omitting the limit: 128Mi is what would have to be low enough to
# kill a `cp` of fourteen small binaries. It matches install-multus-binary
# above, which does the same work, and page cache charged to the cgroup is
# reclaimable under pressure rather than a cause of OOM.
- equal:
path: spec.template.spec.initContainers[1].resources.limits.memory
value: 128Mi
# Writing into the hostPath needs uid 0 IN THE CONTAINER. Nothing sets a
# user today, so it runs as root; a runAsUser or runAsNonRoot at either
# level turns every install into a permission error, and the script then
# reports them, exits 0 by design and leaves the pod Ready with nothing
# staged. hack/multus-install-cni-plugins.bats cannot see this: it runs
# the extracted script as whoever runs the suite, not through the kubelet.
# The pod level matters as much as the container level -- `make update`
# re-fetches this manifest from upstream, and a pod-level hardening added
# there would be inherited without touching the patch.
- notExists:
path: spec.template.spec.initContainers[1].securityContext.runAsUser
- notExists:
path: spec.template.spec.initContainers[1].securityContext.runAsNonRoot
- notExists:
path: spec.template.spec.securityContext.runAsUser
- notExists:
path: spec.template.spec.securityContext.runAsNonRoot
# kube-ovn installs portmap and macvlan into the same /opt/cni/bin with a
# plain cp, so plugins are replaced while the kubelet may be executing them.
# A rename is atomic within the mount; writing onto the live path is not.
- it: installs each plugin atomically rather than writing onto the live path
template: templates/multus-daemonset-thick.yml
set:
stageCniPlugins: true
documentSelector:
path: kind
value: DaemonSet
asserts:
# Index [1] is the right container only while it is still named this, and a
# negative assert against the wrong container passes without checking.
- equal:
path: spec.template.spec.initContainers[1].name
value: install-cni-plugins
# The copy must target the temp name...
- matchRegex:
path: spec.template.spec.initContainers[1].command[2]
pattern: '(?m)^\s*if cp -f "\$plugin" "\$tmp"'
# ...and only the rename may publish the final name.
- matchRegex:
path: spec.template.spec.initContainers[1].command[2]
pattern: '(?m)^\s*mv -f "\$tmp" "/host/opt/cni/bin/\$\{name\}"'
# Catches the literal spellings of the regression. A copy reaching the same
# path through a variable has no literal to match, so
# hack/multus-install-cni-plugins.bats owns the real guarantee by asserting
# the replaced inode changes. (?m) is load-bearing: Go anchors $ to end of
# text, not end of line.
- notMatchRegex:
path: spec.template.spec.initContainers[1].command[2]
pattern: '(?m)^[^#\n]*\b(cp|install)\b[^\n]*/host/opt/cni/bin(/|"|\s|$)'
# Unlinking the live name before the rename is not a replacement by
# rename: it opens a window where the path does not exist at all, and a
# failure after it destroys the copy that was working. The inode still
# changes, so the runtime check in hack/multus-install-cni-plugins.bats
# cannot tell the two apart. The trap's glob is a different path.
- notMatchRegex:
path: spec.template.spec.initContainers[1].command[2]
pattern: '(?m)^[^#\n]*\brm\b[^\n]*/host/opt/cni/bin/"?\$\{?name\}?'
# release-prep bakes the digest, so between a Dockerfile change and the next
# bake the pinned image can predate the staging and have no /cni-plugins.
# A successfully built image always has the directory -- the Dockerfile's
# tar fails otherwise -- so the guard can only ever mask that skew.
- it: skips staging instead of crashlooping when the image predates the plugins
template: templates/multus-daemonset-thick.yml
set:
stageCniPlugins: true
documentSelector:
path: kind
value: DaemonSet
asserts:
- equal:
path: spec.template.spec.initContainers[1].name
value: install-cni-plugins
# Guard and body as one pattern: an unanchored `exit 0` is satisfied by any
# exit in the script. The trailing anchor matters too -- `exit 01` exits 1
# and contains `exit 0`.
- matchRegex:
path: spec.template.spec.initContainers[1].command[2]
pattern: '(?m)^\s*if \[ ! -d /cni-plugins \]; then\n[^\n]*\n[^\n]*exit 0\s*$'
# Deleting any of the next four leaves every other assertion green -- found
# by deleting each executable line of the script in turn and seeing which
# deletions nothing noticed.
#
# The shell options, as the first command: -u turns a mistyped variable
# into an error instead of an empty string, which here would mean cp'ing
# to "" or removing the wrong path.
- matchRegex:
path: spec.template.spec.initContainers[1].command[2]
pattern: '(?m)\A\s*set -eu\s*$'
# The two operator-facing lines of the failure report. The summary and the
# plugin name are pinned by hack/multus-install-cni-plugins.bats; these two
# carry what remains and where to look, and nothing held them.
- matchRegex:
path: spec.template.spec.initContainers[1].command[2]
pattern: '(?m)^\s*echo "those were left as they were" >&2\s*$'
- matchRegex:
path: spec.template.spec.initContainers[1].command[2]
pattern: '(?m)^\s*echo "the cause for each is earlier in this log" >&2\s*$'
# The explicit exit, as the last command. Without it the container's status
# is whatever the preceding command returned; that is 0 today by accident
# of what stands last, and the design does not want to depend on it.
- matchRegex:
path: spec.template.spec.initContainers[1].command[2]
pattern: '(?m)^\s*exit 0\s*\z'
# A kill between the copy and the rename strands a temp file in a directory
# shared with cilium and kube-ovn, as multus's own CopyFileAtomic handles.
# hack/multus-install-cni-plugins.bats covers the behaviour; this pins the
# trap into the rendered manifest.
- it: does not leak temp files into the shared cni bin dir
template: templates/multus-daemonset-thick.yml
set:
stageCniPlugins: true
documentSelector:
path: kind
value: DaemonSet
asserts:
- equal:
path: spec.template.spec.initContainers[1].name
value: install-cni-plugins
- matchRegex:
path: spec.template.spec.initContainers[1].command[2]
pattern: "(?m)^\\s*trap 'rm -rf /host/opt/cni/bin/\\.tmp-cozystack-multus-\\* \\|\\| :' EXIT\\s*$"
# The plugins live only in the cozystack-built image. Regenerated with the
# upstream one, /cni-plugins is absent and the script SKIPS, staging
# nothing on any node.
- matchRegex:
path: spec.template.spec.initContainers[1].image
pattern: '^ghcr\.io/cozystack/cozystack/multus-cni:'
# The script is an argv string in the file the package's `image:` target
# rewrites with `sed s|image: .*multus-cni.*|...|g`, so a line inside it
# carrying both tokens would be replaced by an image reference.
- notMatchRegex:
path: spec.template.spec.initContainers[1].command[2]
pattern: 'image:'