feat(multus)!: stage reference CNI plugins from the multus image

Multus-backed NetworkAttachmentDefinitions delegate to the upstream
containernetworking reference plugins (bridge, macvlan, ipvlan, ...),
and nothing guarantees a node has them in the directory Multus
searches.  On k3s they live under /var/lib/rancher/k3s/data/cni while
Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails
there with `failed to find plugin "bridge" in path [/opt/cni/bin]`.

Bundle 14 of the release's plugins into the multus image and add an
install-cni-plugins init container that installs them into the host
/opt/cni/bin, mirroring the existing install-multus-binary container.
loopback, dhcp, tap and dummy are left out; the Dockerfile records why.

Gate it on a new chart value, stageCniPlugins, resolved per bundle and
on for both.  Talos bundles only bridge, firewall, flannel, host-local,
loopback and portmap, so the other ten are missing there exactly as on
generic Linux; the four that overlap are rewritten with the same
upstream version, v1.9.1 on both sides, rather than an older one.  A
marker in the image Dockerfile records the Talos release that pin was
compared against, and a test holds it to the release the platform
ships.  multus therefore moves out of system.common-packages into the
two variant branches, the way linstor already does for talos.enabled.
networking.stageCniPlugins overrides the variant default in either
direction.

Where staging is on the install is unconditional: a plugin is replaced
on every pod recreation, not only when the destination is missing.  A
destination that is already a directory is reported and left alone,
since replacing it is not something this can do safely.  Turning the
value off afterwards does not undo it; nothing here removes or
restores a plugin.

The container is not privileged and takes no mount propagation.  It
runs in the spc_t SELinux domain, without which an enforcing node
refuses the write.  Each plugin is installed under a temporary name and
swapped in with a rename: /opt/cni/bin is shared with cilium and
kube-ovn.  The plugin tarball is checksummed before extraction.

A plugin that cannot be installed is reported on stderr and to
/dev/termination-log, and the container still exits 0: Multus stays the
node's primary CNI and nothing removes its config, so an init container
that never completes would leave the shim with no daemon behind it.

The manifest change lives in customize-deployment.patch so it survives
make update.

BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI
plugins in /opt/cni/bin on existing nodes, Talos included -- four of the
names it writes are ones the Talos node image also carries.  Opt out
before upgrading: declining afterwards does not restore what was
replaced.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

docs/agents/image-refs.md

@@ -32,9 +32,9 @@ A reference can be stored in three different kinds of file, and **all three are
 
 1. **`packages/<group>/<pkg>/values.yaml`** — the common case, in five YAML sub-shapes (single string, split map with `digest`, split map with the digest inside `tag`, chart-global `global.registry.address` + `global.images[]`, and the OCI-artifact `platformSourceUrl`/`platformSourceRef` pair). The sub-shapes exist because vendored upstream charts choose their own values layout; the list is empirical, not closed.
 2. **`packages/<group>/<pkg>/images/<name>.tag`** — a plain file holding one reference, read by templates via `.Files.Get` rather than through values. Easy to forget precisely because it never appears in a values file; otherwise an ordinary first-party reference that carries the cozystack version like any other.
-3. **A declared file that embeds a reference in surrounding text** — used when a package vendors an upstream manifest verbatim and its `image:` target `sed`s the built reference straight into it. `packages/system/multus/templates/multus-daemonset-thick.yml` is the only instance today.
+3. **A declared file that embeds a reference in surrounding text** — used when a package vendors an upstream manifest and its `image:` target `sed`s the built reference straight into it. `packages/system/multus/templates/multus-daemonset-thick.yml` is the only instance today. It is vendored *and* patched: it carries Helm conditionals, so `yq` cannot parse it at all and the YAML pass yields nothing for it. What keeps its reference visible is the textual scrape `collect_image_refs` runs over every declared extra — for this entry that leg is not a backstop but the only one that works.
 
-`hack/lib/image-refs.sh` is the single enumeration of all three, and the only place that knowledge is allowed to live. `image_ref_files <root>` yields the files; `collect_image_refs <root>` yields the references inside them. Shape 3 is an explicit list (`IMAGE_REF_EXTRA_FILES`) rather than a `templates/**` glob, because such a glob would sweep in vendored upstream chart templates and Helm-templated `image:` lines where a blind rewrite corrupts a value that `make update` regenerates.
+`hack/lib/image-refs.sh` is the single enumeration of all three, and the only place that knowledge is allowed to live. `image_ref_files <root>` yields the files; `collect_image_refs <root>` yields the references inside them. Shape 3 is an explicit list (`IMAGE_REF_EXTRA_FILES`) rather than a `templates/**` glob, because such a glob would sweep in vendored upstream chart templates and Helm-templated `image:` lines where a blind rewrite corrupts a value that `make update` regenerates. The declared entry is safe from that despite now containing Helm conditionals: its `image:` lines are plain, and declaring a file is the deliberate statement that its references are stamped rather than templated.
 
 ### Adding a reference in a new location
 

docs/vm-external-vlan.md

@@ -23,9 +23,40 @@ Three pieces cooperate:
 ## Prerequisites
 
 - The `multus` package is enabled (it provides the `NetworkAttachmentDefinition` CRD and the secondary-network plumbing).
-- The `bridge` CNI plugin is present in `/opt/cni/bin` on every node. This binary comes from the node's CNI-plugins bundle, not from a Cozystack chart: on Talos it ships with the node image, while k3s keeps its copy under `/var/lib/rancher/k3s/data/current/bin` and leaves `/opt/cni/bin` (where Multus and Cilium install) without it. Verify with `ls /opt/cni/bin/bridge`; a missing binary makes the NAD fail with `failed to find plugin "bridge" in path [/opt/cni/bin]`. If it is absent, stage the upstream reference plugins (`bridge`, `macvlan`, `ipvlan`, …) into `/opt/cni/bin` as part of node provisioning.
+- The `bridge` CNI plugin is present in `/opt/cni/bin` on every node. The `multus` package puts it there itself on every platform, which is the subject of the next section — read it before upgrading a cluster whose `/opt/cni/bin` you provision yourself. Verify with `ls /opt/cni/bin/bridge`; a missing binary makes the NAD fail with `failed to find plugin "bridge" in path [/opt/cni/bin]`.
 - There is no IPAM plugin in this path — addresses are assigned inside the guest, not by the CNI. Plan static addresses per VM.
 
+## Staging the reference CNI plugins
+
+The `multus` package installs the reference plugins into `/opt/cni/bin` itself, on every platform, because no platform supplies all of them. That is the `stageCniPlugins` chart value, and the platform turns it on for every bundle. What differs is only how much is missing before staging runs. Talos puts six binaries in that directory — `bridge`, `firewall`, `host-local`, `loopback` and `portmap` from the same upstream release, plus `flannel`, which is a different project — so a NAD naming `macvlan`, `ipvlan` or `vlan` fails there too; staging adds the ten it does not carry and rewrites the four that overlap with the same upstream version (`v1.9.1` on both sides), not an older one. Generic Linux offers no guarantee at all — k3s keeps its own copies under `/var/lib/rancher/k3s/data/cni` (`/var/lib/rancher/k3s/data/current/bin` on releases older than October 2024) and Cilium installs only `cilium-cni` and `loopback`, so `/opt/cni/bin` there can lack even `bridge`.
+
+Carrying the plugins costs roughly 67 MiB uncompressed per architecture in the `multus` image, which every node pulls whether staging is on or off. Size airgap mirrors accordingly.
+
+Where staging is on, it tries to replace any file of the same name already in that directory, every time the Multus pod is recreated, so a hand-placed or locally patched `bridge` will normally not survive — carry such a change in the `multus` image instead. A plugin it fails to write is reported and skipped rather than removed, so whatever was there stays — an older copy where the node had one, and nothing at all where it did not. It installs 14 of the upstream reference plugins and leaves `loopback`, `dhcp`, `dummy` and `tap` alone. Leaving them alone is not a promise that something else provides them: `loopback` is installed by Cilium and kube-ovn, while `dhcp`, `dummy` and `tap` are simply not staged — `dhcp` needs a host daemon this package does not run, and the other two have no consumer here. If you need one of those, put it on the node yourself.
+
+**If you provision `/opt/cni/bin` yourself, set the opt-out before upgrading.** The replacement happens as soon as Multus rolls, and turning `stageCniPlugins` off afterwards does not undo it: nothing in this package removes or restores a plugin, so the copies stay until you re-provision the node. Operators pinned to a particular plugin version should decline first and upgrade second. This applies on Talos too, where four of the names staging writes are ones the node image also carries. Declining looks like this, on the `platform` component of the `cozystack.cozystack-platform` Package CR:
+
+```yaml
+apiVersion: cozystack.io/v1alpha1
+kind: Package
+metadata:
+  name: cozystack.cozystack-platform
+spec:
+  components:
+    platform:
+      values:
+        networking:
+          stageCniPlugins: false
+```
+
+Editing the multus HelmRelease or the child `Package cozystack.multus` directly does not hold: both are re-rendered by the platform reconcile. An unreadable value there — a spelling that is neither a `true` nor a `false` form — stops the whole platform render rather than resolving to off, so a typo in this key halts reconciliation cluster-wide.
+
+If `bridge` is missing on a node where staging is on, read the log with `kubectl --namespace cozy-multus logs <multus-pod> --container install-cni-plugins`. The line `no /cni-plugins in this image; skipping staging` means the running `multus` image predates the plugin staging and installed nothing: the image reference and the manifest are re-pinned on different schedules, so a tree taken between a Dockerfile change and the next release bake can carry one without the other. Move to a release whose `multus` image carries the plugins. A line beginning `failed to install:` names plugins the container could not write, one of which may be the `bridge` you are looking for. The reason for each is earlier in the same log, printed as that plugin was attempted rather than next to the summary — so with more than one failure the causes are spread above it, not adjacent to it, and the one naming your plugin is the one to read. The container reports these and exits successfully on purpose, so the pod shows Ready and `kubectl get` will not point you here: failing it instead would leave the node unable to start any pod that goes through CNI, since Multus remains the primary CNI with no daemon behind it — host-network pods bypass CNI and still start. That means a NAD failing with `failed to find plugin` is the symptom to trust, and this log is where the reason is.
+
+If the init container is absent from the Multus pod altogether, staging is off for this cluster. That means `networking.stageCniPlugins` was set to `false` somewhere: no bundle defaults it off, on Talos or anywhere else. Turn it back on with the same block as above and `true` in place of `false`, or stage the plugins through node provisioning instead.
+
+Changing this value edits the Multus DaemonSet's pod template, so Multus rolls node by node. While a node's `multus-daemon` is down its CNI shim has no daemon behind it, and pod creation and deletion on that node fail for the length of the restart. Flip it during a window where that is acceptable, not while something else is rolling.
+
 ## Step 1 — Linux bridge on the node
 
 Create a bridge that enslaves the tagged VLAN sub-interface. The VLAN sub-interface itself carries no address; the bridge carries the host's presence on that VLAN (optional, but useful for a gateway-reachability sanity path and for any host-side service the VMs must reach).

hack/cni-plugins-staging-contract.bats

@@ -0,0 +1,572 @@
+#!/usr/bin/env bats
+# Contract checks for the CNI plugin staging that a helm-unittest case cannot
+# make: they read files outside the chart, or render it more than one way.
+#
+# Run with: hack/cozytest.sh hack/cni-plugins-staging-contract.bats
+
+# Keep the closing brace on the same line: hack/cozytest.sh rewrites any line
+# that is exactly `}` into `return 0` plus `}`, without caring whether it closes
+# a test, a helper or a plain brace group at file scope.
+command -v yq >/dev/null || { echo "yq is required to read the vendored kube-ovn version" >&2; exit 1; }
+
+# Is $2 one of the plugin names in the extract text $1? Separators differ by
+# position in a continued shell line, so fold them to spaces and pad both ends
+# -- `./bridge` must not be satisfied by `./bridgeport`.
+extract_has() {
+  case " $(printf '%s' "$1" | tr '\n;\\' '   ') " in
+    *" ./$2 "*) return 0 ;;
+  esac
+  return 1
+}
+
+DOCKERFILE="$PWD/packages/system/multus/images/multus-cni/Dockerfile"
+KUBEOVN_VALUES="$PWD/packages/system/kubeovn/values.yaml"
+DOC="$PWD/docs/vm-external-vlan.md"
+PLATFORM_VALUES="$PWD/packages/core/platform/values.yaml"
+TEMPLATE="$PWD/packages/system/multus/templates/multus-daemonset-thick.yml"
+PATCHFILE="$PWD/packages/system/multus/patches/customize-deployment.patch"
+
+@test "membership in the extract list is decided the same way in every position" {
+  middle=' \ ./bridge ./host-local ./vrf; \ '
+  last=' \ ./bridge ./host-local ./loopback; \ '
+  first=' \ ./loopback ./bridge ./host-local; \ '
+  multiline=$(printf ' \\\n ./bridge ./host-local \\\n ./loopback; \\\n')
+
+  extract_has "$middle" bridge      || { echo "FAIL: missed a name in the middle" >&2; return 1; }
+  extract_has "$last" loopback      || { echo "FAIL: missed a name in last position" >&2; return 1; }
+  extract_has "$first" loopback     || { echo "FAIL: missed a name in first position" >&2; return 1; }
+  extract_has "$multiline" loopback || { echo "FAIL: missed a name across a line continuation" >&2; return 1; }
+
+  if extract_has ' \ ./bridgeport ./host-local; \ ' bridge; then
+    echo "FAIL: ./bridgeport satisfied a check for ./bridge" >&2
+    return 1
+  fi
+  if extract_has ' \ ./host-local ./bridgeport; \ ' bridge; then
+    echo "FAIL: ./bridgeport in last position satisfied a check for ./bridge" >&2
+    return 1
+  fi
+  if extract_has "$middle" loopback; then
+    echo "FAIL: reported a name that is not in the list" >&2
+    return 1
+  fi
+}
+
+@test "the Dockerfile stages the plugins and copies them into the final image" {
+  grep -qE '^[[:space:]]*tar -xz -C /cni-plugins -f ' "$DOCKERFILE"
+
+  # The COPY has to land in the LAST stage: one in an earlier stage satisfies
+  # the same pattern while the image that ships carries no plugins at all, and
+  # the init container then skips on every node.
+  final_stage=$(awk '/^FROM/{buf=""} {buf = buf $0 ORS} END{printf "%s", buf}' "$DOCKERFILE")
+  if [ -z "$final_stage" ]; then
+    echo "no FROM in the Dockerfile" >&2
+    return 1
+  fi
+  if ! printf '%s' "$final_stage" |
+       grep -qE '^COPY[[:space:]]+--from=build[[:space:]]+/cni-plugins[[:space:]]+/cni-plugins[[:space:]]*$'; then
+    echo "the final stage does not COPY /cni-plugins, so the plugins are built" >&2
+    echo "and then left behind in the build stage." >&2
+    grep -nE '^FROM|^COPY.*cni-plugins' "$DOCKERFILE" >&2
+    return 1
+  fi
+
+  # And nothing after it may take them out again: the COPY being present says
+  # nothing about the directory still existing when the build ends.
+  after_copy=$(printf '%s' "$final_stage" |
+    sed -n '/^COPY[[:space:]]\{1,\}--from=build[[:space:]]\{1,\}\/cni-plugins[[:space:]]/,$p')
+  if printf '%s' "$after_copy" | grep -qE '^[^#]*\brm\b.*/cni-plugins'; then
+    echo "something in the final stage removes /cni-plugins after it was" >&2
+    echo "copied in, so the image ships without the plugins and the init" >&2
+    echo "container skips on every node." >&2
+    printf '%s' "$after_copy" | grep -nE '^[^#]*\brm\b.*/cni-plugins' >&2
+    return 1
+  fi
+
+  # And nothing anywhere in the file may delete a staged plugin: the check above
+  # only reads the tar arguments, so a later `rm -f /cni-plugins/bridge` in the
+  # build stage ships an image with thirteen. The tarball itself is a different
+  # path (/tmp/cni-plugins.tgz) and removing it is expected.
+  if grep -qE '\brm\b[^#]*[^.[:alnum:]]/cni-plugins(/|[[:space:]]|;|$)' "$DOCKERFILE"; then
+    echo "the Dockerfile removes something from /cni-plugins; whatever is" >&2
+    echo "taken out there is missing from the image, and the init container" >&2
+    echo "installs only what remains." >&2
+    grep -nE '\brm\b[^#]*/cni-plugins' "$DOCKERFILE" >&2
+    return 1
+  fi
+
+  # The pinned digest must still BE the pinned digest when the comparison runs.
+  # Assigning ${sha256} from the downloaded file turns the verification into a
+  # comparison of the download with itself, which any well-formed archive passes.
+  assigns=$(grep -cE '^[[:space:]]*[^#]*\bsha256=' "$DOCKERFILE")
+  pinned=$(grep -cE '^[[:space:]]+[a-z0-9]+\)[[:space:]]+sha256="[0-9a-f]{64}"' "$DOCKERFILE")
+  if [ "$assigns" != "$pinned" ]; then
+    echo "sha256 is assigned $assigns times but only $pinned of those are the" >&2
+    echo "per-architecture pins. Anything else reassigns the digest before the" >&2
+    echo "comparison, which then checks the download against itself." >&2
+    grep -nE 'sha256=' "$DOCKERFILE" >&2
+    return 1
+  fi
+
+  # The uid the init container runs as is decided in three places: the pod
+  # securityContext, the container securityContext and the image's own USER.
+  # The first two are asserted in packages/system/multus/tests/multus_test.yaml;
+  # this is the third. A non-root USER in the final stage makes every write fail
+  # while both Kubernetes levels still read as unset, and the script then
+  # reports the failures, exits 0 and leaves the pod Ready with nothing staged.
+  final_user=$(printf '%s' "$final_stage" | grep -E '^USER[[:space:]]' || true)
+  case "$final_user" in
+    "") ;;
+    *"USER root"*|*"USER 0"*) ;;
+    *)
+      echo "the final stage sets a non-root USER:" >&2
+      printf '%s\n' "$final_user" >&2
+      echo "the init container writes into a hostPath and needs uid 0." >&2
+      return 1 ;;
+  esac
+
+  # Only the plugins may land in /cni-plugins. The init container installs every
+  # file it finds there and chmods it 0755, so anything else that arrives -- a
+  # licence, a README, a stray COPY -- is published into the host CNI directory
+  # as an executable. Measured: adding the licence to that directory makes the
+  # script install 15 files with LICENSE among them.
+  strays=$(grep -nE '^(COPY|ADD)[[:space:]].*[[:space:]]/cni-plugins(/|[[:space:]]*$)' "$DOCKERFILE" |
+    grep -v 'COPY --from=build /cni-plugins /cni-plugins' || true)
+  if [ -n "$strays" ]; then
+    echo "something other than the plugin directory itself is copied into" >&2
+    echo "/cni-plugins; the init container would install it on every node as" >&2
+    echo "an executable CNI plugin." >&2
+    printf '%s\n' "$strays" >&2
+    return 1
+  fi
+
+  # Apache-2.0 binaries are redistributed here, so the licence ships with them.
+  # Terminated on its own line for the same reason as the checksum: a command
+  # chained after the `;` runs inside the same RUN and can undo it.
+  grep -qE '^[[:space:]]*tar -xz -C /cni-plugins-licenses -f [^[:space:]]+ \./LICENSE;[[:space:]]*\\?[[:space:]]*$' "$DOCKERFILE"
+  printf '%s' "$final_stage" |
+    grep -qE '^COPY[[:space:]]+--from=build[[:space:]]+/cni-plugins-licenses[[:space:]]+/cni-plugins-licenses[[:space:]]*$'
+
+  # Matched inside the extract's own continued line, so a mention elsewhere in
+  # the file cannot satisfy it.
+  extract=$(sed -n '/tar -xz -C \/cni-plugins -f /,/;/p' "$DOCKERFILE" | tr '\n' ' ')
+  [ -n "$extract" ]
+  for plugin in loopback dhcp dummy tap; do
+    if extract_has "$extract" "$plugin"; then
+        echo "packages/system/multus/images/multus-cni/Dockerfile now extracts" >&2
+        echo "./$plugin, which this package deliberately does not stage, and" >&2
+        echo "which docs/vm-external-vlan.md names as left alone." >&2
+        if [ "$plugin" = loopback ]; then
+          echo "The init container would then replace the node's loopback plugin" >&2
+          echo "on every restart. The runtime calls loopback for every pod" >&2
+          echo "sandbox, not just those using a NAD, so a bad copy costs the" >&2
+          echo "whole node rather than one attachment -- and cilium and kube-ovn" >&2
+          echo "already install it, so staging adds a writer for no gain." >&2
+        elif [ "$plugin" = dhcp ]; then
+          echo "dhcp is not usable from the binary alone: it needs a" >&2
+          echo "long-running daemon and a socket on the host, which this" >&2
+          echo "package does not run." >&2
+        fi
+        echo "Update the guide in the same commit if this is deliberate." >&2
+        return 1
+    fi
+  done
+
+  # Nothing may take the failure away: tar returns non-zero when a requested
+  # member is absent from the archive, and `|| true` there ships an image
+  # missing that plugin with a successful build. The range ends at the first
+  # `;`, so a chain before it is inside what is checked.
+  if printf '%s' "$extract" | grep -qE '\|\||&&|\|'; then
+    echo "the plugin extraction is chained with another command, which takes" >&2
+    echo "its failure away: a release archive missing one requested member" >&2
+    echo "would then build a successful image without that plugin." >&2
+    printf '%s\n' "$extract" >&2
+    return 1
+  fi
+
+  # The staged set by NAME, not a count: a count is satisfied by one name
+  # appearing twice while another is missing.
+  staged=$(printf '%s' "$extract" | tr ' ' '\n' | sed -n 's|^\./||p' | tr -d ';' | sort)
+  expected=$(printf '%s\n' bandwidth bridge firewall host-device host-local ipvlan \
+    macvlan portmap ptp sbr static tuning vlan vrf | sort)
+  if [ "$staged" != "$expected" ]; then
+    echo "the set of plugins the Dockerfile stages has changed." >&2
+    echo "staged now:" >&2
+    printf '%s\n' "$staged" >&2
+    echo "expected these 14:" >&2
+    printf '%s\n' "$expected" >&2
+    echo "docs/vm-external-vlan.md tells operators the package installs 14 of" >&2
+    echo "the upstream reference plugins and names the four it leaves alone," >&2
+    echo "so that sentence is now wrong. Update it in the same commit." >&2
+    return 1
+  fi
+
+  for plugin in bridge macvlan ipvlan; do
+    if ! extract_has "$extract" "$plugin"; then
+        echo "packages/system/multus/images/multus-cni/Dockerfile no longer extracts" >&2
+        echo "./$plugin into /cni-plugins, but docs/vm-external-vlan.md tells operators" >&2
+        echo "the package stages it. The init container installs whatever is in that" >&2
+        echo "directory and skips silently when it is absent, so a NAD naming" >&2
+        echo "$plugin would go back to failing with" >&2
+        echo "\"failed to find plugin \\\"$plugin\\\" in path [/opt/cni/bin]\"" >&2
+        echo "on every node, with nothing else in this suite going red." >&2
+        return 1
+    fi
+  done
+}
+
+@test "the tarball checksum is present, pinned, and precedes extraction" {
+  # These are TEXT checks. PLATFORM defaults to empty in hack/common-envs.mk and
+  # nothing in this repository sets it, so a plain build produces the runner's
+  # arch as a single manifest; a caller passing PLATFORM gets an index, and both
+  # per-arch digests below are then live.
+
+  # One whole line, and it must TERMINATE. The `;` is what closes the family:
+  # `|| true`, `&& :`, `| cat` (a pipeline's status is its last element and there
+  # is no pipefail in POSIX sh), and `|| true` moved to the next physical line,
+  # since an unterminated command must end in `\`. `${sha256}` in the comparison
+  # is what stops the tarball being checked against itself.
+  #
+  # sha256sum accepts "HASH  FILE" and "HASH *FILE"; both are allowed here.
+  # The file operand is taken from the curl line rather than left open: an
+  # unpinned operand is satisfied by any path, so a decoy file with a pinned
+  # digest passes the form while the downloaded tarball is extracted unchecked.
+  tarball=$(grep -oE '[[:space:]]-o[[:space:]]+[^[:space:]]+' "$DOCKERFILE" | head -1 | awk '{print $2}')
+  if [ -z "$tarball" ]; then
+    echo "no 'curl ... -o <file>' in the Dockerfile, so there is no download" >&2
+    echo "for the checksum to be about." >&2
+    return 1
+  fi
+  tarball_re=$(printf '%s' "$tarball" | sed 's/[].[^$*\\/]/\\&/g')
+
+  if ! grep -qE "^[[:space:]]*echo \"\\\$\\{sha256\\}([[:space:]]+|[[:space:]]\\*)${tarball_re}\" \\| sha256sum -c -;[[:space:]]*\\\\?[[:space:]]*$" "$DOCKERFILE"; then
+    echo "the verify line is not the expected form:" >&2
+    echo "  echo \"\${sha256}  ${tarball}\" | sha256sum -c -;" >&2
+    echo "It must compare the pinned digest against the file curl downloaded," >&2
+    echo "and must terminate on its own line -- anything chained after it, on" >&2
+    echo "this line or the next, takes the failure away and an unverified" >&2
+    echo "tarball is extracted." >&2
+    grep -nE 'sha256sum' "$DOCKERFILE" >&2
+    return 1
+  fi
+
+  # And the file that was verified must be the one unpacked.
+  if ! grep -qE "^[[:space:]]*tar -xz -C /cni-plugins -f ${tarball_re}([[:space:]]|$)" "$DOCKERFILE"; then
+    echo "the extraction does not unpack ${tarball}, the file the checksum" >&2
+    echo "above verified, so the verification decides nothing." >&2
+    grep -nE 'tar -xz' "$DOCKERFILE" >&2
+    return 1
+  fi
+
+  # `\` joins the physical lines before the shell sees them, so `set +e` on any
+  # continuation disables the checksum, curl --fail and tar together while
+  # `RUN set -eux` stays intact.
+  verify_run=$(awk '
+    /^RUN /            { block = $0; inblock = 1; next }
+    inblock            { block = block "\n" $0 }
+    inblock && !/\\$/  { if (block ~ /sha256sum -c -/) { print block; exit }; inblock = 0 }
+  ' "$DOCKERFILE")
+  if [ -z "$verify_run" ]; then
+    echo "could not find the RUN that verifies the tarball" >&2
+    return 1
+  fi
+  # Errexit IS checked, on the logical line the shell actually runs. Four earlier
+  # attempts to decide this by grepping physical lines failed, each in its own
+  # way: `set -ux` (absent), `set +e` later (revoked), `# set -e` (a comment,
+  # which under a continuation also silences the rest of the joined line), and
+  # `echo 'set -e'` (an argument, not a command). Joining the continuations and
+  # cutting every comment first removes all four: what remains is the text the
+  # shell sees, and errexit has to be its FIRST command.
+  #
+  # Both spellings of revocation are rejected: `set +e` and `set +o errexit`.
+  #
+  # Not covered, and not claimed to be: errexit reached through a variable or a
+  # conditional. Deciding that needs a shell, not a pattern.
+  run_start=$(awk '/^RUN /{r=NR} /sha256sum -c -/{print r; exit}' "$DOCKERFILE")
+  if [ -z "$run_start" ]; then
+    echo "no RUN instruction contains the checksum comparison" >&2
+    return 1
+  fi
+  joined=$(tail -n +"$run_start" "$DOCKERFILE" |
+    awk '{print} !/\\[[:space:]]*$/{exit}' |
+    sed 's/#.*//' | tr '\n' ' ' | tr -s ' ')
+  # The same joined line, as the shell parses it. Deleting `case ... in` or its
+  # `esac` leaves a syntax error the image build would catch -- in minutes, and
+  # only on a pull request that touches this package. `sh -n` catches it here in
+  # milliseconds. Continuations are stripped first: with the backslashes left in
+  # place the shell sees them as literals and even a correct RUN fails to parse.
+  if ! printf '%s' "$(tail -n +"$run_start" "$DOCKERFILE" |
+        awk '{print} !/\\[[:space:]]*$/{exit}' |
+        sed -e 's/^RUN[[:space:]]*//' -e 's/[[:space:]]*\\$//' | tr '\n' ' ')" | sh -n 2>/dev/null; then
+    echo "the RUN that verifies and extracts the plugins does not parse as a" >&2
+    echo "shell command once its continuations are joined." >&2
+    return 1
+  fi
+
+  first=$(printf '%s' "$joined" | sed 's/^RUN[[:space:]]*//' | cut -d';' -f1)
+  if ! printf '%s' "$first" | grep -qE '^[[:space:]]*set[[:space:]]+-[a-zA-Z]*e'; then
+    echo "the RUN that verifies the tarball does not begin by enabling errexit." >&2
+    echo "Its first command is: $first" >&2
+    echo "Without it a failed checksum is reported and the build continues to" >&2
+    echo "extract the archive it just rejected." >&2
+    return 1
+  fi
+  if printf '%s' "$joined" | grep -qE 'set[[:space:]]+(\+[a-zA-Z]*e|\+o[[:space:]]+errexit)'; then
+    echo "errexit is revoked later in the same RUN, which puts the checksum" >&2
+    echo "back to being advisory." >&2
+    printf '%s\n' "$joined" >&2
+    return 1
+  fi
+
+  # Both per-arch pins, each separately: an alternation is satisfied by whichever
+  # one survived.
+  for arch in amd64 arm64; do
+    if ! grep -qE "^[[:space:]]+$arch\)[[:space:]]+sha256=\"[0-9a-f]{64}\"" "$DOCKERFILE"; then
+      echo "no pinned 64-hex SHA-256 for $arch in the Dockerfile." >&2
+      echo "That arch would be extracted on the strength of its tag alone, and" >&2
+      echo "for arm64 no build exercises the path that would notice." >&2
+      grep -nE 'sha256=' "$DOCKERFILE" >&2
+      return 1
+    fi
+  done
+
+  # Verification must precede extraction. Located by the strict form above, not
+  # by the substring: an inert `: "sha256sum -c -";` earlier in the file would
+  # satisfy an ordering check anchored on a mention.
+  v=$(grep -nE '^[[:space:]]*echo "\$\{sha256\}([[:space:]]+|[[:space:]]\*)[^"]+" \| sha256sum -c -;[[:space:]]*\\?[[:space:]]*$' "$DOCKERFILE" | head -1 | cut -d: -f1)
+  t=$(grep -nE '^[[:space:]]*tar -xz -C /cni-plugins -f ' "$DOCKERFILE" | head -1 | cut -d: -f1)
+  if [ -z "$v" ] || [ -z "$t" ] || [ "$v" -ge "$t" ]; then
+    echo "the checksum no longer precedes the extraction (verify line $v, tar line $t)" >&2
+    return 1
+  fi
+}
+
+@test "the kube-ovn release the CNI plugin pin was checked against is the one vendored" {
+  # Each side non-empty first: two missing keys compare equal.
+  checked=$(sed -n 's|^# kube-ovn-cni-plugins-checked-against: *\([^ ]*\) *$|\1|p' "$DOCKERFILE")
+  [ -n "$checked" ]
+
+  vendored=$(yq -r '.global.images.kubeovn.tag' "$KUBEOVN_VALUES" | sed 's|@.*||')
+  [ -n "$vendored" ]
+  [ "$vendored" != "null" ]
+
+  if [ "$checked" != "$vendored" ]; then
+    echo "the multus CNI plugin pin was checked against kube-ovn $checked," >&2
+    echo "but packages/system/kubeovn/values.yaml now vendors $vendored." >&2
+    echo >&2
+    echo "Read CNI_PLUGINS_VERSION in kube-ovn $vendored's base image" >&2
+    echo "(dist/images/Dockerfile.base). If it still matches the" >&2
+    echo "CNI_PLUGINS_VERSION in packages/system/multus/images/multus-cni/" >&2
+    echo "Dockerfile, only move the marker there to $vendored. If it does not," >&2
+    echo "the shared portmap and macvlan would flap between the two daemonsets," >&2
+    echo "so bring the multus pin and its per-arch checksums across first." >&2
+    return 1
+  fi
+}
+
+@test "the rendered multus daemon config is valid JSON" {
+  # Every assertion on this document elsewhere is a pattern over its text, and a
+  # pattern is happy with trailing garbage: `"multusMasterCNI": "05-cilium.conflist" garbage,`
+  # keeps them all green while multus gets a config it cannot parse and comes up
+  # with no master CNI at all. Parsing is the only check that decides this, and
+  # helm-unittest cannot do it.
+  #
+  # It sits here rather than beside the other daemon-config assertions because
+  # those are helm-unittest cases in packages/system/multus/tests/multus_test.yaml
+  # and this one needs a parser. This file is where the multus checks that need
+  # something outside helm-unittest live, staging or not.
+  cfg=$(helm template packages/system/multus |
+    yq eval 'select(.kind == "ConfigMap") | .data["daemon-config.json"]' -)
+  [ -n "$cfg" ]
+  if ! printf '%s' "$cfg" | yq eval -p json '.' - >/dev/null 2>&1; then
+    echo "the rendered daemon-config.json does not parse as JSON:" >&2
+    printf '%s\n' "$cfg" >&2
+    return 1
+  fi
+  # And the key that decides which conflist multus treats as primary has to
+  # survive parsing, not merely appear in the text.
+  master=$(printf '%s' "$cfg" | yq eval -p json '.multusMasterCNI' -)
+  if [ "$master" != "05-cilium.conflist" ]; then
+    echo "the parsed multusMasterCNI is '$master', not 05-cilium.conflist" >&2
+    return 1
+  fi
+}
+
+@test "the Talos release the CNI plugin pin was checked against is the one shipped" {
+  # Each side non-empty first: two missing keys compare equal.
+  checked=$(sed -n 's|^# talos-cni-plugins-checked-against: *\([^ ]*\) *$|\1|p' "$DOCKERFILE")
+  [ -n "$checked" ]
+
+  shipped=$(awk '/^version:/ {print $2; exit}' packages/core/talos/images/talos/profiles/installer.yaml)
+  [ -n "$shipped" ]
+
+  if [ "$checked" != "$shipped" ]; then
+    echo "the multus CNI plugin pin was checked against Talos $checked," >&2
+    echo "but packages/core/talos now ships $shipped." >&2
+    echo >&2
+    echo "Read cni_version in siderolabs/pkgs' Pkgfile on the matching release" >&2
+    echo "branch. If it still equals CNI_PLUGINS_VERSION in this Dockerfile," >&2
+    echo "only move the marker to $shipped. If it does not, staging would" >&2
+    echo "replace the node image's bridge, firewall, host-local and portmap" >&2
+    echo "with a different version of the same plugins on every Talos node," >&2
+    echo "so decide which version wins before moving it." >&2
+    return 1
+  fi
+}
+
+@test "the VLAN guide's remediation quotes what the daemonset actually emits" {
+  # The guide names a log line and a container; both spellings live in the
+  # manifest.
+  grep -qF -- 'no /cni-plugins in this image; skipping staging' "$TEMPLATE"
+  grep -qF -- 'no /cni-plugins in this image; skipping staging' "$DOC"
+
+  grep -qF -- 'name: install-cni-plugins' "$TEMPLATE"
+  # Inside the command an operator is told to run, not anywhere in the file: a
+  # mention left in prose ("older notes may say ...") satisfies a bare match
+  # while the command itself names a different container.
+  grep -qE -- 'kubectl [^`]*logs [^`]*--container install-cni-plugins($|[^-[:alnum:]])' "$DOC"
+
+  # The count as a reader meets it. Only the staged count is quoted, never the
+  # upstream total: that is a property of a release this suite cannot read.
+  grep -qE -- '(^|[^0-9])14 of the upstream reference plugins' "$DOC"
+
+  # The half of that sentence naming the four, pinned separately: of the four
+  # only loopback arrives from anywhere else (cilium and kube-ovn install it).
+  grep -qF -- 'and leaves `loopback`, `dhcp`, `dummy` and `tap` alone' "$DOC"
+  if grep -qF -- '`tap` to the node image' "$DOC"; then
+    echo "the guide again attributes the unstaged plugins to the node image;" >&2
+    echo "staging exists because that image may not carry them" >&2
+    return 1
+  fi
+
+  if grep -qF -- 'immediately above' "$DOC"; then
+    echo "the guide again promises the cause sits immediately above the" >&2
+    echo "'failed to install:' summary. It does not when more than one plugin" >&2
+    echo "fails: the summary is printed once, after the loop." >&2
+    return 1
+  fi
+
+  grep -qE -- '(^|[^0-9])14 of the upstream reference plugins' "$PLATFORM_VALUES"
+  grep -qF -- 'loopback, dhcp, dummy and tap are left alone' "$PLATFORM_VALUES"
+
+  # Presence only: a grep cannot decide whether prose still MEANS what it says,
+  # and `do not set the opt-out before upgrading` contains this string. The
+  # reversal is guarded by name, which closes the spellings seen so far and not
+  # the general case -- that one stays open deliberately.
+  grep -qE -- '(^|[^[:alnum:]])set the opt-out before upgrading' "$DOC"
+  if grep -qiE -- '(do not|never|no need to|dont|don.t) set the opt-out' "$DOC"; then
+    echo "the guide now tells operators NOT to set the opt-out before" >&2
+    echo "upgrading. Declining afterwards does not restore a replaced plugin," >&2
+    echo "so the instruction has to stay as it is." >&2
+    return 1
+  fi
+}
+
+@test "every accepted spelling of stageCniPlugins renders the direction it names" {
+  # The chart holds the accepted spellings as TWO lists -- one deciding whether
+  # the value is readable, one deciding whether it means on -- and nothing else
+  # makes them agree, so each spelling is rendered rather than their union.
+  #
+  for v in true TRUE True yes YES on On 1; do
+    n=$(helm template --set-string stageCniPlugins="$v" packages/system/multus |
+          yq eval 'select(.kind == "DaemonSet") | .spec.template.spec.initContainers | length' - |
+          tail -1)
+    if [ "$n" != "2" ]; then
+      echo "stageCniPlugins=$v is documented as ON but rendered $n init" >&2
+      echo "containers (expected 2: install-multus-binary + install-cni-plugins)." >&2
+      return 1
+    fi
+  done
+  for v in false FALSE False no NO off Off 0; do
+    n=$(helm template --set-string stageCniPlugins="$v" packages/system/multus |
+          yq eval 'select(.kind == "DaemonSet") | .spec.template.spec.initContainers | length' - |
+          tail -1)
+    if [ "$n" != "1" ]; then
+      echo "stageCniPlugins=$v is documented as OFF but rendered $n init" >&2
+      echo "containers (expected 1: install-multus-binary alone)." >&2
+      return 1
+    fi
+  done
+  # And a spelling outside the documented set must stop the render rather than
+  # resolving to either direction.
+  if helm template --set-string stageCniPlugins=maybe packages/system/multus >/dev/null 2>&1; then
+    echo "stageCniPlugins=maybe rendered instead of failing the template" >&2
+    return 1
+  fi
+}
+
+@test "every line the patch owns still matches the committed template" {
+  # customize-deployment.patch is the SOURCE: `make update` does
+  # `rm -rf templates && wget && patch`, so anything present only in the template
+  # is discarded on the next run. Reverse-applying the whole patch proves every
+  # line the PATCH OWNS still matches; it says nothing about the rest of the file.
+  #
+  # --fuzz=0 because `patch` defaults to a fuzz factor of 2 and will drop context
+  # lines to make a hunk fit.
+  d=$(mktemp -d)
+  mkdir -p "$d/templates"
+  sed 's|^\( *\)image: .*multus-cni.*|\1image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick|' \
+    "$TEMPLATE" > "$d/templates/multus-daemonset-thick.yml"
+
+  # Non-empty, or a moved template turns this into a patch against nothing.
+  if [ ! -s "$d/templates/multus-daemonset-thick.yml" ]; then
+    echo "the template is empty or missing; nothing to compare the patch to" >&2
+    rm -rf "$d"
+    return 1
+  fi
+
+  # --fuzz=0 for the same reason: a dropped context line is an edit absorbed
+  # rather than reported.
+  if ! (cd "$d" && patch -R --dry-run --fuzz=0 -p4 --input="$PATCHFILE" >"$d/out" 2>&1); then
+    echo "customize-deployment.patch no longer describes the committed template." >&2
+    echo "The patch is the source: anything present only in the template is" >&2
+    echo "discarded by the next 'make update', and anything present only in the" >&2
+    echo "patch appears on every node at that point without ever being tested." >&2
+    cat "$d/out" >&2
+    rm -rf "$d"
+    return 1
+  fi
+  rm -rf "$d"
+}
+
+@test "every multus-cni reference in the manifest is the same string" {
+  # Every container that must run the multus image, found BY NAME. Selecting the
+  # image fields by what they already contain finds only those that still say
+  # multus-cni, so a container repointed at another image leaves the comparison
+  # instead of failing it.
+  rendered=$(helm template --set stageCniPlugins=true packages/system/multus)
+  for c in install-multus-binary install-cni-plugins kube-multus; do
+    img=$(printf '%s' "$rendered" | yq eval "select(.kind == \"DaemonSet\") | [.spec.template.spec.initContainers[], .spec.template.spec.containers[]] | .[] | select(.name == \"$c\") | .image" -)
+    case "$img" in
+      ghcr.io/cozystack/cozystack/multus-cni:*) ;;
+      *)
+        echo "container $c runs '$img', not the cozystack multus image." >&2
+        echo "All three run the same image by construction: the package's" >&2
+        echo "image: target rewrites every multus-cni line together." >&2
+        return 1 ;;
+    esac
+  done
+
+  # `make image` sed's the built reference over every multus-cni line, so they
+  # are one string by construction. Asserted as identity rather than per-index so
+  # a fourth container is covered without touching this test.
+  #
+  # `image:[[:space:]]+`, not a literal space: YAML permits any run, and a
+  # hard-coded one makes the pattern match nothing rather than match differently.
+  refs=$(grep -oE 'image:[[:space:]]+[^[:space:]]*multus-cni[^[:space:]]*' "$TEMPLATE" |
+           sed 's/^image:[[:space:]]*//' | sort -u)
+  count=$(printf '%s\n' "$refs" | grep -c . || true)
+
+  if [ "$count" = "0" ]; then
+    echo "no multus-cni image reference found in the template at all; if the" >&2
+    echo "image was renamed, this test and the package's image: target both" >&2
+    echo "need updating together." >&2
+    return 1
+  fi
+  if [ "$count" != "1" ]; then
+    echo "the manifest carries $count different multus-cni references:" >&2
+    printf '%s\n' "$refs" >&2
+    echo "They must be one string. make image writes all of them together, so a" >&2
+    echo "difference here means one was edited by hand." >&2
+    return 1
+  fi
+}

hack/lib/image-refs.sh

@@ -32,13 +32,12 @@
 # sweep in every vendored upstream chart template and every Helm-templated
 # `image:` line, where a blind rewrite corrupts a value `make update`
 # regenerates. An entry here is a deliberate statement that some package's
-# `image:` target sed's a ref into a file it vendors verbatim.
+# `image:` target sed's a ref into a file it vendors from upstream.
 #
 # system/multus is the only one today: its templates/ is the upstream
 # multus-daemonset-thick.yml fetched by `make update`, and multus's Makefile
-# `image:` target sed's the built ref into two `image:` lines inside it. Left
-# undeclared, multus-cni is invisible to promotion and to the nightly mirror —
-# it receives no stable :vX.Y.Z tag and is never copied to the public registry.
+# `image:` target sed's the built ref into every `multus-cni` `image:` line
+# inside it.
 #
 # KNOWN GAP, deliberately not listed: system/capi-providers-cpprovider stamps
 # its kamaji ref into files/control-plane-components.yaml AND into the
@@ -161,14 +160,25 @@ collect_image_refs() {
 
   # Declared extras additionally get the textual scrape, even though most are
   # YAML and were already parsed above. Parsing ALONE would narrow what this
-  # shape means: an extra is a manifest vendored verbatim from upstream, so it
-  # can carry a ref inside a block scalar (a ConfigMap embedding a whole
-  # deployment, say), where yq returns the enclosing block rather than the ref
-  # and the caller's ownership filter then discards it — and it can stop
-  # parsing entirely when `make update` pulls a version yq chokes on, yielding
-  # nothing at all. Both are silent skips, which is the failure mode this
-  # library exists to remove. Running both and letting the callers' existing
-  # dedup absorb the overlap costs nothing.
+  # shape means: an extra is a manifest vendored from upstream, so it can carry
+  # a ref inside a block scalar (a ConfigMap embedding a whole deployment, say),
+  # where yq returns the enclosing block rather than the ref and the caller's
+  # ownership filter then discards it — and it can stop parsing entirely, when
+  # `make update` pulls a version yq chokes on or when the package patches Helm
+  # syntax into the vendored file. Both are silent skips, which is the failure
+  # mode this library exists to remove. Running both and letting the callers'
+  # existing dedup absorb the overlap costs nothing.
+  #
+  # The multus entry is in the second state today: its staging init container is
+  # wrapped in Helm conditionals, so yq refuses the whole file and this scrape is
+  # not a backstop for it but the only leg that reports anything. One consequence
+  # is worth knowing when reading the tests: the completeness oracle in
+  # hack/promote-rewrite-tags_test.bats builds its expected side with
+  # collect_refs_from_file, so for this file that comparison is inert -- it can
+  # neither fail nor pass on it. What pins the multus entry against the real tree
+  # is `image_ref_files enumerates all three storage shapes` for the path and
+  # `collect_image_refs finds refs the depth-2 values.yaml glob misses` for the
+  # ref, both in that same file.
   for _ir_f in $IMAGE_REF_EXTRA_FILES; do
     [ -f "$_ir_root/$_ir_f" ] || continue
     grep -Eo "[^[:space:]\"']+@sha256:[0-9a-f]{64}" "$_ir_root/$_ir_f" 2>/dev/null || true

hack/multus-install-cni-plugins.bats

@@ -0,0 +1,360 @@
+#!/usr/bin/env bats
+# Behavioural tests for the multus install-cni-plugins init container script:
+# extracted from the rendered DaemonSet and run. The helm-unittest cases in
+# packages/system/multus/tests/multus_test.yaml match its source TEXT instead.
+#
+# Run via hack/cozytest.sh from the repo root (make bats-unit-tests); relative
+# paths resolve against that cwd. The runner has no setup/teardown, so each
+# @test builds its own fixture.
+
+CHART=packages/system/multus
+
+# Extract the init container's script from the rendered chart into $1 and rebind
+# its two absolute paths ($2 = host cni bin, $3 = plugin source) so it can run
+# unprivileged. The non-empty check must be an early `return 1`: hack/cozytest.sh
+# rewrites every line matching ^}$ into `return 0` plus `}`, helpers included.
+render_script() {
+  out=$1
+  # The chart's own default is off -- the platform turns it on per bundle -- so
+  # the init container this file tests only exists when it is set here.
+  helm template --set stageCniPlugins=true "$CHART" \
+    | yq eval 'select(.kind == "DaemonSet") | .spec.template.spec.initContainers[] | select(.name == "install-cni-plugins") | .command[2]' - \
+    | sed -e "s|/host/opt/cni/bin|$2|g" -e "s|/cni-plugins|$3|g" \
+          -e "s|/dev/termination-log|${out}.termlog|g" > "$out"
+  [ -s "$out" ] || return 1
+
+  # Nothing may still point at a real /opt/cni/bin: the rebinding substitutes
+  # /host/opt/cni/bin literally, and these tests EXECUTE the script.
+  if grep -q '/opt/cni/bin' "$out"; then
+    echo "rendered script still references a real /opt/cni/bin after rebinding;" >&2
+    echo "refusing to execute it. Check that the manifest still writes through" >&2
+    echo "/host/opt/cni/bin, which is what this rebinding substitutes." >&2
+    return 1
+  fi
+}
+
+# The names the image actually stages, read from the Dockerfile rather than
+# repeated here: a script that skips a plugin by name is only exercised against
+# a fixture that contains that name.
+plugin_names() {
+  sed -n '/tar -xz -C \/cni-plugins -f /,/;/p' \
+    packages/system/multus/images/multus-cni/Dockerfile |
+    tr ' ' '\n' | sed -n 's|^\./||p' | tr -d ';' | sort
+}
+
+# A fake image payload: plugin binaries that report their own name when run.
+make_plugins() {
+  mkdir -p "$1"
+  for p in $(plugin_names); do
+    printf '#!/bin/sh\necho %s-NEW\n' "$p" > "$1/$p"
+    chmod 0755 "$1/$p"
+  done
+}
+
+# Put an `rm` in $1 that refuses any path containing $2, leaving it in place, and
+# delegates the rest. Injected rather than provoked with permissions: root's DAC
+# override unlinks through a forbidding parent, so a filesystem-shaped fixture
+# stops reproducing wherever the suite runs as root. $2 is narrow so staging can
+# still succeed around one bad path.
+stub_failing_rm() {
+  mkdir -p "$1"
+  sed -e "s|@MATCH@|$2|" > "$1/rm" <<'STUB'
+#!/bin/sh
+for a in "$@"; do
+  case $a in
+    *@MATCH@*)
+      echo "rm: $a: Read-only file system" >&2
+      exit 1
+      ;;
+  esac
+done
+exec /bin/rm "$@"
+STUB
+  chmod 0755 "$1/rm"
+}
+
+@test "a render that produces no script fails instead of passing as a no-op" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  mkdir -p "$tmp/dst"
+  # Guards the guard. Two tests below assert nothing landed in the destination,
+  # so an unrendered script satisfies them. This also fails if the check is
+  # moved into last-command position, where cozytest.sh's `return 0` rewrite
+  # swallows its status. Each @test runs in its own subshell, so reassigning
+  # CHART cannot leak.
+  CHART=$tmp/no-such-chart
+  if render_script "$tmp/s.sh" "$tmp/dst" "$tmp/src" 2>/dev/null; then
+    echo "FAIL: render_script reported success with no chart to render"
+    false
+  fi
+}
+
+@test "installs every staged plugin into the host cni bin dir, executable" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  make_plugins "$tmp/src"; mkdir -p "$tmp/dst"
+  render_script "$tmp/s.sh" "$tmp/dst" "$tmp/src"
+  sh "$tmp/s.sh"
+
+  names=$(plugin_names)
+  [ -n "$names" ]
+  [ "$(printf '%s\n' "$names" | grep -c .)" = "14" ]
+  [ "$(ls -1 "$tmp/dst" | wc -l | tr -d ' ')" = "14" ]
+  # Each by name: a count is satisfied by that many of anything, and one plugin
+  # landing unexecutable leaves the others to carry the assertion.
+  for p in $names; do
+    [ -f "$tmp/dst/$p" ]
+    [ -x "$tmp/dst/$p" ]
+    [ "$("$tmp/dst/$p")" = "$p-NEW" ]
+  done
+}
+
+@test "installs a plugin executable even when the staged copy is not" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  make_plugins "$tmp/src"; mkdir -p "$tmp/dst"
+  # cp reproduces the source mode, so the chmod is only load-bearing when a
+  # staged plugin is not already 0755.
+  chmod 0644 "$tmp/src/bridge"
+  render_script "$tmp/s.sh" "$tmp/dst" "$tmp/src"
+  sh "$tmp/s.sh"
+
+  # Two statements, not `[ -f X ] && [ -x X ]`: a failing command that is not the
+  # last in an AND-OR list is exempt from errexit, so the pair silently continues
+  # into cozytest.sh's appended `return 0` and the whole assertion is discarded.
+  # Measured in dash and bash -- the `-f` half of every such pair was dead.
+  [ -f "$tmp/dst/bridge" ]
+  [ -x "$tmp/dst/bridge" ]
+}
+
+@test "replaces a plugin by rename, not by writing onto the live path" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  make_plugins "$tmp/src"; mkdir -p "$tmp/dst"
+  # Stand in for a plugin another CNI already installed and the kubelet may exec.
+  printf '#!/bin/sh\necho portmap-OLD\n' > "$tmp/dst/portmap"
+  chmod 0755 "$tmp/dst/portmap"
+  before=$(ls -i "$tmp/dst/portmap" | awk '{print $1}')
+
+  render_script "$tmp/s.sh" "$tmp/dst" "$tmp/src"
+  sh "$tmp/s.sh"
+
+  after=$(ls -i "$tmp/dst/portmap" | awk '{print $1}')
+  [ "$("$tmp/dst/portmap")" = "portmap-NEW" ]
+  # A rename swaps in a new inode, so an exec holding the old one keeps a
+  # complete binary; a cp onto the live path truncates and rewrites in place.
+  if [ "$before" = "$after" ]; then echo "FAIL: portmap replaced in place (inode $before unchanged) — copy was not atomic"; false; fi
+}
+
+@test "leaves no temp files behind, and clears ones stranded by an earlier kill" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  make_plugins "$tmp/src"; mkdir -p "$tmp/dst"
+  # Named for a plugin this image does not stage, so no iteration renames it
+  # away -- only the cleanup can remove it.
+  echo stranded > "$tmp/dst/.tmp-cozystack-multus-plugin-from-an-older-image"
+
+  render_script "$tmp/s.sh" "$tmp/dst" "$tmp/src"
+  sh "$tmp/s.sh"
+
+  [ "$(ls -a1 "$tmp/dst" | grep -c '^\.tmp-cozystack-multus-' || true)" = "0" ]
+}
+
+@test "skips staging instead of failing when the image predates the plugins" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  mkdir -p "$tmp/dst"
+  # No source dir at all: the release-prep digest re-pin lags a Dockerfile
+  # change, so the pinned image can have no /cni-plugins.
+  render_script "$tmp/s.sh" "$tmp/dst" "$tmp/absent"
+  sh "$tmp/s.sh" 2>"$tmp/err"
+
+  [ "$(ls -1 "$tmp/dst" | wc -l | tr -d ' ')" = "0" ]
+  # The empty destination alone is satisfied by the guard being gone: the glob
+  # stays literal and `[ -e "$plugin" ]` skips it. Only this line separates them.
+  grep -q 'in this image; skipping staging' "$tmp/err"
+  # And nothing may be reported as failed: a skip is not a partial install.
+  if grep -q 'failed to install:' "$tmp/err"; then
+    echo "FAIL: an absent plugin directory was reported as failed installs"
+    cat "$tmp/err"
+    false
+  fi
+}
+
+@test "does not fail when the staged plugin directory is empty" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  mkdir -p "$tmp/src" "$tmp/dst"
+  render_script "$tmp/s.sh" "$tmp/dst" "$tmp/src"
+  # An unmatched glob stays literal in sh, so without the `[ -e ]` guard the loop
+  # runs once for a path that does not exist and reports a plugin named `*`.
+  sh "$tmp/s.sh" 2>"$tmp/err"
+
+  [ "$(ls -1 "$tmp/dst" | wc -l | tr -d ' ')" = "0" ]
+  # Asserted, because the empty destination alone is satisfied by the guard
+  # being gone.
+  if grep -q 'failed to install:' "$tmp/err"; then
+    echo "FAIL: an empty plugin directory was reported as failed installs"
+    cat "$tmp/err"
+    false
+  fi
+}
+
+@test "a leftover directory is cleared, and the plugin installs as a file" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  make_plugins "$tmp/src"; mkdir -p "$tmp/dst"
+  # The temp path as an ordinary directory `rm -rf` can clear: a previous run
+  # killed between the cp and the rename. Staging proceeds.
+  #
+  # The assertions check for a regular FILE: `-x` alone is true of a directory.
+  mkdir -p "$tmp/dst/.tmp-cozystack-multus-bridge"
+  render_script "$tmp/s.sh" "$tmp/dst" "$tmp/src"
+  sh "$tmp/s.sh"
+
+  [ -f "$tmp/dst/bridge" ]
+  [ -x "$tmp/dst/bridge" ]
+  [ "$("$tmp/dst/bridge")" = "bridge-NEW" ]
+}
+
+@test "a temp path that will not clear is refused, not published as a plugin" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  make_plugins "$tmp/src"; mkdir -p "$tmp/dst"
+  # A temp path that is still a directory when the copy runs: cp writes INTO it,
+  # chmod works on it, and mv publishes it as the plugin, because a rename onto
+  # a free name succeeds. The runtime cannot exec a directory.
+  mkdir -p "$tmp/dst/.tmp-cozystack-multus-bridge"
+  stub_failing_rm "$tmp/bin" ".tmp-cozystack-multus-bridge"
+  render_script "$tmp/s.sh" "$tmp/dst" "$tmp/src"
+
+  # Still must not fail the container. hostNetwork pods bypass CNI and would
+  # start either way; every other pod on the node would not.
+  if ! PATH="$tmp/bin:$PATH" sh "$tmp/s.sh" 2>"$tmp/err"; then
+    echo "FAIL: an unclearable temp path failed the container"
+    cat "$tmp/err"
+    false
+  fi
+  if [ -d "$tmp/dst/bridge" ]; then
+    echo "FAIL: a directory was published as the plugin"
+    ls -la "$tmp/dst/bridge"
+    false
+  fi
+  # Refused, so nothing is there at all -- not a half-installed something.
+  if [ -e "$tmp/dst/bridge" ]; then
+    echo "FAIL: expected bridge to be absent after a refusal"
+    false
+  fi
+  # The injection has to have happened: if PATH stopped being honoured, the real
+  # `rm` would clear the fixture and every assertion above would still pass.
+  if [ ! -d "$tmp/dst/.tmp-cozystack-multus-bridge" ]; then
+    echo "FAIL: the temp path was removed, so no failure was injected;"
+    echo "      this test proved nothing. Check that stub_failing_rm is on PATH"
+    echo "      and that the script still invokes rm by bare name."
+    false
+  fi
+  # And WHICH plugin: anchored at line start, matched as a whole token. The stub
+  # prints the bridge path itself, `.*bridge` would also accept `bridgeport`,
+  # and without `^` the contradictory `not failed to install:` passes too.
+  grep -qE '^failed to install:( [^ ]+)* bridge( |$)' "$tmp/err"
+  # The termination log is the channel the stub cannot write to, so assert there
+  # too -- it is also what `kubectl describe` shows.
+  grep -qE '^failed to install:( [^ ]+)* bridge( |$)' "$tmp/s.sh.termlog"
+  # The other plugins are unaffected: one bad temp path is not a stop signal.
+  [ -f "$tmp/dst/macvlan" ]
+  [ -x "$tmp/dst/macvlan" ]
+  [ "$("$tmp/dst/macvlan")" = "macvlan-NEW" ]
+}
+
+@test "refuses to nest a plugin inside a directory that already holds its name" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  make_plugins "$tmp/src"; mkdir -p "$tmp/dst"
+  # mv moves its source INTO an existing directory, under the SOURCE's basename
+  # -- the temp name, never <dst>/bridge/bridge. Assert the directory gained no
+  # entries: a temp file one level down is deeper than the trap's glob.
+  mkdir -p "$tmp/dst/bridge/occupied"
+  render_script "$tmp/s.sh" "$tmp/dst" "$tmp/src"
+  sh "$tmp/s.sh" 2>"$tmp/err"
+
+  nested=$(ls -A "$tmp/dst/bridge" | grep -v '^occupied$' || :)
+  if [ -n "$nested" ]; then
+    echo "FAIL: something was moved inside the existing directory: $nested"
+    false
+  fi
+  # One line, summary and name together: as two greps this passes when the
+  # summary names a different plugin and `mv` merely printed this one.
+  grep -qE '^failed to install:( [^ ]+)* bridge( |$)' "$tmp/err"
+  # And a CAUSE: this is the one branch that records a failure no command
+  # printed an error for, so without this line the summary is unexplained.
+  grep -q 'is a directory, not a plugin binary' "$tmp/err"
+}
+
+@test "survives a leftover it cannot clean up" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  make_plugins "$tmp/src"; mkdir -p "$tmp/dst"
+  # A leftover only the trap's glob reaches. Its `rm` is guarded because a
+  # command failing inside an EXIT trap sets the status even after `exit 0`.
+  mkdir -p "$tmp/dst/.tmp-cozystack-multus-stranded"
+  stub_failing_rm "$tmp/bin" "stranded"
+  render_script "$tmp/s.sh" "$tmp/dst" "$tmp/src"
+
+  if ! PATH="$tmp/bin:$PATH" sh "$tmp/s.sh" 2>"$tmp/err"; then
+    echo "FAIL: a leftover that could not be removed failed the container;"
+    echo "      every pod that needs CNI would then fail to start here"
+    cat "$tmp/err"
+    false
+  fi
+  # The leftover has to have SURVIVED: that is the only proof the failure was
+  # injected, since a real `rm -rf` would clear it and staging would succeed.
+  if [ ! -d "$tmp/dst/.tmp-cozystack-multus-stranded" ]; then
+    echo "FAIL: the stranded leftover was removed, so no failure was injected;"
+    echo "      this test proved nothing about the EXIT trap's guard."
+    false
+  fi
+  # And the staging still has to have happened.
+  [ -f "$tmp/dst/bridge" ]
+  [ -x "$tmp/dst/bridge" ]
+  [ "$("$tmp/dst/bridge")" = "bridge-NEW" ]
+}
+
+@test "reports a plugin it could not install, and still leaves the node usable" {
+  tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+  make_plugins "$tmp/src"; mkdir -p "$tmp/dst"
+  # An entry cp cannot copy, sorted ahead of the good plugins so a later
+  # iteration still runs.
+  mkdir -p "$tmp/src/aaa-uncopyable"
+  # Plus a leftover the trap cannot remove, so both failure paths run at once:
+  # unguarded, the script dies in the EXIT trap before the summary.
+  mkdir -p "$tmp/dst/.tmp-cozystack-multus-stranded"
+  stub_failing_rm "$tmp/bin" "stranded"
+  render_script "$tmp/s.sh" "$tmp/dst" "$tmp/src"
+
+  # Both halves: exit 0 alone would pass with the message dropped, and the log
+  # line is the only remaining signal.
+  if ! PATH="$tmp/bin:$PATH" sh "$tmp/s.sh" 2>"$tmp/err"; then
+    echo "FAIL: a plugin that could not be installed failed the container;"
+    echo "      every pod that needs CNI would then fail to start here"
+    false
+  fi
+  # The script's own wording AND the name, as one line: cp prints the path it
+  # could not copy, so either half alone passes with the report deleted.
+  if ! grep -qE '^failed to install:( [^ ]+)* aaa-uncopyable( |$)' "$tmp/err"; then
+    echo "FAIL: the script did not report the plugin it could not install"
+    cat "$tmp/err"
+    false
+  fi
+
+  # And into the termination log, which is what `kubectl describe` shows.
+  # invisible to anyone not reading that container's logs specifically.
+  if ! grep -q 'failed to install:.*aaa-uncopyable' "$tmp/s.sh.termlog" 2>/dev/null; then
+    echo "FAIL: the failure was not written to the termination log"
+    false
+  fi
+
+  # The stranded leftover has to have survived, or the injected failure never
+  # happened and the both-go-wrong path was not exercised.
+  if [ ! -d "$tmp/dst/.tmp-cozystack-multus-stranded" ]; then
+    echo "FAIL: the stranded leftover was removed, so no rm failure was injected"
+    false
+  fi
+
+  # The failure must not take the rest of the set with it.
+  [ -f "$tmp/dst/bridge" ]
+  [ -x "$tmp/dst/bridge" ]
+  [ "$("$tmp/dst/bridge")" = "bridge-NEW" ]
+  # And no temp FILE of its own. The planted directory is expected to survive:
+  # stub_failing_rm refuses that path, which is what makes the cleanup fail.
+  # Counting only regular files keeps the fixture from hiding a real leak.
+  [ "$(find "$tmp/dst" -maxdepth 1 -type f -name '.tmp-cozystack-multus-*' | wc -l | tr -d ' ')" = "0" ]
+}

hack/promote-rewrite-tags_test.bats

@@ -596,5 +596,8 @@ ALLOW
 
   refs=$(collect_image_refs packages)
   echo "$refs" | grep -q 'cozystack/grafana-dashboards@sha256:\|cozystack/grafana-dashboards:[^ ]*@sha256:'
-  echo "$refs" | grep -q 'multus-cni'
+  # The whole ref, not just the name: the completeness oracle above builds its
+  # expected side with collect_refs_from_file, which cannot parse this file at
+  # all, so this is where the multus entry is held to a real digest.
+  echo "$refs" | grep -qE 'multus-cni[^[:space:]]*@sha256:[0-9a-f]{64}'
 }

packages/core/platform/templates/_helpers.tpl

@@ -55,16 +55,60 @@ spec:
 {{- include "cozystack.platform.package.optional" (list $name "default" $root) }}
 {{ end }}
 
+{{/*
+Resolve networking.stageCniPlugins for a bundle, given that bundle's default.
+Call as (list $ <default-bool>); emits the string "true" or "false".
+
+Two things this exists for, both of which a bare `dig` gets wrong.
+
+`dig` returns its default only when the key is ABSENT. A key present with a nil
+value returns nil, and nil is how an operator clears an inherited override --
+deleting `stageCniPlugins: true` from spec.components.platform.values by setting
+it to null. A bare dig would hand that straight through as nil, the chart would
+read it as off, and a generic-Linux cluster would silently go back to failing
+bridge-type NADs while values.yaml documents the default as on. Nil is therefore
+folded back onto the variant default, which is what "unset" means to a reader.
+
+And the value can arrive as a string. Today a values file does not produce one
+for a boolean spelling -- measured on Helm v4: yes/no/on/off and true/false all
+resolve to booleans, YAML 1.1 style -- but --set-string does, and that "today"
+is why the spellings are normalised rather than relied on: which of them the
+parser folds is a property of whichever YAML library Helm links, not of this
+chart. A parser that stopped folding yes/no would hand this template the string
+instead, and normalising costs nothing either way. Since this key is how an
+operator declines a behaviour change, a spelling the template does not
+understand must not quietly mean the opposite of what was written: anything
+unrecognised stops the render.
+*/}}
+{{- define "cozystack.platform.stageCniPlugins" -}}
+{{- $root := index . 0 -}}
+{{- $default := index . 1 -}}
+{{- $value := dig "stageCniPlugins" nil ($root.Values.networking | default dict) -}}
+{{- if kindIs "invalid" $value -}}
+{{- $default | toString -}}
+{{- else -}}
+{{- $spelling := $value | toString | lower -}}
+{{- if or (eq $spelling "true") (eq $spelling "yes") (eq $spelling "on") (eq $spelling "1") -}}
+true
+{{- else if or (eq $spelling "false") (eq $spelling "no") (eq $spelling "off") (eq $spelling "0") -}}
+false
+{{- else -}}
+{{- fail (printf "networking.stageCniPlugins: expected a boolean, got %q -- leaving it unset follows the bundle variant" ($value | toString)) -}}
+{{- end -}}
+{{- end -}}
+{{- end }}
+
 {{/*
 Common system packages shared between isp-full and isp-full-generic bundles.
-Does NOT include: networking (variant differs), linstor (talos.enabled differs)
+Does NOT include the packages each variant emits itself: networking (variant
+differs), linstor (talos.enabled differs), multus (stageCniPlugins differs) and
+cozystack-scheduler (emitted with its own variant in both branches)
 */}}
 {{- define "cozystack.platform.system.common-packages" -}}
 {{- $root := . -}}
 {{include "cozystack.platform.package.default" (list "cozystack.kubeovn-webhook" $root) }}
 {{include "cozystack.platform.package.default" (list "cozystack.kubeovn-plunger" $root) }}
 {{include "cozystack.platform.package.default" (list "cozystack.cozy-proxy" $root) }}
-{{include "cozystack.platform.package.default" (list "cozystack.multus" $root) }}
 {{include "cozystack.platform.package.default" (list "cozystack.metallb" $root) }}
 {{include "cozystack.platform.package.default" (list "cozystack.reloader" $root) }}
 {{include "cozystack.platform.package.default" (list "cozystack.linstor-scheduler" $root) }}

packages/core/platform/templates/bundles/system.yaml

@@ -37,6 +37,20 @@
 {{- end -}}
 {{include "cozystack.platform.package" (list "cozystack.networking" "kubeovn-cilium" $ $networkingComponents) }}
 {{include "cozystack.platform.system.common-packages" $ }}
+{{- /* Talos bundles only bridge, firewall, flannel, host-local, loopback and
+       portmap in /opt/cni/bin from its own node image; the other ten this
+       package carries are not there, so a NetworkAttachmentDefinition naming
+       macvlan, ipvlan or vlan fails on Talos exactly as it does on generic
+       Linux. Staging is therefore on here too. The four names that overlap are
+       rewritten on every pod recreation with the SAME upstream version --
+       Talos pins containernetworking/plugins v1.9.1 in siderolabs/pkgs and so
+       do we -- so this is not a downgrade of a system component; see the
+       talos-cni-plugins-checked-against marker in the multus image Dockerfile.
+       networking.stageCniPlugins turns it off, for nodes whose operator
+       manages that directory. The value is always emitted rather than left to
+       the chart default, so the rendered Package states which way it went. */ -}}
+{{- $multusComponents := dict "multus" (dict "values" (dict "stageCniPlugins" (eq (include "cozystack.platform.stageCniPlugins" (list $ true)) "true"))) -}}
+{{include "cozystack.platform.package" (list "cozystack.multus" "default" $ $multusComponents) }}
 {{include "cozystack.platform.package.default" (list "cozystack.linstor" $) }}
 {{include "cozystack.platform.package" (list "cozystack.cozystack-scheduler" "linstor" $) }}
 {{- end }}
@@ -112,6 +126,16 @@
 {{- /* Use kubeovn-cilium-generic variant (no values-talos.yaml) */ -}}
 {{include "cozystack.platform.package" (list "cozystack.networking" "kubeovn-cilium-generic" $ $networkingComponents) }}
 {{include "cozystack.platform.system.common-packages" $ }}
+{{- /* multus staging: same default as the isp-full branch above, and here the
+       gap is wider -- generic Linux has no guarantee that /opt/cni/bin holds
+       any of the reference plugins, since k3s keeps its copies elsewhere and
+       Cilium installs only cilium-cni + loopback. Where staging runs it
+       installs on every pod recreation rather than only when a file is absent:
+       that rewrite is the only path a later plugin version has onto an existing
+       node, so installing only when absent would pin each node to whichever
+       version arrived there first. */ -}}
+{{- $multusComponents := dict "multus" (dict "values" (dict "stageCniPlugins" (eq (include "cozystack.platform.stageCniPlugins" (list $ true)) "true"))) -}}
+{{include "cozystack.platform.package" (list "cozystack.multus" "default" $ $multusComponents) }}
 {{- /* Pass talos.enabled: false to linstor for generic Linux */ -}}
 {{- $linstorComponents := dict "linstor" (dict "values" (dict "talos" (dict "enabled" false))) -}}
 {{include "cozystack.platform.package" (list "cozystack.linstor" "default" $ $linstorComponents) }}

packages/core/platform/tests/bundles_multus_staging_test.yaml

@@ -0,0 +1,287 @@
+suite: networking.stageCniPlugins bundle wiring
+templates:
+  - templates/bundles/system.yaml
+release:
+  name: cozystack
+  namespace: cozy-system
+tests:
+  # The line that delivers the feature: nothing asserted in
+  # packages/system/multus or hack/cni-plugins-staging-contract.bats can tell
+  # whether any bundle switches it on.
+  - it: stages the reference CNI plugins on generic Linux
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full-generic
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: true
+
+  # Talos is the default variant, and it bundles only bridge, firewall, flannel,
+  # host-local, loopback and portmap. The other ten are absent there too, so
+  # staging is on. The four that overlap are rewritten with the same upstream
+  # version rather than an older one, which is what makes this safe.
+  - it: stages the reference CNI plugins on talos as well
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: true
+
+  # Turning staging on is a behaviour change for existing generic clusters, and
+  # the way out cannot be "disable the package": that takes the
+  # NetworkAttachmentDefinition CRD and the vpc app with it.
+  - it: honours an explicit opt-out on generic Linux
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full-generic
+      networking:
+        stageCniPlugins: false
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: false
+
+  # The other direction of the same lever on Talos: an operator who provisions
+  # /opt/cni/bin himself declines there too.
+  - it: honours an explicit opt-out on talos
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full
+      networking:
+        stageCniPlugins: false
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: false
+
+  # Clearing an override is written in Helm by setting the key to null rather
+  # than by deleting it, and `dig` returns its default only for an ABSENT key --
+  # a present-but-nil value comes through as nil. Both variants are asserted,
+  # because the bug would be in the resolution and not in either default.
+  - it: treats an explicit null as unset on generic Linux
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full-generic
+      networking:
+        stageCniPlugins: null
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: true
+
+  - it: treats an explicit null as unset on talos
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full
+      networking:
+        stageCniPlugins: null
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: true
+
+  # A values file can hand the key over as a string, so a spelling the template
+  # does not recognise must not resolve to the opposite of what was written.
+  - it: normalises a quoted spelling instead of reading it as off
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full
+      networking:
+        stageCniPlugins: "True"
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: true
+
+  # The helper holds the accepted spellings as two independent branches and
+  # nothing else makes them agree; a spelling moved across stays accepted while
+  # resolving to the opposite, since only an UNrecognised value calls fail().
+  # The chart's own two lists are a different pair, covered separately.
+  - it: resolves a non-canonical ON spelling as on, not as off
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full
+      networking:
+        stageCniPlugins: "yes"
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: true
+
+  - it: resolves a non-canonical OFF spelling as off, not as on
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full-generic
+      networking:
+        stageCniPlugins: "off"
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: false
+
+  - it: resolves the ON spelling 'on' the way it is documented
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full
+      networking:
+        stageCniPlugins: "on"
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: true
+
+  - it: resolves the ON spelling '1' the way it is documented
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full
+      networking:
+        stageCniPlugins: "1"
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: true
+
+  - it: resolves the OFF spelling 'no' the way it is documented
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full-generic
+      networking:
+        stageCniPlugins: "no"
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: false
+
+  - it: resolves the OFF spelling '0' the way it is documented
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full-generic
+      networking:
+        stageCniPlugins: "0"
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.components.multus.values.stageCniPlugins
+          value: false
+
+  - it: refuses a value that is neither true nor false
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full-generic
+      networking:
+        stageCniPlugins: maybe
+    asserts:
+      - failedTemplate:
+          errorMessage: 'networking.stageCniPlugins: expected a boolean, got "maybe" -- leaving it unset follows the bundle variant'
+
+  # multus is included by hand in both variant branches rather than through
+  # cozystack.platform.system.common-packages. The two branches currently pass
+  # the same value, and that is not an oversight: the value is variant-scoped by
+  # construction, not because it happens to differ. It differed when Talos was
+  # thought to carry the whole reference set; it stopped differing when that
+  # turned out to be five plugins out of fourteen. Folding multus back into the
+  # shared helper would edit a template every package renders through, and is
+  # worth doing only if variant scoping is decided to be wrong in principle --
+  # not because two lines read alike today.
+  #
+  # What the two cases below hold is that multus is shipped at all in each
+  # variant: dropping it takes the NetworkAttachmentDefinition CRD with it, and
+  # the vpc app dependsOn cozystack.multus -- that edge is asserted in
+  # tests/vpc_multus_dependency_test.yaml, which renders a different template and
+  # so cannot share this suite.
+  - it: ships multus in the talos bundle
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.variant
+          value: default
+
+  - it: ships multus in the generic bundle
+    set:
+      bundles:
+        system:
+          enabled: true
+          variant: isp-full-generic
+    documentSelector:
+      path: metadata.name
+      value: cozystack.multus
+    asserts:
+      - equal:
+          path: spec.variant
+          value: default

packages/core/platform/tests/vpc_multus_dependency_test.yaml

@@ -0,0 +1,25 @@
+suite: the vpc application depends on multus
+# Its own suite because it renders templates/sources.yaml: adding that template
+# to a suite built around templates/bundles/system.yaml makes every
+# metadata.name selector there match in two documents at once.
+templates:
+  - templates/sources.yaml
+release:
+  name: cozystack
+  namespace: cozy-system
+tests:
+  # The vpc application uses NetworkAttachmentDefinitions, whose CRD multus
+  # installs, so vpc has to install AFTER multus; the edge below is what makes
+  # Flux wait rather than race. It lives in a
+  # static PackageSource the chart ships verbatim through .Files.Glob, which is
+  # easy to mistake for something no rendered document carries: `helm template .`
+  # on this chart fails whole on an OCIRepository lookup and emits nothing at
+  # all, and that empty output reads like absence rather than like an error.
+  - it: keeps the vpc application depending on multus
+    documentSelector:
+      path: metadata.name
+      value: cozystack.virtualprivatecloud-application
+    asserts:
+      - contains:
+          path: spec.variants[0].dependsOn
+          content: cozystack.multus

packages/core/platform/values.yaml

@@ -109,6 +109,37 @@ networking:
   # has no effect.
   encryption:
     enabled: false
+  # Install the upstream reference CNI plugins (bridge, macvlan, ipvlan, ...)
+  # into /opt/cni/bin on every node, from the multus image. Multus-backed
+  # NetworkAttachmentDefinitions delegate to these. Talos puts six binaries in
+  # that directory -- bridge, firewall, host-local, loopback and portmap from the
+  # same upstream release, plus flannel from a different repo -- and generic Linux
+  # gives no guarantee at all, so a bridge-type NAD fails there with
+  # `failed to find plugin "bridge" in path [/opt/cni/bin]`. A NAD naming one of
+  # the other ten needs staging on either platform.
+  #
+  # On for both variants when unset. Four of the names Talos carries are ones
+  # staging writes, so it rewrites those with the same upstream version (v1.9.1
+  # on both sides) and adds the ten it does not carry; loopback is not staged and
+  # flannel was never part of the reference set. Set it explicitly to decline.
+  #
+  # A spelling this chart does not recognise stops the whole platform render, not
+  # just multus, so a typo here halts reconciliation cluster-wide. That is
+  # deliberate -- reading an unreadable value as "off" would defeat the opt-out --
+  # but it is why the accepted spellings are worth getting right first time.
+  #
+  # When on, the plugins are written every time the Multus pod is recreated,
+  # replacing files of the same name. 14 of the upstream reference plugins are
+  # installed; loopback, dhcp, dummy and tap are left alone. See
+  # docs/vm-external-vlan.md.
+  #
+  # On a running cluster this is set on the `platform` component of the
+  # `cozystack.cozystack-platform` Package CR, the same path as backupStorage
+  # (see docs/operations/backup-classes.md). Editing the multus HelmRelease or
+  # the child Package cozystack.multus directly is reverted by the platform
+  # reconcile.
+  # stageCniPlugins: true
+
 # Service publishing and ingress configuration
 publishing:
   host: "example.org"

packages/system/multus/Makefile

@@ -7,13 +7,20 @@ include ../../../hack/package.mk
 test:
 	helm unittest .
 
+# The patch seeds the init container it adds with the UPSTREAM image
+# reference, so the release rewrite runs after it and covers that line too --
+# leaving every `multus-cni` line at one reference rather than the added one
+# alone at `snapshot-thick`. Only `image` below stamps the built reference.
+# A tree committed between the two targets is caught by the assertion on
+# ^ghcr\.io/cozystack/cozystack/multus-cni: in
+# packages/system/multus/tests/multus_test.yaml.
 update:
 	rm -rf templates
 	mkdir templates
 	$(eval RELEASE := $(shell curl -s https://api.github.com/repos/k8snetworkplumbingwg/multus-cni/releases/latest?per_page=1 | jq -r '.name'))
 	wget -q https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/refs/tags/$(RELEASE)/deployments/multus-daemonset-thick.yml -O templates/multus-daemonset-thick.yml
-	sed -i '/multus-cni/s/snapshot-thick/$(RELEASE)-thick/' templates/multus-daemonset-thick.yml && \
-	patch --no-backup-if-mismatch -p4 < patches/customize-deployment.patch
+	patch --no-backup-if-mismatch -p4 < patches/customize-deployment.patch && \
+	sed -i '/multus-cni/s/snapshot-thick/$(RELEASE)-thick/' templates/multus-daemonset-thick.yml
 	$(SED_INPLACE) "/ARG VERSION/ s|=.*|=$(RELEASE)|g" images/multus-cni/Dockerfile
 
 image:

packages/system/multus/images/multus-cni/Dockerfile

@@ -14,12 +14,68 @@ RUN git init && git config user.email "build@cozystack" && git config user.name
     git apply /patches/*.diff
 
 RUN ./hack/build-go.sh
+# The upstream reference CNI plugins, installed into the host /opt/cni/bin by
+# the install-cni-plugins init container. See packages/system/multus/values.yaml
+# for the switch.
+#
+# Four of the release's plugins are not staged:
+#
+#   loopback   the runtime calls it for every pod sandbox, not only for pods
+#              using a NAD; cilium and kube-ovn both install it already.
+#   dhcp       needs a long-running host daemon this package does not run.
+#   tap, dummy no consumer in this tree.
+#
+# The list here, the count, and docs/vm-external-vlan.md are asserted against
+# each other in hack/cni-plugins-staging-contract.bats.
+#
+# kube-ovn installs portmap and macvlan into the same directory from its own
+# image, so those two have a second writer, and its plugins version lives inside
+# its base image rather than in this tree. The marker below records the kube-ovn
+# release CNI_PLUGINS_VERSION was last checked against; the same test compares
+# the two.
+#
+# kube-ovn-cni-plugins-checked-against: v1.15.10
+#
+# Talos strips the same upstream release in siderolabs/talos' Dockerfile down to
+# bridge, firewall, host-local, loopback and portmap, and adds flannel from a
+# different repo. Four of those are names this image stages -- loopback is not,
+# it is excluded above, and flannel was never in the reference set. Staging is on
+# for Talos too, so those four are rewritten on every pod recreation; that is
+# safe only while both sides pin the SAME containernetworking/plugins version.
+# The marker records the Talos release whose pin was compared with
+# CNI_PLUGINS_VERSION above -- it is about the VERSIONS matching, not about
+# which plugins exist.
+#
+# talos-cni-plugins-checked-against: v1.13.6
+ARG TARGETARCH
+ARG CNI_PLUGINS_VERSION=v1.9.1
+# SHA-256 of cni-plugins-linux-<arch>-<version>.tgz, from the release's per-asset
+# .sha256 sidecar. Bump together with CNI_PLUGINS_VERSION.
+RUN set -eux; \
+    case "${TARGETARCH}" in \
+      amd64) sha256="b98f74a0f8522f0a83867178729c1aa70f2158f90c45a2ca8fa791db1c76b303" ;; \
+      arm64) sha256="56171987d3947707c3563db2f4001bccaf50fd63468611b9f3cbecb1375ee7ec" ;; \
+      *) echo "unsupported TARGETARCH=${TARGETARCH:-<empty>} (build with buildx)" >&2; exit 1 ;; \
+    esac; \
+    curl --fail --silent --show-error --location -o /tmp/cni-plugins.tgz \
+      "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_VERSION}/cni-plugins-linux-${TARGETARCH}-${CNI_PLUGINS_VERSION}.tgz"; \
+    echo "${sha256}  /tmp/cni-plugins.tgz" | sha256sum -c -; \
+    mkdir -p /cni-plugins /cni-plugins-licenses; \
+    tar -xz -C /cni-plugins -f /tmp/cni-plugins.tgz \
+      ./bridge ./host-local ./static ./macvlan ./ipvlan ./vlan \
+      ./tuning ./portmap ./firewall ./bandwidth ./host-device ./ptp ./sbr ./vrf; \
+    tar -xz -C /cni-plugins-licenses -f /tmp/cni-plugins.tgz ./LICENSE; \
+    rm -f /tmp/cni-plugins.tgz
 
 FROM debian:stable-slim@sha256:ee12ffb55625b99d62837a72f037d9b2f18fd0c787a89c2b9a4f09666c48776c
 LABEL org.opencontainers.image.source=https://github.com/cozystack/cozystack
 COPY --from=build /usr/src/multus-cni/bin /usr/src/multus-cni/bin
 COPY --from=build /usr/src/multus-cni/LICENSE /usr/src/multus-cni/LICENSE
 COPY --from=build /usr/src/multus-cni/bin/cert-approver /
+COPY --from=build /cni-plugins /cni-plugins
+# Apache-2.0 licence of the plugins redistributed above. Kept out of
+# /cni-plugins so it is not installed as a plugin.
+COPY --from=build /cni-plugins-licenses /cni-plugins-licenses
 WORKDIR /
 
 ENTRYPOINT ["/usr/src/multus-cni/bin/multus-daemon"]

packages/system/multus/patches/customize-deployment.patch

@@ -53,3 +53,80 @@
            securityContext:
              privileged: true
            terminationMessagePolicy: FallbackToLogsOnError
+@@ -220,6 +220,76 @@
+             - name: cnibin
+               mountPath: /host/opt/cni/bin
+               mountPropagation: Bidirectional
++{{- $stageCniPlugins := lower (ternary "false" (printf "%v" .Values.stageCniPlugins) (kindIs "invalid" .Values.stageCniPlugins)) }}
++{{- if not (has $stageCniPlugins (list "true" "yes" "on" "1" "false" "no" "off" "0")) }}{{- fail (printf "stageCniPlugins: expected a boolean, got %q" $stageCniPlugins) }}{{- end }}
++{{- if has $stageCniPlugins (list "true" "yes" "on" "1") }}
++        - name: install-cni-plugins
++          image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick
++          command:
++            - "sh"
++            - "-c"
++            - |
++              set -eu
++              # cilium and kube-ovn write into this directory too, and a
++              # plugin here may be executing, so each install is a rename.
++              trap 'rm -rf /host/opt/cni/bin/.tmp-cozystack-multus-* || :' EXIT
++              if [ ! -d /cni-plugins ]; then
++                echo "no /cni-plugins in this image; skipping staging" >&2
++                exit 0
++              fi
++              failed=
++              for plugin in /cni-plugins/*; do
++                [ -e "$plugin" ] || continue
++                name="${plugin##*/}"
++                tmp="/host/opt/cni/bin/.tmp-cozystack-multus-${name}"
++                # A directory at either end makes every step succeed while
++                # installing nothing.
++                rm -rf "$tmp" || { failed="${failed} ${name}"; continue; }
++                if [ -d "/host/opt/cni/bin/${name}" ]; then
++                  echo "${name} is a directory, not a plugin binary; left as is" >&2
++                  failed="${failed} ${name}"
++                  continue
++                fi
++                if cp -f "$plugin" "$tmp" && chmod 0755 "$tmp" &&
++                   mv -f "$tmp" "/host/opt/cni/bin/${name}"; then
++                  continue
++                fi
++                rm -rf "$tmp" || :
++                failed="${failed} ${name}"
++              done
++              if [ -n "$failed" ]; then
++                echo "failed to install:${failed}" >&2
++                echo "those were left as they were" >&2
++                echo "the cause for each is earlier in this log" >&2
++                echo "failed to install:${failed}" > /dev/termination-log 2>/dev/null || true
++              fi
++              exit 0
++          resources:
++            requests:
++              cpu: "10m"
++              memory: "15Mi"
++          # cilium's install-cni-binaries copies plugin binaries into this same
++          # directory with ALL capabilities dropped, in the spc_t domain, with
++          # no privileged flag. On a node with SELinux enforcing, a container
++          # that has dropped every capability still cannot write a host-labelled
++          # directory without a domain that permits it.
++          #
++          # mountPropagation is absent: it governs mounts made INSIDE the
++          # container, which this script does not make, and the API server
++          # rejects Bidirectional unless the container is privileged.
++          securityContext:
++            allowPrivilegeEscalation: false
++            capabilities:
++              drop:
++                - ALL
++            seLinuxOptions:
++              level: s0
++              type: spc_t
++          terminationMessagePolicy: FallbackToLogsOnError
++          volumeMounts:
++            - name: cnibin
++              mountPath: /host/opt/cni/bin
++{{- end }}
+       terminationGracePeriodSeconds: 10
+       volumes:
+         - name: cni

packages/system/multus/templates/multus-daemonset-thick.yml

@@ -220,6 +220,76 @@ spec:
             - name: cnibin
               mountPath: /host/opt/cni/bin
               mountPropagation: Bidirectional
+{{- $stageCniPlugins := lower (ternary "false" (printf "%v" .Values.stageCniPlugins) (kindIs "invalid" .Values.stageCniPlugins)) }}
+{{- if not (has $stageCniPlugins (list "true" "yes" "on" "1" "false" "no" "off" "0")) }}{{- fail (printf "stageCniPlugins: expected a boolean, got %q" $stageCniPlugins) }}{{- end }}
+{{- if has $stageCniPlugins (list "true" "yes" "on" "1") }}
+        - name: install-cni-plugins
+          image: ghcr.io/cozystack/cozystack/multus-cni:v1.6.0@sha256:0a92e861ead179dd6b46f475fd8878d44b3cfa40ce4d7a8ff92f790d54250d25
+          command:
+            - "sh"
+            - "-c"
+            - |
+              set -eu
+              # cilium and kube-ovn write into this directory too, and a
+              # plugin here may be executing, so each install is a rename.
+              trap 'rm -rf /host/opt/cni/bin/.tmp-cozystack-multus-* || :' EXIT
+              if [ ! -d /cni-plugins ]; then
+                echo "no /cni-plugins in this image; skipping staging" >&2
+                exit 0
+              fi
+              failed=
+              for plugin in /cni-plugins/*; do
+                [ -e "$plugin" ] || continue
+                name="${plugin##*/}"
+                tmp="/host/opt/cni/bin/.tmp-cozystack-multus-${name}"
+                # A directory at either end makes every step succeed while
+                # installing nothing.
+                rm -rf "$tmp" || { failed="${failed} ${name}"; continue; }
+                if [ -d "/host/opt/cni/bin/${name}" ]; then
+                  echo "${name} is a directory, not a plugin binary; left as is" >&2
+                  failed="${failed} ${name}"
+                  continue
+                fi
+                if cp -f "$plugin" "$tmp" && chmod 0755 "$tmp" &&
+                   mv -f "$tmp" "/host/opt/cni/bin/${name}"; then
+                  continue
+                fi
+                rm -rf "$tmp" || :
+                failed="${failed} ${name}"
+              done
+              if [ -n "$failed" ]; then
+                echo "failed to install:${failed}" >&2
+                echo "those were left as they were" >&2
+                echo "the cause for each is earlier in this log" >&2
+                echo "failed to install:${failed}" > /dev/termination-log 2>/dev/null || true
+              fi
+              exit 0
+          resources:
+            requests:
+              cpu: "10m"
+              memory: "15Mi"
+          # cilium's install-cni-binaries copies plugin binaries into this same
+          # directory with ALL capabilities dropped, in the spc_t domain, with
+          # no privileged flag. On a node with SELinux enforcing, a container
+          # that has dropped every capability still cannot write a host-labelled
+          # directory without a domain that permits it.
+          #
+          # mountPropagation is absent: it governs mounts made INSIDE the
+          # container, which this script does not make, and the API server
+          # rejects Bidirectional unless the container is privileged.
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+            seLinuxOptions:
+              level: s0
+              type: spc_t
+          terminationMessagePolicy: FallbackToLogsOnError
+          volumeMounts:
+            - name: cnibin
+              mountPath: /host/opt/cni/bin
+{{- end }}
       terminationGracePeriodSeconds: 10
       volumes:
         - name: cni

packages/system/multus/tests/multus_test.yaml

@@ -29,3 +29,373 @@ tests:
       - matchRegex:
           path: data["daemon-config.json"]
           pattern: '"multusMasterCNI": "05-cilium\.conflist"'
+
+  # A quoted "false" is a non-empty string; a truthiness test would read it as
+  # on. The value is compared as text.
+  - it: treats a quoted "false" as off, not as a non-empty string
+    set:
+      stageCniPlugins: "false"
+    template: templates/multus-daemonset-thick.yml
+    documentSelector:
+      path: kind
+      value: DaemonSet
+    asserts:
+      - lengthEqual:
+          path: spec.template.spec.initContainers
+          count: 1
+      - equal:
+          path: spec.template.spec.initContainers[0].name
+          value: install-multus-binary
+
+  - it: stages when the value arrives as a quoted "true"
+    set:
+      stageCniPlugins: "true"
+    template: templates/multus-daemonset-thick.yml
+    documentSelector:
+      path: kind
+      value: DaemonSet
+    asserts:
+      - equal:
+          path: spec.template.spec.initContainers[1].name
+          value: install-cni-plugins
+
+  # The chart is reachable directly (helm install, the cozypkg dev loop), so it
+  # holds the same line as the platform helper: an unreadable spelling stops the
+  # render instead of resolving to off.
+  - it: refuses a value that is neither true nor false
+    set:
+      stageCniPlugins: maybe
+    asserts:
+      - failedTemplate:
+          errorMessage: 'stageCniPlugins: expected a boolean, got "maybe"'
+
+  # An explicit null is how a values file clears the key, and the platform helper
+  # folds it back onto its default rather than treating it as a spelling. The
+  # chart does the same with ITS default, off -- without this the render fails
+  # naming "<nil>", a spelling the operator never typed.
+  - it: reads an explicit null as the chart default rather than refusing it
+    set:
+      stageCniPlugins: null
+    template: templates/multus-daemonset-thick.yml
+    documentSelector:
+      path: kind
+      value: DaemonSet
+    asserts:
+      - lengthEqual:
+          path: spec.template.spec.initContainers
+          count: 1
+      - equal:
+          path: spec.template.spec.initContainers[0].name
+          value: install-multus-binary
+
+  - it: does not stage anything unless asked to
+    template: templates/multus-daemonset-thick.yml
+    documentSelector:
+      path: kind
+      value: DaemonSet
+    asserts:
+      - equal:
+          path: spec.template.spec.initContainers[0].name
+          value: install-multus-binary
+      - lengthEqual:
+          path: spec.template.spec.initContainers
+          count: 1
+
+  # The staging init container installs the plugins bundled in the image into
+  # /opt/cni/bin. Distros like k3s do not populate it with them. The container
+  # lives in patches/customize-deployment.patch so it survives every make update.
+  - it: stages the reference CNI plugins into the host cni bin dir
+    template: templates/multus-daemonset-thick.yml
+    set:
+      stageCniPlugins: true
+    documentSelector:
+      path: kind
+      value: DaemonSet
+    asserts:
+      - equal:
+          path: spec.template.spec.initContainers[1].name
+          value: install-cni-plugins
+      # What runs the script, not only what it says. The behavioural suite
+      # executes command[2] under sh and never reads command[0] and [1], so a
+      # container handed the same text to `echo` would print it, exit 0 and
+      # stage nothing while every suite stays green.
+      - equal:
+          path: spec.template.spec.initContainers[1].command[0]
+          value: sh
+      - equal:
+          path: spec.template.spec.initContainers[1].command[1]
+          value: -c
+      # restartPolicy on an init container makes it a native sidecar: the
+      # kubelet restarts it when it exits, so the staging loop would rewrite
+      # /opt/cni/bin continuously instead of once per pod recreation, which is
+      # what the bundle comment and the commit message both claim it does.
+      - notExists:
+          path: spec.template.spec.initContainers[1].restartPolicy
+      # args are appended after command[2], where they become the positional
+      # parameters of the script rather than more script.
+      - notExists:
+          path: spec.template.spec.initContainers[1].args
+      # The volume NAME, not only the mountPath: which hostPath sits behind it is
+      # decided here. Swapping cnibin for hostroot, mounted at /, keeps every
+      # other assertion true while the plugins land elsewhere.
+      - equal:
+          path: spec.template.spec.initContainers[1].volumeMounts[0].name
+          value: cnibin
+      - equal:
+          path: spec.template.spec.initContainers[1].volumeMounts[0].mountPath
+          value: /host/opt/cni/bin
+      # subPath moves what the mountPath actually resolves to: with
+      # `subPath: staging` the container writes into <hostPath>/staging and
+      # every step still succeeds, so the script reports a clean install while
+      # the live plugin directory is untouched. subPathExpr is the same field
+      # with variable expansion.
+      - notExists:
+          path: spec.template.spec.initContainers[1].volumeMounts[0].subPath
+      - notExists:
+          path: spec.template.spec.initContainers[1].volumeMounts[0].subPathExpr
+      # And exactly one mount: asserting [0] says nothing about a second one.
+      # A mount at /cni-plugins masks the image payload, so the loop would
+      # install whatever that volume holds instead of the fourteen plugins.
+      - lengthEqual:
+          path: spec.template.spec.initContainers[1].volumeMounts
+          count: 1
+      # The script writes its report to the literal /dev/termination-log, and
+      # this field decides where the kubelet reads it from. Point it elsewhere
+      # and `kubectl describe` shows nothing while the container still exits 0.
+      - notExists:
+          path: spec.template.spec.initContainers[1].terminationMessagePath
+      # FallbackToLogsOnError reads the container log into the status message
+      # only when the container exits with an ERROR and the termination file is
+      # empty (k8s.io/api core/v1: TerminationMessageFallbackToLogsOnError).
+      # This container exits 0 by design, so on the ordinary path the file is
+      # what `kubectl describe` shows and the policy changes nothing -- a failed
+      # write leaves the message empty either way, and only the container log
+      # keeps the report. It is pinned for the other path: if the script ever
+      # dies before reaching `exit 0`, the log is then the only place the reason
+      # exists, and `File` would report nothing at all.
+      - equal:
+          path: spec.template.spec.initContainers[1].terminationMessagePolicy
+          value: FallbackToLogsOnError
+      # The LSM side that is not SELinux, and the one place this container asks
+      # for nothing. Leaving the field unset means containerd's default profile
+      # applies, which permits ordinary file writes, so staging works under it.
+      # Unset is therefore the state to hold: setting it is how the container
+      # would gain LSM privilege it does not need, and cilium's nsenter escape
+      # hatch in packages/system/cilium/values-apparmor.yaml shows what asking
+      # looks like. Should a future default profile ever refuse these writes,
+      # the remedy is `appArmorProfile: {type: Unconfined}` -- and changing this
+      # assertion is then a deliberate act rather than an accident.
+      - notExists:
+          path: spec.template.spec.initContainers[1].securityContext.appArmorProfile
+      - notExists:
+          path: spec.template.spec.securityContext.appArmorProfile
+      # A read-only mount is refused by the kernel on every write, so the script
+      # reports each plugin as failed, exits 0 by design, and the pod goes Ready
+      # having staged nothing.
+      - notEqual:
+          path: spec.template.spec.initContainers[1].volumeMounts[0].readOnly
+          value: true
+      # The volume itself, name and hostPath together: the mount names cnibin,
+      # but which host directory sits behind that name is decided in volumes,
+      # and nothing above reads it. Asserted as a pair rather than by index.
+      - contains:
+          path: spec.template.spec.volumes
+          content:
+            name: cnibin
+            hostPath:
+              path: /opt/cni/bin
+      # mountPropagation governs mounts made INSIDE the container; this script
+      # makes none, and its only other effect is to force privileged.
+      - notExists:
+          path: spec.template.spec.initContainers[1].volumeMounts[0].mountPropagation
+      - notExists:
+          path: spec.template.spec.initContainers[1].securityContext.privileged
+      - equal:
+          path: spec.template.spec.initContainers[1].securityContext.allowPrivilegeEscalation
+          value: false
+      - contains:
+          path: spec.template.spec.initContainers[1].securityContext.capabilities.drop
+          content: ALL
+      # `drop: [ALL]` alone is not "every capability dropped": `add` is applied
+      # after it, so a later NET_ADMIN would be granted with this assertion
+      # still green.
+      - notExists:
+          path: spec.template.spec.initContainers[1].securityContext.capabilities.add
+      # On an enforcing node a container with every capability dropped cannot
+      # write a host-labelled /opt/cni/bin without a domain that allows it.
+      - equal:
+          path: spec.template.spec.initContainers[1].securityContext.seLinuxOptions.type
+          value: spc_t
+      # The level is part of the label, not decoration: an invalid or
+      # incompatible one stops the kubelet from creating the container at all,
+      # before the script has a chance to run.
+      - equal:
+          path: spec.template.spec.initContainers[1].securityContext.seLinuxOptions.level
+          value: s0
+      # env is the other half of "what runs": the command is pinned above, but a
+      # PATH pointing at nothing makes sh itself unresolvable, and the container
+      # then stages nothing while every assertion here still holds.
+      - notExists:
+          path: spec.template.spec.initContainers[1].env
+      # envFrom reaches the same variables by another route, and a referenced
+      # ConfigMap that is missing and not optional stops the container from
+      # starting at all.
+      - notExists:
+          path: spec.template.spec.initContainers[1].envFrom
+      # workingDir is resolved before the command runs: point it at a directory
+      # that does not exist and the container never starts, so the same 'nothing
+      # was staged' outcome arrives as a crashloop instead of a clean exit.
+      - notExists:
+          path: spec.template.spec.initContainers[1].workingDir
+      # What this costs on every node. Multus is a DaemonSet, so an init
+      # container's requests are charged once per node in the fleet, and an
+      # init container's requests also set the floor for the whole pod while it
+      # runs. Pinned so a change is a decision rather than a drift.
+      - equal:
+          path: spec.template.spec.initContainers[1].resources.requests.cpu
+          value: 10m
+      - equal:
+          path: spec.template.spec.initContainers[1].resources.requests.memory
+          value: 15Mi
+      # And no limits: a memory limit low enough to kill the copy turns staging
+      # into a crashloop, and there is nothing here whose runtime is worth
+      # capping -- it copies fourteen files and exits.
+      - notExists:
+          path: spec.template.spec.initContainers[1].resources.limits
+      # Writing into the hostPath needs uid 0 IN THE CONTAINER. Nothing sets a
+      # user today, so it runs as root; a runAsUser or runAsNonRoot at either
+      # level turns every install into a permission error, and the script then
+      # reports them, exits 0 by design and leaves the pod Ready with nothing
+      # staged. hack/multus-install-cni-plugins.bats cannot see this: it runs
+      # the extracted script as whoever runs the suite, not through the kubelet.
+      # The pod level matters as much as the container level -- `make update`
+      # re-fetches this manifest from upstream, and a pod-level hardening added
+      # there would be inherited without touching the patch.
+      - notExists:
+          path: spec.template.spec.initContainers[1].securityContext.runAsUser
+      - notExists:
+          path: spec.template.spec.initContainers[1].securityContext.runAsNonRoot
+      - notExists:
+          path: spec.template.spec.securityContext.runAsUser
+      - notExists:
+          path: spec.template.spec.securityContext.runAsNonRoot
+
+  # kube-ovn installs portmap and macvlan into the same /opt/cni/bin with a
+  # plain cp, so plugins are replaced while the kubelet may be executing them.
+  # A rename is atomic within the mount; writing onto the live path is not.
+  - it: installs each plugin atomically rather than writing onto the live path
+    template: templates/multus-daemonset-thick.yml
+    set:
+      stageCniPlugins: true
+    documentSelector:
+      path: kind
+      value: DaemonSet
+    asserts:
+      # Index [1] is the right container only while it is still named this, and a
+      # negative assert against the wrong container passes without checking.
+      - equal:
+          path: spec.template.spec.initContainers[1].name
+          value: install-cni-plugins
+      # The copy must target the temp name...
+      - matchRegex:
+          path: spec.template.spec.initContainers[1].command[2]
+          pattern: '(?m)^\s*if cp -f "\$plugin" "\$tmp"'
+      # ...and only the rename may publish the final name.
+      - matchRegex:
+          path: spec.template.spec.initContainers[1].command[2]
+          pattern: '(?m)^\s*mv -f "\$tmp" "/host/opt/cni/bin/\$\{name\}"'
+      # Catches the literal spellings of the regression. A copy reaching the same
+      # path through a variable has no literal to match, so
+      # hack/multus-install-cni-plugins.bats owns the real guarantee by asserting
+      # the replaced inode changes. (?m) is load-bearing: Go anchors $ to end of
+      # text, not end of line.
+      - notMatchRegex:
+          path: spec.template.spec.initContainers[1].command[2]
+          pattern: '(?m)^[^#\n]*\b(cp|install)\b[^\n]*/host/opt/cni/bin(/|"|\s|$)'
+
+      # Unlinking the live name before the rename is not a replacement by
+      # rename: it opens a window where the path does not exist at all, and a
+      # failure after it destroys the copy that was working. The inode still
+      # changes, so the runtime check in hack/multus-install-cni-plugins.bats
+      # cannot tell the two apart. The trap's glob is a different path.
+      - notMatchRegex:
+          path: spec.template.spec.initContainers[1].command[2]
+          pattern: '(?m)^[^#\n]*\brm\b[^\n]*/host/opt/cni/bin/"?\$\{?name\}?'
+
+      # release-prep bakes the digest, so between a Dockerfile change and the next
+      # bake the pinned image can predate the staging and have no /cni-plugins.
+      # A successfully built image always has the directory -- the Dockerfile's
+      # tar fails otherwise -- so the guard can only ever mask that skew.
+  - it: skips staging instead of crashlooping when the image predates the plugins
+    template: templates/multus-daemonset-thick.yml
+    set:
+      stageCniPlugins: true
+    documentSelector:
+      path: kind
+      value: DaemonSet
+    asserts:
+      - equal:
+          path: spec.template.spec.initContainers[1].name
+          value: install-cni-plugins
+      # Guard and body as one pattern: an unanchored `exit 0` is satisfied by any
+      # exit in the script. The trailing anchor matters too -- `exit 01` exits 1
+      # and contains `exit 0`.
+      - matchRegex:
+          path: spec.template.spec.initContainers[1].command[2]
+          pattern: '(?m)^\s*if \[ ! -d /cni-plugins \]; then\n[^\n]*\n[^\n]*exit 0\s*$'
+      # Deleting any of the next four leaves every other assertion green -- found
+      # by deleting each executable line of the script in turn and seeing which
+      # deletions nothing noticed.
+      #
+      # The shell options, as the first command: -u turns a mistyped variable
+      # into an error instead of an empty string, which here would mean cp'ing
+      # to "" or removing the wrong path.
+      - matchRegex:
+          path: spec.template.spec.initContainers[1].command[2]
+          pattern: '(?m)\A\s*set -eu\s*$'
+      # The two operator-facing lines of the failure report. The summary and the
+      # plugin name are pinned by hack/multus-install-cni-plugins.bats; these two
+      # carry what remains and where to look, and nothing held them.
+      - matchRegex:
+          path: spec.template.spec.initContainers[1].command[2]
+          pattern: '(?m)^\s*echo "those were left as they were" >&2\s*$'
+      - matchRegex:
+          path: spec.template.spec.initContainers[1].command[2]
+          pattern: '(?m)^\s*echo "the cause for each is earlier in this log" >&2\s*$'
+      # The explicit exit, as the last command. Without it the container's status
+      # is whatever the preceding command returned; that is 0 today by accident
+      # of what stands last, and the design does not want to depend on it.
+      - matchRegex:
+          path: spec.template.spec.initContainers[1].command[2]
+          pattern: '(?m)^\s*exit 0\s*\z'
+
+      # A kill between the copy and the rename strands a temp file in a directory
+      # shared with cilium and kube-ovn, as multus's own CopyFileAtomic handles.
+      # hack/multus-install-cni-plugins.bats covers the behaviour; this pins the
+      # trap into the rendered manifest.
+  - it: does not leak temp files into the shared cni bin dir
+    template: templates/multus-daemonset-thick.yml
+    set:
+      stageCniPlugins: true
+    documentSelector:
+      path: kind
+      value: DaemonSet
+    asserts:
+      - equal:
+          path: spec.template.spec.initContainers[1].name
+          value: install-cni-plugins
+      - matchRegex:
+          path: spec.template.spec.initContainers[1].command[2]
+          pattern: "(?m)^\\s*trap 'rm -rf /host/opt/cni/bin/\\.tmp-cozystack-multus-\\* \\|\\| :' EXIT\\s*$"
+      # The plugins live only in the cozystack-built image. Regenerated with the
+      # upstream one, /cni-plugins is absent and the script SKIPS, staging
+      # nothing on any node.
+      - matchRegex:
+          path: spec.template.spec.initContainers[1].image
+          pattern: '^ghcr\.io/cozystack/cozystack/multus-cni:'
+      # The script is an argv string in the file the package's `image:` target
+      # rewrites with `sed s|image: .*multus-cni.*|...|g`, so a line inside it
+      # carrying both tokens would be replaced by an image reference.
+      - notMatchRegex:
+          path: spec.template.spec.initContainers[1].command[2]
+          pattern: 'image:'

packages/system/multus/values.yaml

@@ -0,0 +1,8 @@
+## @param stageCniPlugins install the reference CNI plugins bundled in the multus image into the node's /opt/cni/bin
+## Set this through the platform, not here: networking.stageCniPlugins on the cozystack-platform
+## Package resolves the per-variant default. See packages/core/platform/values.yaml.
+##
+## Accepted spellings are true/yes/on/1 and false/no/off/0, in any case; anything else stops the
+## render rather than resolving to off.
+##
+stageCniPlugins: false