cozystack/docs/vm-external-vlan.md

Attaching a virtual machine to an external VLAN

This document describes how to attach a Cozystack virtual machine (the vm-instance application) directly to an external, physically-routed VLAN — the layer-2 segment a VM needs when it must appear on the same broadcast domain as external hardware (a licensing appliance, a storage box, a gateway managed outside the cluster), with an address from that VLAN's subnet rather than from the cluster overlay.

The default Cozystack VM networking is overlay-only (the pod network, plus optional KubeOVN VPC subnets). Bridging a VM onto a real VLAN is a different pattern and has one non-obvious constraint: it works with a Linux bridge and the bridge CNI plugin, and it does not work with macvlan. The rest of this guide explains why and gives a working recipe.

Why bridge and not macvlan

KubeVirt attaches a VM interface to a secondary network using bridge binding by default (this is what the vm-instance chart emits for every network in .spec.networks). With bridge binding the guest's own MAC address is placed on the wire — the launcher pod does not masquerade or translate it.

A macvlan attachment is incompatible with that model. macvlan demultiplexes inbound frames strictly by the MAC address of the macvlan child interface. Because KubeVirt puts the guest's MAC on the wire — not the macvlan child's — replies from the gateway or other hosts arrive at the parent interface addressed to the guest MAC, do not match any macvlan child, and are silently dropped before they ever reach the VM. The symptom is a guest that can transmit (ARP requests and pings leave, visible in tcpdump on the parent interface) but never receives a reply (its neighbor entry for the gateway stays FAILED). As a secondary consequence, the host cannot reach macvlan children through the parent interface either, so a host-side service on the parent IP is unreachable from the VMs.

A Linux bridge does not have this limitation: it forwards by learned MAC on all bridged ports, so the guest MAC is reachable, and the host can carry an address on the bridge itself to talk to the VMs. Attach the VLAN sub-interface to a bridge and point a bridge-type NAD at it.

Overview

Three pieces cooperate:

1. A Linux bridge on each node that enslaves the tagged VLAN sub-interface. This is node-level networking — it is configured by your node provisioning (netplan / Talos machine config / systemd-networkd), not by a Cozystack chart.
2. A NetworkAttachmentDefinition of type bridge referencing that bridge, created in the VM's tenant namespace.
3. The vm-instance application referencing the NAD by name in .networks, with the guest's static address supplied through cloud-init.

Prerequisites

• The multus package is enabled (it provides the NetworkAttachmentDefinition CRD and the secondary-network plumbing).
• The bridge CNI plugin is present in /opt/cni/bin on every node. The multus package puts it there itself on every platform, which is the subject of the next section — read it before upgrading a cluster whose /opt/cni/bin you provision yourself. Verify with ls /opt/cni/bin/bridge; a missing binary makes the NAD fail with failed to find plugin "bridge" in path [/opt/cni/bin].
• There is no IPAM plugin in this path — addresses are assigned inside the guest, not by the CNI. Plan static addresses per VM.

Staging the reference CNI plugins

The multus package installs the reference plugins into /opt/cni/bin itself, on every platform, because no platform supplies all of them. That is the stageCniPlugins chart value, and the platform turns it on for every bundle. What differs is only how much is missing before staging runs. Talos puts six binaries in that directory — bridge, firewall, host-local, loopback and portmap from the same upstream release, plus flannel, which is a different project — so a NAD naming macvlan, ipvlan or vlan fails there too; staging adds the ten it does not carry and rewrites the four that overlap with the same upstream version (v1.9.1 on both sides), not an older one. Generic Linux offers no guarantee at all — k3s keeps its own copies under /var/lib/rancher/k3s/data/cni (/var/lib/rancher/k3s/data/current/bin on releases older than October 2024) and Cilium installs only cilium-cni and loopback, so /opt/cni/bin there can lack even bridge.

Carrying the plugins costs roughly 67 MiB uncompressed per architecture in the multus image, which every node pulls whether staging is on or off. Size airgap mirrors accordingly.

Where staging is on, it tries to replace any file of the same name already in that directory, every time the Multus pod is recreated, so a hand-placed or locally patched bridge will normally not survive — carry such a change in the multus image instead. A plugin it fails to write is reported and skipped rather than removed, so whatever was there stays — an older copy where the node had one, and nothing at all where it did not. It installs 14 of the upstream reference plugins and leaves loopback, dhcp, dummy and tap alone. Leaving them alone is not a promise that something else provides them: loopback is installed by Cilium and kube-ovn, while dhcp, dummy and tap are simply not staged — dhcp needs a host daemon this package does not run, and the other two have no consumer here. If you need one of those, put it on the node yourself.

If you provision /opt/cni/bin yourself, set the opt-out before upgrading. The replacement happens as soon as Multus rolls, and turning stageCniPlugins off afterwards does not undo it: nothing in this package removes or restores a plugin, so the copies stay until you re-provision the node. Operators pinned to a particular plugin version should decline first and upgrade second. This applies on Talos too, where four of the names staging writes are ones the node image also carries. Declining looks like this, on the platform component of the cozystack.cozystack-platform Package CR:

apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
  name: cozystack.cozystack-platform
spec:
  components:
    platform:
      values:
        networking:
          stageCniPlugins: false

Editing the multus HelmRelease or the child Package cozystack.multus directly does not hold: both are re-rendered by the platform reconcile. An unreadable value there — a spelling that is neither a true nor a false form — stops the whole platform render rather than resolving to off, so a typo in this key halts reconciliation cluster-wide.

If bridge is missing on a node where staging is on, read the log with kubectl --namespace cozy-multus logs <multus-pod> --container install-cni-plugins. The line no /cni-plugins in this image; skipping staging means the running multus image predates the plugin staging and installed nothing: the image reference and the manifest are re-pinned on different schedules, so a tree taken between a Dockerfile change and the next release bake can carry one without the other. Move to a release whose multus image carries the plugins. A line beginning failed to install: names plugins the container could not write, one of which may be the bridge you are looking for. The reason for each is earlier in the same log, printed as that plugin was attempted rather than next to the summary — so with more than one failure the causes are spread above it, not adjacent to it, and the one naming your plugin is the one to read. The container reports these and exits successfully on purpose, so the pod shows Ready and kubectl get will not point you here: failing it instead would leave the node unable to start any pod that goes through CNI, since Multus remains the primary CNI with no daemon behind it — host-network pods bypass CNI and still start. That means a NAD failing with failed to find plugin is the symptom to trust, and this log is where the reason is.

If the init container is absent from the Multus pod altogether, staging is off for this cluster. That means networking.stageCniPlugins was set to false somewhere: no bundle defaults it off, on Talos or anywhere else. Turn it back on with the same block as above and true in place of false, or stage the plugins through node provisioning instead.

Changing this value edits the Multus DaemonSet's pod template, so Multus rolls node by node. While a node's multus-daemon is down its CNI shim has no daemon behind it, and pod creation and deletion on that node fail for the length of the restart. Flip it during a window where that is acceptable, not while something else is rolling.

Step 1 — Linux bridge on the node

Create a bridge that enslaves the tagged VLAN sub-interface. The VLAN sub-interface itself carries no address; the bridge carries the host's presence on that VLAN (optional, but useful for a gateway-reachability sanity path and for any host-side service the VMs must reach).

This example uses netplan on an Ubuntu/Debian node; the VLAN id and subnet are illustrative (203.0.113.0/24, VLAN 100, gateway 203.0.113.1). Adapt to your uplink naming and to Talos/systemd-networkd if that is your provisioning:

network:
  version: 2
  vlans:
    # Tagged VLAN sub-interface, no address of its own — enslaved to the bridge.
    uplink.100:
      id: 100
      link: uplink
  bridges:
    br100:
      interfaces:
        - uplink.100
      # Optional host presence on the VLAN. Keep the node's default route on
      # its management interface — do not add a default route here.
      addresses:
        - 203.0.113.2/24

Notes:

• The node's default route must stay on the management interface. The bridge address (if any) is only for on-VLAN reachability, not a second default gateway.
• netplan apply cannot move an interface into a bridge while a consumer still holds it (for example a virt-launcher pod using a previous macvlan attachment). Remove the consumer first (delete the VMI so the launcher releases the interface), then reconfigure.
• After a reboot, systemd-networkd may briefly report the bridge "routable" while the link is not yet actually up. If your VMs need the VLAN immediately at boot, gate their start on a reachability check, or re-run netplan apply until the gateway answers.

Step 2 — NetworkAttachmentDefinition

Create a bridge-type NAD in the tenant namespace that will host the VM. The vm-instance chart resolves a network by name in the VM's own namespace, so one copy of the NAD must exist in every tenant namespace that runs VMs on this VLAN.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: vlan100
  namespace: tenant-example
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "bridge",
      "bridge": "br100",
      "ipam": {}
    }

• bridge must match the bridge name from step 1 (br100 here).
• ipam: {} — no cluster-side address assignment; the guest configures its address itself (step 3).

Step 3 — Attach the VM and assign a static address

Reference the NAD by name in the vm-instance values. Because the chart does not support networkData, the static address goes into cloud-init userData (cloudInit), written by the guest at first boot:

# vm-instance values
instanceType: u1.medium
instanceProfile: ubuntu
disks:
  - name: example-system
networks:
  - name: vlan100
cloudInit: |
  #cloud-config
  write_files:
    - path: /etc/netplan/60-vlan100.yaml
      permissions: "0600"
      content: |
        network:
          version: 2
          ethernets:
            # Match the second NIC (the pod-network NIC is the first). Use the
            # interface that comes up without a DHCP lease.
            enp2s0:
              addresses:
                - 203.0.113.10/24
  runcmd:
    - netplan apply

The VM ends up with two interfaces: the always-present pod-network NIC (default, used for cluster-internal traffic and for the vm-instance external-access features) and the VLAN NIC. The /24 address above brings up only the connected route for the VLAN subnet — it adds no default route, so the guest's egress stays wherever you want it (typically the pod NIC). If the VLAN is meant to be the guest's default gateway instead, add a default route under the VLAN NIC and remove it from the pod NIC.

Gotchas

• VMs are dual-homed. The vm-instance chart always adds the pod-network NIC in addition to any networks you declare; there is no single-homed (VLAN-only) option today. Address the VLAN NIC inside the guest and leave the pod NIC to the cluster.
• No networkData. The chart wires cloud-init through userData only, so in-guest static configuration (netplan write_files + netplan apply, as above) is the way to assign the VLAN address.
• MAC changes on VM re-creation. KubeVirt generates a fresh guest MAC each time the VM object is re-created, and vm-instance exposes no way to pin it, so re-creating a VM changes its MAC. The upstream gateway then holds a stale ARP entry for the old MAC for a few minutes, so "gateway unreachable" immediately after re-creating a VM is expected — wait for the ARP entry to age out (~5 min) rather than treating it as a fault.
• Host ↔ VM traffic. If the host must talk to the VMs (a proxy, a health check), give the bridge a host address on the VLAN (step 1) — traffic through a bare VLAN sub-interface to bridge-attached guests will not work the way macvlan users expect.

See also

• docs/gpu-vgpu.md — passing NVIDIA GPUs and vGPU profiles into the same VMs.
• KubeVirt user guide, Interfaces and Networks — bridge binding vs other binding methods.