cozystack/docs/gpu-vgpu.md

GPU Operator: passthrough and vGPU

This document describes how to configure the GPU Operator package with NVIDIA vGPU support so that a single physical GPU can be sliced and shared across multiple virtual machines.

Last verified: 2026-04-29 against KubeVirt main (virt-handler nightly 20260429_74d7c52588) + the vgpu variant + NVIDIA vGPU 20.0 host driver 595.58.02 + GRID guest driver 595.58.03.

Two driver models

NVIDIA's vGPU driver uses two different host-side models depending on GPU generation:

• Mediated devices (mdev) — Pascal / Volta / Turing / Ampere up to A100 / A30. The driver creates mdev parent devices under /sys/class/mdev_bus/; KubeVirt advertises them via permittedHostDevices.mediatedDevices.
• SR-IOV with per-VF sysfs — Ada Lovelace (L4, L40, L40S, …) and Blackwell (B100, …) on the vGPU 17/20 driver branch. The driver creates SR-IOV virtual functions; profile selection happens via /sys/bus/pci/devices/<VF>/nvidia/current_vgpu_type. KubeVirt advertises VFs via permittedHostDevices.pciHostDevices after kubevirt/kubevirt#16890.

This guide focuses on the SR-IOV path, which is the only model NVIDIA supports for current data-centre GPUs. Mdev is mentioned for completeness; for Pascal–Ampere refer to the upstream NVIDIA GPU Operator docs.

Prerequisites

• An Ada Lovelace (or newer) NVIDIA GPU that supports SR-IOV vGPU (L4, L40, L40S, etc.).
• Ubuntu 24.04 host OS. Older Ubuntu releases also work if the upstream gpu-driver-container repo has a matching vgpu-manager/ Dockerfile. Talos Linux is not recommended for vGPU. NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero closed siderolabs/extensions#461 noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that includes the driver in-tree is therefore not feasible without a private fork that violates the EULA.
• KubeVirt v1.9.0 or newer, which is the release the platform bundles. kubevirt/kubevirt#16890 ("vGPU: SRIOV support") shipped in v1.9.0; v1.8.x and older do not include it and backports are not planned.
• An NVIDIA vGPU Software / NVIDIA AI Enterprise subscription (the .run is not redistributable).
• A reachable NVIDIA Delegated License Service (DLS) instance and a matching client_configuration_token.tok file.

Variants

The gpu-operator package exposes three variants. This document is vGPU-focused; the variant inventory is shared.

• default — passthrough mode (vfio-pci). Whole GPU goes to a single VM. Talos is supported here; the kernel module is the open-source vfio-pci, no proprietary driver is needed on the host.
• vgpu — SR-IOV vGPU mode. One physical GPU is sliced into multiple VFs, each VF bound to a vGPU profile that the guest sees as its own GPU.
• container — containerized GPU workloads (CUDA pods, ML training) via the standard NVIDIA device plugin on hosts that already provide both the NVIDIA driver and nvidia-container-toolkit (the typical apt-installed Ubuntu/Debian shape). Sandbox workloads are off, devicePlugin is on, driver / toolkit / vfioManager / cdi are off so that the operator does not fight the host install. Orthogonal to the two VM variants — it does not pass GPUs to KubeVirt VMs. Note that apt install nvidia-container-toolkit installs binaries only — it does not configure containerd. Because this variant disables the operator's toolkit component (which would normally do that wiring), the host must additionally have run nvidia-ctk runtime configure --runtime=containerd (followed by a containerd restart) and exposed the nvidia runtime as the default or via a RuntimeClass before the device plugin can serve GPUs.

Passthrough variant: host preparation

Last verified: 2026-05-28 against cozystack.gpu-operator chart gpu-operator v26.3.1 + image nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0 (distroless image with a Go ELF binary /usr/bin/driver-manager, entrypoint ["driver-manager", "preflight_check"]; verified via crane export … | tar -x usr/bin/driver-manager and strings over the resulting binary) + Ubuntu 24.04 host with nvidia-driver-580-open 580.82.07 pre-installed. Tracks #2763.

The default (passthrough) variant assumes the GPU is owned by the host kernel's vfio-pci driver and nothing else.

How the operator detects a pre-installed host driver. The chart's vfio-manager DaemonSet runs the upstream NVIDIA k8s-driver-manager init container with the uninstall_driver subcommand; that path calls the Go method (*DriverManager).isHostDriver, which runs chroot /host nvidia-smi --query-gpu=driver_version --format=csv,noheader and treats any non-empty stdout as "host driver present" (file-existence is not pre-tested — if nvidia-smi is missing, the chroot exec errors and isHostDriver returns false, which is the intended path on a clean host: the operator then proceeds with the uninstall flow and vfio-manager binds vfio-pci as designed). On a positive detection the binary logs Host driver detected: <ver>, labels the node nvidia.com/gpu.deploy.driver=pre-installed, exits, and nvidia-sandbox-validator then crashloops with device not bound to 'vfio-pci'.

FORCE_REINSTALL does not bypass this. k8s-driver-manager v0.10.0 exposes a FORCE_REINSTALL / --force-reinstall env+flag pair (visible in the binary's strings table), but it gates a later "same-config already loaded" branch inside uninstallDriver, not the isHostDriver short-circuit at the top — operators who set it and expect a bypass will report a false bug. There is currently no opt-out for the isHostDriver guard itself.

Current mitigation. The only mitigation that keeps VM passthrough working is to remove the host NVIDIA stack before enabling the variant (the "Clean-host workaround" section below).

Alternatives and scope. The container variant of cozystack.gpu-operator is the no-purge path: it keeps the host driver and exposes GPUs to pods rather than VMs, so it does not replace passthrough for anyone who actually needs a GPU inside a VM. Talos is unaffected because the Talos image ships only the vfio-pci extension; this section applies to Linux distributions where you installed the host driver yourself (typically Ubuntu / Debian / RHEL with apt install nvidia-driver-* or equivalent). See also packages/system/gpu-operator/examples/README.md for the Talos native-pod-workload reference flow.

Symptom

The first thing you see is kubectl get pods -n cozy-gpu-operator showing nvidia-vfio-manager-* stuck in Init:Error / Init:CrashLoopBackOff — the init container exits non-zero after detecting the host driver, so kubelet keeps restarting it.

Its log shows it skipped the bind step:

Host driver detected: 580.82.07
NVIDIA GPU driver is already pre-installed on the node,
  disabling the containerized driver
Labeling node <NODE> with nvidia.com/gpu.deploy.driver=pre-installed

nvidia-sandbox-validator then crashloops:

Error: error validating vfio-pci driver installation:
  device not bound to 'vfio-pci'; device: 0000:18:00.0 driver: 'nvidia'

lspci -nnk -d 10de: still shows Kernel driver in use: nvidia on every target GPU, the node carries nvidia.com/gpu.deploy.driver=pre-installed, and kubectl get node <NODE> -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com/")))' reports {} — no GPU resource was registered.

Clean-host workaround

Purge the NVIDIA host stack and blacklist the kernel modules so the host never re-claims the GPU. Two pitfalls to avoid: apt autoremove is dangerous here because the nvidia- prefix is shared with NVIDIA DOCA / Mellanox / InfiniBand userspace, so a blanket autoremove can take RDMA out on a converged GPU + RDMA host; and a hardcoded apt purge 'nvidia-*' 'cuda-*' pattern list is fragile — apt treats * as a cache-wide regex and aborts the entire transaction, purging nothing, if any pattern matches nothing in the cache (e.g. cuda-* on a host without NVIDIA's CUDA repo). Build the list from what is actually installed instead:

# dpkg-query patterns are true globs over INSTALLED packages: no
# zero-match abort (unlike apt's cache-wide regex) and no accidental
# substring over-match. List first, then review before purging.
dpkg-query -W -f '${Package}\n' 'nvidia-*' 'libnvidia-*' 'cuda-*' 2>/dev/null

Review the list before purging:

• nvidia-dkms-* and nvidia-kernel-* are the load-bearing kernel pieces — without removing them DKMS rebuilds nvidia.ko on the next reboot and the blacklist below is bypassed by any explicit modprobe.
• On a converged GPU + RDMA host, drop any libnvidia-* that belong to NVIDIA DOCA / Mellanox OFED — purging those breaks RDMA.
• If the list is empty, the driver was installed with NVIDIA's .run installer rather than apt — run sudo nvidia-uninstall instead of the purge below.

Then purge the reviewed list, blacklist the modules, and rebuild the initramfs:

# Replace with the packages you kept from the list above.
sudo apt purge nvidia-driver-580-open nvidia-dkms-580-open <...>

sudo tee /etc/modprobe.d/blacklist-nvidia.conf > /dev/null <<'EOF'
blacklist nouveau
blacklist nvidia
blacklist nvidia_drm
blacklist nvidia_modeset
blacklist nvidia_uvm
blacklist nvidia_peermem
EOF

# -k all rebuilds EVERY installed kernel's initramfs (plain -u touches
# only the running kernel), so a just-upgraded kernel also boots with
# the blacklist and cannot re-claim the GPU.
sudo update-initramfs -u -k all
sudo reboot

After the reboot, confirm the host is clean. The init container exits non-zero after detecting a host driver, so both DaemonSet pods sit in Init:CrashLoopBackOff and will retry on their own — deleting them just skips the up-to-5-minute backoff window:

# Should print nothing.
lsmod | grep -E '^(nvidia|nouveau)'

# command -v matches what isHostDriver does (PATH lookup inside the
# chroot), so it also catches /usr/local/bin/nvidia-smi left by a .run
# or CUDA-toolkit install. Prints "ok: gone" on success.
command -v nvidia-smi >/dev/null && echo "STILL PRESENT — purge incomplete" || echo "ok: gone"

# A leftover DKMS module rebuilds nvidia.ko on the next kernel update
# and an explicit modprobe bypasses the blacklist — should print nothing.
dkms status | grep -i nvidia

# Skip the CrashLoopBackOff backoff window by deleting the stuck pods.
# The DaemonSet labels are operator-managed and not stable across
# gpu-operator versions, so delete by name pattern — anchored so the
# match is the two DaemonSets and nothing else.
kubectl -n cozy-gpu-operator get pods -o name \
  | grep -E '^pod/(nvidia-vfio-manager|nvidia-sandbox-validator)-' \
  | xargs -r kubectl -n cozy-gpu-operator delete

Within a couple of minutes vfio-manager should bind every target GPU to vfio-pci and the node's allocatable will gain the registered resource:

lspci -nnk -d 10de: | grep 'Kernel driver in use'
# Kernel driver in use: vfio-pci

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable}{"\n"}{end}' | grep nvidia.com

One label is not cleaned up automatically. The init container set nvidia.com/gpu.deploy.driver=pre-installed, and the operator's success path restores only the operand labels — it never resets this one (rescheduleGPUOperatorComponents in k8s-driver-manager v0.10.0 touches the validator / toolkit / device-plugin / etc. labels, not deploy.driver). So the node keeps pre-installed indefinitely: it is the same label the Symptom section uses as evidence, and it would disable the containerized-driver DaemonSet if the node later switches to the container workload. Clear it once recovery is confirmed:

# Nothing on the success path resets this label; clear it so it does not
# mislead future debugging or block a later container-workload switch.
kubectl label node <NODE> nvidia.com/gpu.deploy.driver-

Known limitation

The skip-on-pre-installed behavior lives in the upstream NVIDIA/k8s-driver-manager Go binary at cmd/driver-manager/main.go: the method (*DriverManager).isHostDriver is called from (*DriverManager).uninstallDriver and has no in-band opt-out in :v0.10.0. FORCE_REINSTALL exists as an env / CLI flag in the same binary but gates the later "same-config already loaded" branch inside uninstallDriver, not the isHostDriver short-circuit. Hosts that need to keep the NVIDIA host driver installed for non-Kubernetes workloads cannot currently share the same GPU with the passthrough variant. Two mitigation paths:

• container variant — a third variant of cozystack.gpu-operator that targets the apt-installed-driver host shape and exposes GPUs to pods (not VMs) without unbinding the host driver. Available now, and selectable from platform values with bundles.iaas.gpuOperatorVariant: container. It solves the host-driver conflict by giving up VM passthrough, so it is a mitigation only for workloads that can run in a pod.
• Upstream override — an env-var override of isHostDriver is the only structural fix that would let the passthrough variant coexist with a host driver. Not available today; requested in NVIDIA/k8s-driver-manager#191.

Building the vGPU Manager image

The proprietary vGPU Manager driver must be obtained from NVIDIA and packaged into a container image that the gpu-operator chart pulls — it is not installed from a raw .run at runtime. NVIDIA owns this build path; their gpu-driver-container repository ships per-OS Dockerfiles under vgpu-manager/<os>/ and is the source of truth for build args, base images, and supported OS releases. Follow the README in that repository.

The proprietary .run is the Linux KVM variant (not the Ubuntu KVM .deb, which ships pre-built modules for stock kernels only). It comes from the NVIDIA Licensing Portal under an NVIDIA AI Enterprise / vGPU subscription.

EULA: never push the resulting image to a publicly readable registry. Use a private registry (in-cluster Harbor works well as a non-proxy project).

Deploying with the vgpu variant

The platform's iaas bundle deploys the gpu-operator Package CR when cozystack.gpu-operator is in bundles.enabledPackages and bundles.iaas.gpuOperatorVariant: vgpu is set. The vGPU Manager image is proprietary and not redistributable, so the bundle does not ship a default tag — the operator builds the container per the upstream gpu-driver-container recipe and supplies the private-registry coordinates through platform values:

bundles:
  iaas:
    enabled: true
    gpuOperatorVariant: vgpu
  enabledPackages:
  - cozystack.gpu-operator

gpu:
  vgpuManager:
    repository: registry.example.com/nvidia
    image: vgpu-manager
    version: "595.58.02-ubuntu24.04"
    # imagePullSecrets lives per-component (vgpuManager, driver,
    # validator, dcgmExporter, …). The value is a list of strings,
    # not [{name: ...}].
    imagePullSecrets:
    - nvidia-registry-secret

The platform forwards gpu.vgpuManager into the emitted gpu-operator Package CR's components.gpu-operator.values.gpu-operator.vgpuManager, so the bundle handles the variant + image coordinates in one place. If you need to override anything else on the gpu-operator chart (driver, validator, dcgmExporter, custom node selectors), hand-craft a Package CR named cozystack.gpu-operator with the full components.gpu-operator.values block and put that name in bundles.disabledPackages. A hand-written Package CR does not take precedence over the bundle: as long as the platform emits its own copy of that package, the next platform render replaces whatever you wrote.

The nvidia-registry-secret should be a docker-registry Secret created beforehand in cozy-gpu-operator.

Verify the DaemonSet is running and nvidia.ko loads on every GPU node:

kubectl -n cozy-gpu-operator get pods -l app=nvidia-vgpu-manager-daemonset
kubectl -n cozy-gpu-operator exec -it <pod> -- nvidia-smi

nvidia-smi should enumerate the physical GPUs and report Host VGPU Mode : SR-IOV.

Profile assignment (SR-IOV path)

The vgpu variant is experimental on Ada+ and ships without a profile-assignment loop. NVIDIA's vgpu-device-manager walks /sys/class/mdev_bus/, which does not exist on Ada+ — the DaemonSet errors with "no parent devices found for GPU at index '0'" and is therefore disabled by default in values-vgpu.yaml. Until an SR-IOV-aware controller is shipped, profile assignment is an out-of-band step that must be re-applied after every node reboot (current_vgpu_type resets to 0 on PCIe re-enumeration). Without this step, permittedHostDevices.pciHostDevices will report zero allocatable resources and no VM can request the vGPU. Do not deploy the vgpu variant in production until you have an automated profile-assignment mechanism in place — typically a small DaemonSet that reads a ConfigMap (<bus-id> = <profile-id>) and writes the corresponding current_vgpu_type files at boot.

Once nvidia.ko is loaded the driver enables SR-IOV (16 VFs per L40S by default). Each VF needs a vGPU profile written to its sysfs:

# from inside the nvidia-vgpu-manager-daemonset pod (privileged, hostPID)
echo 1155 > /sys/bus/pci/devices/0000:02:00.5/nvidia/current_vgpu_type

The numeric profile ID can be discovered per-VF:

cat /sys/bus/pci/devices/0000:02:00.5/nvidia/creatable_vgpu_types

For Pascal–Ampere GPUs (V100, T4, A100, A30) the mdev model still applies. Flip vgpuDeviceManager.enabled: true in your Package CR overrides — NVIDIA's device manager works correctly there.

KubeVirt configuration

When cozystack.gpu-operator is in bundles.enabledPackages (and not also in bundles.disabledPackages), the platform mirrors the chosen GPU variant into the KubeVirt CR automatically. There is no manual kubectl patch step. This applies to the two VM variants only: container serves GPUs to pods and never to VMs, so selecting it emits the gpu-operator Package with no host-device wiring on the KubeVirt side.

If you opt out of bundle management and hand-craft a cozystack.gpu-operator Package CR directly — typically to apply overrides the bundle does not expose (driver settings, custom node selectors, validator / dcgmExporter tweaks, etc.) — the platform does NOT auto-wire HostDevices or permittedHostDevices into the KubeVirt CR. In that flow you also hand-craft a cozystack.kubevirt Package CR with components.kubevirt.values.extraFeatureGates: [HostDevices] and the appropriate permittedHostDevices block. The escape-hatch values shape under .gpu (below) is intentionally documented in the bundle-managed flow only. Both hand-crafted Package CRs need their names in bundles.disabledPackages: the iaas bundle emits cozystack.kubevirt on every render and cozystack.gpu-operator on every render that has it in bundles.enabledPackages, and that render replaces the hand-written spec rather than merging with it.

• developerConfiguration.featureGates gets HostDevices appended (current KubeVirt splits this from the GPU gate; the admission webhook rejects spec.template.spec.domain.devices.hostDevices without it).
• permittedHostDevices.pciHostDevices is filled from packages/core/platform/files/gpu-passthrough-defaults.yaml when bundles.iaas.gpuOperatorVariant: default (the package default) and gpu.replaceDefaults is left at false. The table covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4), Volta (V100/V100S). All entries carry externalResourceProvider: true because the resource names come from nvidia-sandbox-device-plugin, not from KubeVirt's in-tree device plugin.
• permittedHostDevices.mediatedDevices is filled from packages/core/platform/files/gpu-vgpu-defaults.yaml when bundles.iaas.gpuOperatorVariant: vgpu and gpu.vgpuDeviceManager.enabled: true — that knob defaults to false, so a stock vgpu cluster ships no mdev table at all (see the Ada Lovelace / Blackwell note below for why). gpu.replaceDefaults: true also suppresses it. This list only EXPOSES, by profile name (mdevNameSelector), mdevs that the GPU Operator's vGPU Device Manager CREATES on the node; the platform does not ship a numeric mediatedDevicesConfiguration default (those nvidia-NNN type ids are per-SKU/driver sysfs indices with no portable value — set .gpu.mediatedDevicesConfiguration yourself, with host-verified ids, only if you want KubeVirt rather than the Device Manager to create mdevs). The starter set covers Pascal–Ampere mdev profiles (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family range as the upstream vgpu-device-manager walks /sys/class/mdev_bus/ for. Ada Lovelace / Blackwell SR-IOV vGPU is out of scope for the chart's default list; advertise those VFs via the user-override hook below.

Extending or replacing the default table

The platform exposes three knobs under .gpu:

gpu:
  # Extend the platform defaults with cluster-specific entries. Read by
  # the two VM variants only; the container variant reads neither key.
  # pciHostDevices is read in both of them — it feeds the passthrough
  # (vfio-pci) path AND the post-kubevirt#16890 SR-IOV vGPU VF path on
  # Ada Lovelace / Blackwell. mediatedDevices is read in vgpu alone, for
  # the pre-#16890 mdev path on Pascal–Ampere; the passthrough variant
  # ignores it. Both render into the same KubeVirt CR.
  permittedHostDevices:
    pciHostDevices:
    - pciVendorSelector: "10DE:26B9"   # L40S, advertised as a VF for SR-IOV vGPU
      resourceName: nvidia.com/L40S-24Q
      # externalResourceProvider is intentionally omitted here: after
      # kubevirt/kubevirt#16890, virt-handler's in-tree device plugin
      # advertises the resource directly, no sandbox plugin in the loop.
    mediatedDevices: []
  # mediatedDevicesConfiguration makes KubeVirt itself create mdevs (vgpu mode). No platform default: mdev creation is normally delegated to the vGPU Device Manager (name-based), and these mediatedDeviceTypes are host/driver-specific nvidia-NNN sysfs indices (look yours up via /sys/bus/pci/devices/<BDF>/mdev_supported_types/*/name). Set this only to opt into KubeVirt-driven creation; mergeOverwrite REPLACES a supplied top-level key wholesale.
  mediatedDevicesConfiguration: {}
  # Wipe the platform defaults entirely and ship only the cluster's
  # curated lists. Useful for non-NVIDIA-only clusters and strict
  # allowlist requirements.
  replaceDefaults: false

replaceDefaults: false (the default) appends user entries to the NVIDIA defaults. replaceDefaults: true drops the NVIDIA table entirely — if you don't then supply your own pciHostDevices / mediatedDevices list, the rendered KubeVirt CR has no permittedHostDevices block and the admission webhook will reject every GPU VM.

Notes on nvidia-sandbox-device-plugin resource names

The resourceName strings in gpu-passthrough-defaults.yaml are what nvidia-sandbox-device-plugin (nvcr.io/nvidia/kubevirt-gpu-device-plugin) advertises: it derives each slug mechanically from the device's PCI-IDs database name by uppercasing it, turning /, . and whitespace into _, and stripping the remaining non-alphanumerics (the [ / ]). So TU104GL [Tesla T4] becomes nvidia.com/TU104GL_TESLA_T4 and GA100GL [A30 PCIe] becomes nvidia.com/GA100GL_A30_PCIE — the slug carries every token the PCI-IDs string holds (the GL die suffix, the Tesla brand on Turing/Volta, form factor, memory), not a tidy <arch>_<model>. The names track the pci.ids snapshot bundled in the plugin image, so a different plugin build can publish a different string — check with kubectl describe node <node> | grep nvidia.com/ and override via .gpu.permittedHostDevices.pciHostDevices (or wipe the table with replaceDefaults: true and curate it yourself). PCI vendor:device IDs themselves are stable across driver versions.

SR-IOV PF vs VF (Ada Lovelace and newer)

On L40S (and other Ada-Lovelace cards) the SR-IOV VFs report the same PCI device ID as the PF — lspci -nn -d 10de: on the host shows both as [10de:26b9]. virt-handler distinguishes them by is-VF + has-vGPU-profile, so a single pciVendorSelector matches the right set. Verify on your specific GPU before assuming this — some other generations split PF/VF IDs.

externalResourceProvider: true is not required when the resource is advertised by virt-handler's in-tree device plugin (the SR-IOV path post kubevirt#16890). The platform passthrough defaults include the flag because that path is driven by the external sandbox plugin.

Verifying allocatable capacity

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable.nvidia\.com/L40S-24Q}{"\n"}{end}'

Licensing (DLS)

vGPU 17/20 uses the NVIDIA Delegated License Service. The legacy ServerAddress= / ServerPort=7070 lines in gridd.conf are no longer authoritative — nvidia-gridd (running inside the guest) reads the DLS endpoint from the ClientConfigToken file directly.

The host vGPU Manager DaemonSet does not request a license — it only enables SR-IOV and loads nvidia.ko. Licensing is consumed entirely by the guest. The gpu-operator chart's driver.licensingConfig.secretName would mount the Secret into the driver pod on the host, where it has no effect for SR-IOV vGPU; do not wire the licensing Secret through it.

Instead, deliver the token and gridd.conf to the guest via cloud-init or a containerDisk overlay:

# inside the VirtualMachine cloudInitNoCloud userData
write_files:
- path: /etc/nvidia/ClientConfigToken/client_configuration_token.tok
  # 0744 follows NVIDIA's recommendation in the Virtual GPU Software
  # Licensing User Guide ("Configuring a Licensed Client on Linux"):
  # nvidia-gridd does not necessarily run as the file owner.
  # https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/
  permissions: '0744'
  encoding: b64
  content: <base64 token>
- path: /etc/nvidia/gridd.conf
  permissions: '0644'
  content: |
    # FeatureType selects which vGPU Software license the guest requests.
    # 0 — unlicensed state (no license requested; Q profiles run in
    #     reduced mode after the grace period).
    # 1 — NVIDIA vGPU. The driver auto-selects the correct license
    #     type from the configured vGPU profile (Q → vWS, B → vPC,
    #     A → vCS / Compute). Use this for SR-IOV vGPU profiles.
    # 2 — explicitly NVIDIA RTX Virtual Workstation.
    # 4 — explicitly NVIDIA Virtual Compute Server.
    FeatureType=1

Verify activation inside the guest:

nvidia-smi -q | grep 'License Status'
# License Status   : Licensed

If the guest reports Unlicensed (Unrestricted) for more than a couple of minutes, check journalctl _COMM=nvidia-gridd for handshake errors against the DLS endpoint baked into the token.

Migrating from chart v25.x

Operators upgrading from the previous Cozystack release (gpu-operator chart v25.3.0) should also note that the upstream chart deprecated driver.licensingConfig.configMapName in favour of driver.licensingConfig.secretName. The old key still works but emits a deprecation warning at render time. If your existing Package CR set the licensing reference via configMapName, switch it to secretName on this upgrade — the Secret content (gridd.conf and the ClientConfigToken) does not need to change. This applies to passthrough deployments that drove host-side licensing through the gpu-operator chart; SR-IOV vGPU does not consume the host-side licensing knob at all (see "Licensing (DLS)" above).

Sample VirtualMachine

Either hostDevices or gpus accepts the resource (the upstream KubeVirt API resolves both PCI and mediated-device pools), but the convention is to use hostDevices for VF-style PCI passthrough:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vgpu-smoke
  namespace: tenant-example
spec:
  runStrategy: Always
  template:
    spec:
      domain:
        cpu:
          cores: 4
        memory:
          guest: 8Gi
        devices:
          disks:
          - name: rootdisk
            disk:
              bus: virtio
          interfaces:
          - name: default
            masquerade: {}
          hostDevices:
          - name: gpu0
            deviceName: nvidia.com/L40S-24Q
      networks:
      - name: default
        pod: {}
      volumes:
      - name: rootdisk
        # A 2.4 GiB containerDisk overlay is too small to install
        # the GRID guest driver in-place. Use a CDI DataVolume of
        # 20 GiB+ in production.
        containerDisk:
          image: quay.io/containerdisks/ubuntu:24.04

Inside the guest, install the GRID driver from the .run (the GUEST .run, distinct from the host vgpu-kvm package), then nvidia-smi should report the configured profile:

| 0  NVIDIA L40S-24Q                Off |   00000000:0E:00.0 Off |                    0 |
|        17 MiB / 24576 MiB    P0    Default                                                |

Profile reference (L40S)

L40S supports the full Q (RTX vWS), B (vPC), A (vCS / Compute) profile families. The numeric IDs come from the driver and are visible in creatable_vgpu_types:

Profile	Frame Buffer	Max instances per L40S	Use case
L40S-1Q	1 GB	48	Light 3D / VDI
L40S-2Q	2 GB	24	Medium 3D / VDI
L40S-4Q	4 GB	12	Heavy 3D / VDI
L40S-6Q	6 GB	8	Professional 3D
L40S-8Q	8 GB	6	AI / ML inference
L40S-12Q	12 GB	4	AI / ML training
L40S-24Q	24 GB	2	Large AI workloads
L40S-48Q	48 GB	1	Full GPU equivalent

Other GPU families have analogous tables in the NVIDIA Virtual GPU Software Documentation.

OS support summary

The container variant column assumes the host already ships the NVIDIA driver and nvidia-container-toolkit via the distro package manager, with the nvidia runtime registered in containerd (nvidia-ctk runtime configure --runtime=containerd). With driver.enabled=false the operator uses the pre-installed host driver at its standard location, so a stock apt install needs no hostPaths.driverInstallDir override. Talos installs the driver under a non-standard prefix, so the operator does not find it at the default location — see packages/system/gpu-operator/examples/ for the Talos-specific path with a compat DaemonSet and an explicit hostPaths.driverInstallDir override.

Host OS	passthrough (default)	vGPU (vgpu)	container (container)
Ubuntu 24.04	⚠️ supported upstream, but the host must be clean of any apt-installed NVIDIA driver — see "Passthrough variant: host preparation"	✅ supported upstream (vgpu-manager/ubuntu24.04)	✅ apt-installed driver + nvidia-container-toolkit
Ubuntu 22.04	⚠️ same clean-host requirement as 24.04	✅	✅
Ubuntu 20.04	⚠️ same clean-host requirement as 24.04	✅	✅
Ubuntu 26.04	⚠️ same clean-host requirement as 24.04, plus a nvidia-driver patch for usr-merge (details pending)	⚠️ same patch + own Dockerfile fork	✅
Talos Linux	✅ (open vfio-pci; the Talos image ships no host NVIDIA stack, so the clean-host check passes trivially)	❌ NVIDIA does not grant redistribution rights for the proprietary .run; we tried and the path is blocked	⚠️ host driver lands in a non-standard prefix — use examples/values-native-talos.yaml (compat DaemonSet + hostPaths.driverInstallDir override) as a starting point instead