# GPU Operator: passthrough and vGPU This document describes how to configure the GPU Operator package with NVIDIA vGPU support so that a single physical GPU can be sliced and shared across multiple virtual machines. **Last verified:** 2026-04-29 against KubeVirt `main` (`virt-handler` nightly `20260429_74d7c52588`) + the `vgpu` variant + NVIDIA vGPU 20.0 host driver `595.58.02` + GRID guest driver `595.58.03`. ## Two driver models NVIDIA's vGPU driver uses two different host-side models depending on GPU generation: - **Mediated devices (mdev)** — Pascal / Volta / Turing / Ampere up to A100 / A30. The driver creates `mdev` parent devices under `/sys/class/mdev_bus/`; KubeVirt advertises them via `permittedHostDevices.mediatedDevices`. - **SR-IOV with per-VF sysfs** — Ada Lovelace (L4, L40, L40S, …) and Blackwell (B100, …) on the vGPU 17/20 driver branch. The driver creates SR-IOV virtual functions; profile selection happens via `/sys/bus/pci/devices//nvidia/current_vgpu_type`. KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` after [kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890). This guide focuses on the **SR-IOV path**, which is the only model NVIDIA supports for current data-centre GPUs. Mdev is mentioned for completeness; for Pascal–Ampere refer to the upstream NVIDIA GPU Operator docs. ## Prerequisites - An Ada Lovelace (or newer) NVIDIA GPU that supports SR-IOV vGPU (L4, L40, L40S, etc.). - Ubuntu 24.04 host OS. Older Ubuntu releases also work if the upstream `gpu-driver-container` repo has a matching `vgpu-manager/` Dockerfile. **Talos Linux is not recommended** for vGPU. NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero [closed siderolabs/extensions#461](https://github.com/siderolabs/extensions/issues/461) noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that includes the driver in-tree is therefore not feasible without a private fork that violates the EULA. - KubeVirt v1.9.0 or newer, which is the release the platform bundles. [kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890) ("vGPU: SRIOV support") shipped in v1.9.0; v1.8.x and older do not include it and backports are not planned. - An NVIDIA vGPU Software / NVIDIA AI Enterprise subscription (the `.run` is not redistributable). - A reachable NVIDIA Delegated License Service (DLS) instance and a matching `client_configuration_token.tok` file. ## Variants The `gpu-operator` package exposes three variants. This document is vGPU-focused; the variant inventory is shared. - **`default`** — passthrough mode (`vfio-pci`). Whole GPU goes to a single VM. Talos is supported here; the kernel module is the open-source `vfio-pci`, no proprietary driver is needed on the host. - **`vgpu`** — SR-IOV vGPU mode. One physical GPU is sliced into multiple VFs, each VF bound to a vGPU profile that the guest sees as its own GPU. - **`container`** — containerized GPU workloads (CUDA pods, ML training) via the standard NVIDIA device plugin on hosts that already provide both the NVIDIA driver and `nvidia-container-toolkit` (the typical apt-installed Ubuntu/Debian shape). Sandbox workloads are off, `devicePlugin` is on, `driver` / `toolkit` / `vfioManager` / `cdi` are off so that the operator does not fight the host install. Orthogonal to the two VM variants — it does not pass GPUs to KubeVirt VMs. Note that `apt install nvidia-container-toolkit` installs binaries only — it does not configure containerd. Because this variant disables the operator's toolkit component (which would normally do that wiring), the host must additionally have run `nvidia-ctk runtime configure --runtime=containerd` (followed by a containerd restart) and exposed the `nvidia` runtime as the default or via a RuntimeClass before the device plugin can serve GPUs. ## Passthrough variant: host preparation **Last verified:** 2026-05-28 against `cozystack.gpu-operator` chart `gpu-operator` v26.3.1 + image `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0` (distroless image with a Go ELF binary `/usr/bin/driver-manager`, entrypoint `["driver-manager", "preflight_check"]`; verified via `crane export … | tar -x usr/bin/driver-manager` and `strings` over the resulting binary) + Ubuntu 24.04 host with `nvidia-driver-580-open` 580.82.07 pre-installed. Tracks [#2763](https://github.com/cozystack/cozystack/issues/2763). The `default` (passthrough) variant assumes the GPU is **owned by the host kernel's `vfio-pci` driver and nothing else**. **How the operator detects a pre-installed host driver.** The chart's `vfio-manager` DaemonSet runs the upstream NVIDIA `k8s-driver-manager` init container with the `uninstall_driver` subcommand; that path calls the Go method `(*DriverManager).isHostDriver`, which runs `chroot /host nvidia-smi --query-gpu=driver_version --format=csv,noheader` and treats any non-empty stdout as "host driver present" (file-existence is not pre-tested — if `nvidia-smi` is missing, the chroot exec errors and `isHostDriver` returns false, which is the intended path on a clean host: the operator then proceeds with the uninstall flow and `vfio-manager` binds `vfio-pci` as designed). On a positive detection the binary logs `Host driver detected: `, labels the node `nvidia.com/gpu.deploy.driver=pre-installed`, exits, and `nvidia-sandbox-validator` then crashloops with `device not bound to 'vfio-pci'`. **`FORCE_REINSTALL` does not bypass this.** `k8s-driver-manager` v0.10.0 exposes a `FORCE_REINSTALL` / `--force-reinstall` env+flag pair (visible in the binary's strings table), but it gates a later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit at the top — operators who set it and expect a bypass will report a false bug. There is currently no opt-out for the `isHostDriver` guard itself. **Current mitigation.** The only mitigation that keeps VM passthrough working is to remove the host NVIDIA stack before enabling the variant (the "Clean-host workaround" section below). **Alternatives and scope.** The `container` variant of `cozystack.gpu-operator` is the no-purge path: it keeps the host driver and exposes GPUs to pods rather than VMs, so it does not replace passthrough for anyone who actually needs a GPU inside a VM. Talos is unaffected because the Talos image ships only the `vfio-pci` extension; this section applies to Linux distributions where you installed the host driver yourself (typically Ubuntu / Debian / RHEL with `apt install nvidia-driver-*` or equivalent). See also [`packages/system/gpu-operator/examples/README.md`](../packages/system/gpu-operator/examples/README.md) for the Talos native-pod-workload reference flow. ### Symptom The first thing you see is `kubectl get pods -n cozy-gpu-operator` showing `nvidia-vfio-manager-*` stuck in `Init:Error` / `Init:CrashLoopBackOff` — the init container exits non-zero after detecting the host driver, so kubelet keeps restarting it. Its log shows it skipped the bind step: ```text Host driver detected: 580.82.07 NVIDIA GPU driver is already pre-installed on the node, disabling the containerized driver Labeling node with nvidia.com/gpu.deploy.driver=pre-installed ``` `nvidia-sandbox-validator` then crashloops: ```text Error: error validating vfio-pci driver installation: device not bound to 'vfio-pci'; device: 0000:18:00.0 driver: 'nvidia' ``` `lspci -nnk -d 10de:` still shows `Kernel driver in use: nvidia` on every target GPU, the node carries `nvidia.com/gpu.deploy.driver=pre-installed`, and `kubectl get node -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com/")))'` reports `{}` — no GPU resource was registered. ### Clean-host workaround Purge the NVIDIA host stack and blacklist the kernel modules so the host never re-claims the GPU. Two pitfalls to avoid: `apt autoremove` is dangerous here because the `nvidia-` prefix is shared with NVIDIA DOCA / Mellanox / InfiniBand userspace, so a blanket autoremove can take RDMA out on a converged GPU + RDMA host; and a hardcoded `apt purge 'nvidia-*' 'cuda-*'` pattern list is fragile — `apt` treats `*` as a cache-wide regex and **aborts the entire transaction, purging nothing**, if any pattern matches nothing in the cache (e.g. `cuda-*` on a host without NVIDIA's CUDA repo). Build the list from what is actually installed instead: ```bash # dpkg-query patterns are true globs over INSTALLED packages: no # zero-match abort (unlike apt's cache-wide regex) and no accidental # substring over-match. List first, then review before purging. dpkg-query -W -f '${Package}\n' 'nvidia-*' 'libnvidia-*' 'cuda-*' 2>/dev/null ``` Review the list before purging: - `nvidia-dkms-*` and `nvidia-kernel-*` are the load-bearing kernel pieces — without removing them DKMS rebuilds `nvidia.ko` on the next reboot and the blacklist below is bypassed by any explicit `modprobe`. - On a **converged GPU + RDMA host**, drop any `libnvidia-*` that belong to NVIDIA DOCA / Mellanox OFED — purging those breaks RDMA. - If the list is **empty**, the driver was installed with NVIDIA's `.run` installer rather than apt — run `sudo nvidia-uninstall` instead of the purge below. Then purge the reviewed list, blacklist the modules, and rebuild the initramfs: ```bash # Replace with the packages you kept from the list above. sudo apt purge nvidia-driver-580-open nvidia-dkms-580-open <...> sudo tee /etc/modprobe.d/blacklist-nvidia.conf > /dev/null <<'EOF' blacklist nouveau blacklist nvidia blacklist nvidia_drm blacklist nvidia_modeset blacklist nvidia_uvm blacklist nvidia_peermem EOF # -k all rebuilds EVERY installed kernel's initramfs (plain -u touches # only the running kernel), so a just-upgraded kernel also boots with # the blacklist and cannot re-claim the GPU. sudo update-initramfs -u -k all sudo reboot ``` After the reboot, confirm the host is clean. The init container exits non-zero after detecting a host driver, so both DaemonSet pods sit in `Init:CrashLoopBackOff` and **will retry on their own** — deleting them just skips the up-to-5-minute backoff window: ```bash # Should print nothing. lsmod | grep -E '^(nvidia|nouveau)' # command -v matches what isHostDriver does (PATH lookup inside the # chroot), so it also catches /usr/local/bin/nvidia-smi left by a .run # or CUDA-toolkit install. Prints "ok: gone" on success. command -v nvidia-smi >/dev/null && echo "STILL PRESENT — purge incomplete" || echo "ok: gone" # A leftover DKMS module rebuilds nvidia.ko on the next kernel update # and an explicit modprobe bypasses the blacklist — should print nothing. dkms status | grep -i nvidia # Skip the CrashLoopBackOff backoff window by deleting the stuck pods. # The DaemonSet labels are operator-managed and not stable across # gpu-operator versions, so delete by name pattern — anchored so the # match is the two DaemonSets and nothing else. kubectl -n cozy-gpu-operator get pods -o name \ | grep -E '^pod/(nvidia-vfio-manager|nvidia-sandbox-validator)-' \ | xargs -r kubectl -n cozy-gpu-operator delete ``` Within a couple of minutes `vfio-manager` should bind every target GPU to `vfio-pci` and the node's `allocatable` will gain the registered resource: ```bash lspci -nnk -d 10de: | grep 'Kernel driver in use' # Kernel driver in use: vfio-pci kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable}{"\n"}{end}' | grep nvidia.com ``` One label is **not** cleaned up automatically. The init container set `nvidia.com/gpu.deploy.driver=pre-installed`, and the operator's success path restores only the operand labels — it never resets this one (`rescheduleGPUOperatorComponents` in `k8s-driver-manager` v0.10.0 touches the validator / toolkit / device-plugin / etc. labels, not `deploy.driver`). So the node keeps `pre-installed` indefinitely: it is the same label the Symptom section uses as evidence, and it would disable the containerized-driver DaemonSet if the node later switches to the `container` workload. Clear it once recovery is confirmed: ```bash # Nothing on the success path resets this label; clear it so it does not # mislead future debugging or block a later container-workload switch. kubectl label node nvidia.com/gpu.deploy.driver- ``` ### Known limitation The skip-on-pre-installed behavior lives in the upstream [`NVIDIA/k8s-driver-manager`](https://github.com/NVIDIA/k8s-driver-manager) Go binary at `cmd/driver-manager/main.go`: the method `(*DriverManager).isHostDriver` is called from `(*DriverManager).uninstallDriver` and has no in-band opt-out in `:v0.10.0`. `FORCE_REINSTALL` exists as an env / CLI flag in the same binary but gates the later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit. Hosts that need to keep the NVIDIA host driver installed for non-Kubernetes workloads cannot currently share the same GPU with the passthrough variant. Two mitigation paths: - **`container` variant** — a third variant of `cozystack.gpu-operator` that targets the apt-installed-driver host shape and exposes GPUs to pods (not VMs) without unbinding the host driver. Available now, and selectable from platform values with `bundles.iaas.gpuOperatorVariant: container`. It solves the host-driver conflict by giving up VM passthrough, so it is a mitigation only for workloads that can run in a pod. - **Upstream override** — an env-var override of `isHostDriver` is the only structural fix that would let the passthrough variant coexist with a host driver. Not available today; requested in [NVIDIA/k8s-driver-manager#191](https://github.com/NVIDIA/k8s-driver-manager/issues/191). ## Building the vGPU Manager image The proprietary vGPU Manager driver must be obtained from NVIDIA and packaged into a container image that the gpu-operator chart pulls — it is not installed from a raw `.run` at runtime. NVIDIA owns this build path; their [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) repository ships per-OS Dockerfiles under `vgpu-manager//` and is the source of truth for build args, base images, and supported OS releases. Follow the README in that repository. The proprietary `.run` is the **Linux KVM** variant (not the Ubuntu KVM `.deb`, which ships pre-built modules for stock kernels only). It comes from the [NVIDIA Licensing Portal](https://ui.licensing.nvidia.com) under an NVIDIA AI Enterprise / vGPU subscription. > **EULA:** never push the resulting image to a publicly readable registry. Use a private registry (in-cluster Harbor works well as a non-proxy project). ## Deploying with the vgpu variant The platform's `iaas` bundle deploys the gpu-operator Package CR when `cozystack.gpu-operator` is in `bundles.enabledPackages` and `bundles.iaas.gpuOperatorVariant: vgpu` is set. The vGPU Manager image is proprietary and not redistributable, so the bundle does not ship a default tag — the operator builds the container per the upstream [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) recipe and supplies the private-registry coordinates through platform values: ```yaml bundles: iaas: enabled: true gpuOperatorVariant: vgpu enabledPackages: - cozystack.gpu-operator gpu: vgpuManager: repository: registry.example.com/nvidia image: vgpu-manager version: "595.58.02-ubuntu24.04" # imagePullSecrets lives per-component (vgpuManager, driver, # validator, dcgmExporter, …). The value is a list of strings, # not [{name: ...}]. imagePullSecrets: - nvidia-registry-secret ``` The platform forwards `gpu.vgpuManager` into the emitted gpu-operator Package CR's `components.gpu-operator.values.gpu-operator.vgpuManager`, so the bundle handles the variant + image coordinates in one place. If you need to override anything else on the gpu-operator chart (driver, validator, dcgmExporter, custom node selectors), hand-craft a `Package` CR named `cozystack.gpu-operator` with the full `components.gpu-operator.values` block and put that name in `bundles.disabledPackages`. A hand-written Package CR does not take precedence over the bundle: as long as the platform emits its own copy of that package, the next platform render replaces whatever you wrote. The `nvidia-registry-secret` should be a docker-registry Secret created beforehand in `cozy-gpu-operator`. Verify the DaemonSet is running and `nvidia.ko` loads on every GPU node: ```bash kubectl -n cozy-gpu-operator get pods -l app=nvidia-vgpu-manager-daemonset kubectl -n cozy-gpu-operator exec -it -- nvidia-smi ``` `nvidia-smi` should enumerate the physical GPUs and report `Host VGPU Mode : SR-IOV`. ## Profile assignment (SR-IOV path) > **The `vgpu` variant is experimental on Ada+ and ships without a profile-assignment loop.** NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does not exist on Ada+ — the DaemonSet errors with "no parent devices found for GPU at index '0'" and is therefore disabled by default in `values-vgpu.yaml`. Until an SR-IOV-aware controller is shipped, profile assignment is an out-of-band step that must be re-applied after every node reboot (`current_vgpu_type` resets to 0 on PCIe re-enumeration). Without this step, `permittedHostDevices.pciHostDevices` will report zero allocatable resources and no VM can request the vGPU. **Do not deploy the `vgpu` variant in production until you have an automated profile-assignment mechanism in place** — typically a small DaemonSet that reads a ConfigMap (` = `) and writes the corresponding `current_vgpu_type` files at boot. Once `nvidia.ko` is loaded the driver enables SR-IOV (16 VFs per L40S by default). Each VF needs a vGPU profile written to its sysfs: ```bash # from inside the nvidia-vgpu-manager-daemonset pod (privileged, hostPID) echo 1155 > /sys/bus/pci/devices/0000:02:00.5/nvidia/current_vgpu_type ``` The numeric profile ID can be discovered per-VF: ```bash cat /sys/bus/pci/devices/0000:02:00.5/nvidia/creatable_vgpu_types ``` For Pascal–Ampere GPUs (V100, T4, A100, A30) the mdev model still applies. Flip `vgpuDeviceManager.enabled: true` in your Package CR overrides — NVIDIA's device manager works correctly there. ## KubeVirt configuration When `cozystack.gpu-operator` is in `bundles.enabledPackages` (and not also in `bundles.disabledPackages`), the platform mirrors the chosen GPU variant into the `KubeVirt` CR automatically. There is no manual `kubectl patch` step. This applies to the two VM variants only: `container` serves GPUs to pods and never to VMs, so selecting it emits the gpu-operator Package with no host-device wiring on the KubeVirt side. If you opt out of bundle management and hand-craft a `cozystack.gpu-operator` Package CR directly — typically to apply overrides the bundle does not expose (driver settings, custom node selectors, validator / dcgmExporter tweaks, etc.) — the platform does NOT auto-wire `HostDevices` or `permittedHostDevices` into the KubeVirt CR. In that flow you also hand-craft a `cozystack.kubevirt` Package CR with `components.kubevirt.values.extraFeatureGates: [HostDevices]` and the appropriate `permittedHostDevices` block. The escape-hatch values shape under `.gpu` (below) is intentionally documented in the bundle-managed flow only. Both hand-crafted Package CRs need their names in `bundles.disabledPackages`: the iaas bundle emits `cozystack.kubevirt` on every render and `cozystack.gpu-operator` on every render that has it in `bundles.enabledPackages`, and that render replaces the hand-written spec rather than merging with it. - `developerConfiguration.featureGates` gets `HostDevices` appended (current KubeVirt splits this from the `GPU` gate; the admission webhook rejects `spec.template.spec.domain.devices.hostDevices` without it). - `permittedHostDevices.pciHostDevices` is filled from `packages/core/platform/files/gpu-passthrough-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: default` (the package default) and `gpu.replaceDefaults` is left at `false`. The table covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4), Volta (V100/V100S). All entries carry `externalResourceProvider: true` because the resource names come from `nvidia-sandbox-device-plugin`, not from KubeVirt's in-tree device plugin. - `permittedHostDevices.mediatedDevices` is filled from `packages/core/platform/files/gpu-vgpu-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: vgpu` **and** `gpu.vgpuDeviceManager.enabled: true` — that knob defaults to `false`, so a stock vgpu cluster ships no mdev table at all (see the Ada Lovelace / Blackwell note below for why). `gpu.replaceDefaults: true` also suppresses it. This list only EXPOSES, by profile name (`mdevNameSelector`), mdevs that the GPU Operator's vGPU Device Manager CREATES on the node; the platform does not ship a numeric `mediatedDevicesConfiguration` default (those `nvidia-NNN` type ids are per-SKU/driver sysfs indices with no portable value — set `.gpu.mediatedDevicesConfiguration` yourself, with host-verified ids, only if you want KubeVirt rather than the Device Manager to create mdevs). The starter set covers Pascal–Ampere mdev profiles (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family range as the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace / Blackwell SR-IOV vGPU is out of scope for the chart's default list; advertise those VFs via the user-override hook below. ### Extending or replacing the default table The platform exposes three knobs under `.gpu`: ```yaml gpu: # Extend the platform defaults with cluster-specific entries. Read by # the two VM variants only; the container variant reads neither key. # pciHostDevices is read in both of them — it feeds the passthrough # (vfio-pci) path AND the post-kubevirt#16890 SR-IOV vGPU VF path on # Ada Lovelace / Blackwell. mediatedDevices is read in vgpu alone, for # the pre-#16890 mdev path on Pascal–Ampere; the passthrough variant # ignores it. Both render into the same KubeVirt CR. permittedHostDevices: pciHostDevices: - pciVendorSelector: "10DE:26B9" # L40S, advertised as a VF for SR-IOV vGPU resourceName: nvidia.com/L40S-24Q # externalResourceProvider is intentionally omitted here: after # kubevirt/kubevirt#16890, virt-handler's in-tree device plugin # advertises the resource directly, no sandbox plugin in the loop. mediatedDevices: [] # mediatedDevicesConfiguration makes KubeVirt itself create mdevs (vgpu mode). No platform default: mdev creation is normally delegated to the vGPU Device Manager (name-based), and these mediatedDeviceTypes are host/driver-specific nvidia-NNN sysfs indices (look yours up via /sys/bus/pci/devices//mdev_supported_types/*/name). Set this only to opt into KubeVirt-driven creation; mergeOverwrite REPLACES a supplied top-level key wholesale. mediatedDevicesConfiguration: {} # Wipe the platform defaults entirely and ship only the cluster's # curated lists. Useful for non-NVIDIA-only clusters and strict # allowlist requirements. replaceDefaults: false ``` `replaceDefaults: false` (the default) appends user entries to the NVIDIA defaults. `replaceDefaults: true` drops the NVIDIA table entirely — if you don't then supply your own pciHostDevices / mediatedDevices list, the rendered KubeVirt CR has no `permittedHostDevices` block and the admission webhook will reject every GPU VM. ### Notes on `nvidia-sandbox-device-plugin` resource names The `resourceName` strings in `gpu-passthrough-defaults.yaml` are what `nvidia-sandbox-device-plugin` (`nvcr.io/nvidia/kubevirt-gpu-device-plugin`) advertises: it derives each slug mechanically from the device's PCI-IDs database name by uppercasing it, turning `/`, `.` and whitespace into `_`, and stripping the remaining non-alphanumerics (the `[` / `]`). So `TU104GL [Tesla T4]` becomes `nvidia.com/TU104GL_TESLA_T4` and `GA100GL [A30 PCIe]` becomes `nvidia.com/GA100GL_A30_PCIE` — the slug carries every token the PCI-IDs string holds (the `GL` die suffix, the `Tesla` brand on Turing/Volta, form factor, memory), not a tidy `_`. The names track the pci.ids snapshot bundled in the plugin image, so a different plugin build can publish a different string — check with `kubectl describe node | grep nvidia.com/` and override via `.gpu.permittedHostDevices.pciHostDevices` (or wipe the table with `replaceDefaults: true` and curate it yourself). PCI vendor:device IDs themselves are stable across driver versions. ### SR-IOV PF vs VF (Ada Lovelace and newer) On L40S (and other Ada-Lovelace cards) the SR-IOV VFs report the same PCI device ID as the PF — `lspci -nn -d 10de:` on the host shows both as `[10de:26b9]`. `virt-handler` distinguishes them by `is-VF + has-vGPU-profile`, so a single `pciVendorSelector` matches the right set. Verify on your specific GPU before assuming this — some other generations split PF/VF IDs. `externalResourceProvider: true` is **not** required when the resource is advertised by `virt-handler`'s in-tree device plugin (the SR-IOV path post kubevirt#16890). The platform passthrough defaults include the flag because that path is driven by the external sandbox plugin. ### Verifying allocatable capacity ```bash kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable.nvidia\.com/L40S-24Q}{"\n"}{end}' ``` ## Licensing (DLS) vGPU 17/20 uses the NVIDIA Delegated License Service. The legacy `ServerAddress=` / `ServerPort=7070` lines in `gridd.conf` are no longer authoritative — `nvidia-gridd` (running **inside the guest**) reads the DLS endpoint from the ClientConfigToken file directly. The host vGPU Manager DaemonSet does not request a license — it only enables SR-IOV and loads `nvidia.ko`. Licensing is consumed entirely by the guest. The gpu-operator chart's `driver.licensingConfig.secretName` would mount the Secret into the **driver pod on the host**, where it has no effect for SR-IOV vGPU; do not wire the licensing Secret through it. Instead, deliver the token and `gridd.conf` to the guest via cloud-init or a containerDisk overlay: ```yaml # inside the VirtualMachine cloudInitNoCloud userData write_files: - path: /etc/nvidia/ClientConfigToken/client_configuration_token.tok # 0744 follows NVIDIA's recommendation in the Virtual GPU Software # Licensing User Guide ("Configuring a Licensed Client on Linux"): # nvidia-gridd does not necessarily run as the file owner. # https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/ permissions: '0744' encoding: b64 content: - path: /etc/nvidia/gridd.conf permissions: '0644' content: | # FeatureType selects which vGPU Software license the guest requests. # 0 — unlicensed state (no license requested; Q profiles run in # reduced mode after the grace period). # 1 — NVIDIA vGPU. The driver auto-selects the correct license # type from the configured vGPU profile (Q → vWS, B → vPC, # A → vCS / Compute). Use this for SR-IOV vGPU profiles. # 2 — explicitly NVIDIA RTX Virtual Workstation. # 4 — explicitly NVIDIA Virtual Compute Server. FeatureType=1 ``` Verify activation inside the guest: ```bash nvidia-smi -q | grep 'License Status' # License Status : Licensed ``` If the guest reports `Unlicensed (Unrestricted)` for more than a couple of minutes, check `journalctl _COMM=nvidia-gridd` for handshake errors against the DLS endpoint baked into the token. ### Migrating from chart v25.x Operators upgrading from the previous Cozystack release (gpu-operator chart v25.3.0) should also note that the upstream chart deprecated `driver.licensingConfig.configMapName` in favour of `driver.licensingConfig.secretName`. The old key still works but emits a deprecation warning at render time. If your existing `Package` CR set the licensing reference via `configMapName`, switch it to `secretName` on this upgrade — the Secret content (`gridd.conf` and the ClientConfigToken) does not need to change. This applies to passthrough deployments that drove host-side licensing through the gpu-operator chart; SR-IOV vGPU does not consume the host-side licensing knob at all (see "Licensing (DLS)" above). ## Sample VirtualMachine Either `hostDevices` or `gpus` accepts the resource (the upstream KubeVirt API resolves both PCI and mediated-device pools), but the convention is to use `hostDevices` for VF-style PCI passthrough: ```yaml apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: vgpu-smoke namespace: tenant-example spec: runStrategy: Always template: spec: domain: cpu: cores: 4 memory: guest: 8Gi devices: disks: - name: rootdisk disk: bus: virtio interfaces: - name: default masquerade: {} hostDevices: - name: gpu0 deviceName: nvidia.com/L40S-24Q networks: - name: default pod: {} volumes: - name: rootdisk # A 2.4 GiB containerDisk overlay is too small to install # the GRID guest driver in-place. Use a CDI DataVolume of # 20 GiB+ in production. containerDisk: image: quay.io/containerdisks/ubuntu:24.04 ``` Inside the guest, install the GRID driver from the `.run` (the GUEST `.run`, distinct from the host `vgpu-kvm` package), then `nvidia-smi` should report the configured profile: ```text | 0 NVIDIA L40S-24Q Off | 00000000:0E:00.0 Off | 0 | | 17 MiB / 24576 MiB P0 Default | ``` ## Profile reference (L40S) L40S supports the full Q (RTX vWS), B (vPC), A (vCS / Compute) profile families. The numeric IDs come from the driver and are visible in `creatable_vgpu_types`: | Profile | Frame Buffer | Max instances per L40S | Use case | | --- | --- | --- | --- | | L40S-1Q | 1 GB | 48 | Light 3D / VDI | | L40S-2Q | 2 GB | 24 | Medium 3D / VDI | | L40S-4Q | 4 GB | 12 | Heavy 3D / VDI | | L40S-6Q | 6 GB | 8 | Professional 3D | | L40S-8Q | 8 GB | 6 | AI / ML inference | | L40S-12Q | 12 GB | 4 | AI / ML training | | L40S-24Q | 24 GB | 2 | Large AI workloads | | L40S-48Q | 48 GB | 1 | Full GPU equivalent | Other GPU families have analogous tables in the [NVIDIA Virtual GPU Software Documentation](https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/). ## OS support summary The `container` variant column assumes the host already ships the NVIDIA driver and `nvidia-container-toolkit` via the distro package manager, with the `nvidia` runtime registered in containerd (`nvidia-ctk runtime configure --runtime=containerd`). With `driver.enabled=false` the operator uses the pre-installed host driver at its standard location, so a stock apt install needs no `hostPaths.driverInstallDir` override. Talos installs the driver under a non-standard prefix, so the operator does not find it at the default location — see `packages/system/gpu-operator/examples/` for the Talos-specific path with a compat DaemonSet and an explicit `hostPaths.driverInstallDir` override. | Host OS | passthrough (`default`) | vGPU (`vgpu`) | container (`container`) | | --- | --- | --- | --- | | Ubuntu 24.04 | ⚠️ supported upstream, but the host must be clean of any apt-installed NVIDIA driver — see ["Passthrough variant: host preparation"](#passthrough-variant-host-preparation) | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | ✅ apt-installed driver + nvidia-container-toolkit | | Ubuntu 22.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ | | Ubuntu 20.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ | | Ubuntu 26.04 | ⚠️ same clean-host requirement as 24.04, plus a `nvidia-driver` patch for usr-merge (details pending) | ⚠️ same patch + own Dockerfile fork | ✅ | | Talos Linux | ✅ (open `vfio-pci`; the Talos image ships no host NVIDIA stack, so the clean-host check passes trivially) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked | ⚠️ host driver lands in a non-standard prefix — use `examples/values-native-talos.yaml` (compat DaemonSet + `hostPaths.driverInstallDir` override) as a starting point instead |