cozystack/docs/gpu-vgpu.md

# GPU Operator: passthrough and vGPU

This document describes how to configure the GPU Operator package with NVIDIA vGPU support so that a single physical GPU can be sliced and shared across multiple virtual machines.

**Last verified:** 2026-04-29 against KubeVirt `main` (`virt-handler` nightly `20260429_74d7c52588`) + the `vgpu` variant + NVIDIA vGPU 20.0 host driver `595.58.02` + GRID guest driver `595.58.03`.

## Two driver models

NVIDIA's vGPU driver uses two different host-side models depending on GPU generation:

- **Mediated devices (mdev)** — Pascal / Volta / Turing / Ampere up to A100 / A30. The driver creates `mdev` parent devices under `/sys/class/mdev_bus/`; KubeVirt advertises them via `permittedHostDevices.mediatedDevices`.
- **SR-IOV with per-VF sysfs** — Ada Lovelace (L4, L40, L40S, …) and Blackwell (B100, …) on the vGPU 17/20 driver branch. The driver creates SR-IOV virtual functions; profile selection happens via `/sys/bus/pci/devices/<VF>/nvidia/current_vgpu_type`. KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` after [kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890).

This guide focuses on the **SR-IOV path**, which is the only model NVIDIA supports for current data-centre GPUs. Mdev is mentioned for completeness; for Pascal–Ampere refer to the upstream NVIDIA GPU Operator docs.

## Prerequisites

- An Ada Lovelace (or newer) NVIDIA GPU that supports SR-IOV vGPU (L4, L40, L40S, etc.).
- Ubuntu 24.04 host OS. Older Ubuntu releases also work if the upstream `gpu-driver-container` repo has a matching `vgpu-manager/` Dockerfile. **Talos Linux is not recommended** for vGPU. NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero [closed siderolabs/extensions#461](https://github.com/siderolabs/extensions/issues/461) noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that includes the driver in-tree is therefore not feasible without a private fork that violates the EULA.
- KubeVirt v1.9.0 or newer, which is the release the platform bundles. [kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890) ("vGPU: SRIOV support") shipped in v1.9.0; v1.8.x and older do not include it and backports are not planned.
- An NVIDIA vGPU Software / NVIDIA AI Enterprise subscription (the `.run` is not redistributable).
- A reachable NVIDIA Delegated License Service (DLS) instance and a matching `client_configuration_token.tok` file.

## Variants

The `gpu-operator` package exposes three variants. This document is vGPU-focused; the variant inventory is shared.

- **`default`** — passthrough mode (`vfio-pci`). Whole GPU goes to a single VM. Talos is supported here; the kernel module is the open-source `vfio-pci`, no proprietary driver is needed on the host.
- **`vgpu`** — SR-IOV vGPU mode. One physical GPU is sliced into multiple VFs, each VF bound to a vGPU profile that the guest sees as its own GPU.
- **`container`** — containerized GPU workloads (CUDA pods, ML training) via the standard NVIDIA device plugin on hosts that already provide both the NVIDIA driver and `nvidia-container-toolkit` (the typical apt-installed Ubuntu/Debian shape). Sandbox workloads are off, `devicePlugin` is on, `driver` / `toolkit` / `vfioManager` / `cdi` are off so that the operator does not fight the host install. Orthogonal to the two VM variants — it does not pass GPUs to KubeVirt VMs. Note that `apt install nvidia-container-toolkit` installs binaries only — it does not configure containerd. Because this variant disables the operator's toolkit component (which would normally do that wiring), the host must additionally have run `nvidia-ctk runtime configure --runtime=containerd` (followed by a containerd restart) and exposed the `nvidia` runtime as the default or via a RuntimeClass before the device plugin can serve GPUs.

## Passthrough variant: host preparation

**Last verified:** 2026-05-28 against `cozystack.gpu-operator` chart `gpu-operator` v26.3.1 + image `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0` (distroless image with a Go ELF binary `/usr/bin/driver-manager`, entrypoint `["driver-manager", "preflight_check"]`; verified via `crane export … | tar -x usr/bin/driver-manager` and `strings` over the resulting binary) + Ubuntu 24.04 host with `nvidia-driver-580-open` 580.82.07 pre-installed. Tracks [#2763](https://github.com/cozystack/cozystack/issues/2763).

The `default` (passthrough) variant assumes the GPU is **owned by the host kernel's `vfio-pci` driver and nothing else**.

**How the operator detects a pre-installed host driver.** The chart's `vfio-manager` DaemonSet runs the upstream NVIDIA `k8s-driver-manager` init container with the `uninstall_driver` subcommand; that path calls the Go method `(*DriverManager).isHostDriver`, which runs `chroot /host nvidia-smi --query-gpu=driver_version --format=csv,noheader` and treats any non-empty stdout as "host driver present" (file-existence is not pre-tested — if `nvidia-smi` is missing, the chroot exec errors and `isHostDriver` returns false, which is the intended path on a clean host: the operator then proceeds with the uninstall flow and `vfio-manager` binds `vfio-pci` as designed). On a positive detection the binary logs `Host driver detected: <ver>`, labels the node `nvidia.com/gpu.deploy.driver=pre-installed`, exits, and `nvidia-sandbox-validator` then crashloops with `device not bound to 'vfio-pci'`.

**`FORCE_REINSTALL` does not bypass this.** `k8s-driver-manager` v0.10.0 exposes a `FORCE_REINSTALL` / `--force-reinstall` env+flag pair (visible in the binary's strings table), but it gates a later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit at the top — operators who set it and expect a bypass will report a false bug. There is currently no opt-out for the `isHostDriver` guard itself.

**Current mitigation.** The only mitigation that keeps VM passthrough working is to remove the host NVIDIA stack before enabling the variant (the "Clean-host workaround" section below).

**Alternatives and scope.** The `container` variant of `cozystack.gpu-operator` is the no-purge path: it keeps the host driver and exposes GPUs to pods rather than VMs, so it does not replace passthrough for anyone who actually needs a GPU inside a VM. Talos is unaffected because the Talos image ships only the `vfio-pci` extension; this section applies to Linux distributions where you installed the host driver yourself (typically Ubuntu / Debian / RHEL with `apt install nvidia-driver-*` or equivalent). See also [`packages/system/gpu-operator/examples/README.md`](../packages/system/gpu-operator/examples/README.md) for the Talos native-pod-workload reference flow.

### Symptom

The first thing you see is `kubectl get pods -n cozy-gpu-operator` showing `nvidia-vfio-manager-*` stuck in `Init:Error` / `Init:CrashLoopBackOff` — the init container exits non-zero after detecting the host driver, so kubelet keeps restarting it.

Its log shows it skipped the bind step:

```text
Host driver detected: 580.82.07
NVIDIA GPU driver is already pre-installed on the node,
  disabling the containerized driver
Labeling node <NODE> with nvidia.com/gpu.deploy.driver=pre-installed
```

`nvidia-sandbox-validator` then crashloops:

```text
Error: error validating vfio-pci driver installation:
  device not bound to 'vfio-pci'; device: 0000:18:00.0 driver: 'nvidia'
```

`lspci -nnk -d 10de:` still shows `Kernel driver in use: nvidia` on every target GPU, the node carries `nvidia.com/gpu.deploy.driver=pre-installed`, and `kubectl get node <NODE> -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com/")))'` reports `{}` — no GPU resource was registered.

### Clean-host workaround

Purge the NVIDIA host stack and blacklist the kernel modules so the host never re-claims the GPU. Two pitfalls to avoid: `apt autoremove` is dangerous here because the `nvidia-` prefix is shared with NVIDIA DOCA / Mellanox / InfiniBand userspace, so a blanket autoremove can take RDMA out on a converged GPU + RDMA host; and a hardcoded `apt purge 'nvidia-*' 'cuda-*'` pattern list is fragile — `apt` treats `*` as a cache-wide regex and **aborts the entire transaction, purging nothing**, if any pattern matches nothing in the cache (e.g. `cuda-*` on a host without NVIDIA's CUDA repo). Build the list from what is actually installed instead:

```bash
# dpkg-query patterns are true globs over INSTALLED packages: no
# zero-match abort (unlike apt's cache-wide regex) and no accidental
# substring over-match. List first, then review before purging.
dpkg-query -W -f '${Package}\n' 'nvidia-*' 'libnvidia-*' 'cuda-*' 2>/dev/null
```

Review the list before purging:

- `nvidia-dkms-*` and `nvidia-kernel-*` are the load-bearing kernel pieces — without removing them DKMS rebuilds `nvidia.ko` on the next reboot and the blacklist below is bypassed by any explicit `modprobe`.
- On a **converged GPU + RDMA host**, drop any `libnvidia-*` that belong to NVIDIA DOCA / Mellanox OFED — purging those breaks RDMA.
- If the list is **empty**, the driver was installed with NVIDIA's `.run` installer rather than apt — run `sudo nvidia-uninstall` instead of the purge below.

Then purge the reviewed list, blacklist the modules, and rebuild the initramfs:

```bash
# Replace with the packages you kept from the list above.
sudo apt purge nvidia-driver-580-open nvidia-dkms-580-open <...>

sudo tee /etc/modprobe.d/blacklist-nvidia.conf > /dev/null <<'EOF'
blacklist nouveau
blacklist nvidia
blacklist nvidia_drm
blacklist nvidia_modeset
blacklist nvidia_uvm
blacklist nvidia_peermem
EOF

# -k all rebuilds EVERY installed kernel's initramfs (plain -u touches
# only the running kernel), so a just-upgraded kernel also boots with
# the blacklist and cannot re-claim the GPU.
sudo update-initramfs -u -k all
sudo reboot
```

After the reboot, confirm the host is clean. The init container exits non-zero after detecting a host driver, so both DaemonSet pods sit in `Init:CrashLoopBackOff` and **will retry on their own** — deleting them just skips the up-to-5-minute backoff window:

```bash
# Should print nothing.
lsmod | grep -E '^(nvidia|nouveau)'

# command -v matches what isHostDriver does (PATH lookup inside the
# chroot), so it also catches /usr/local/bin/nvidia-smi left by a .run
# or CUDA-toolkit install. Prints "ok: gone" on success.
command -v nvidia-smi >/dev/null && echo "STILL PRESENT — purge incomplete" || echo "ok: gone"

# A leftover DKMS module rebuilds nvidia.ko on the next kernel update
# and an explicit modprobe bypasses the blacklist — should print nothing.
dkms status | grep -i nvidia

# Skip the CrashLoopBackOff backoff window by deleting the stuck pods.
# The DaemonSet labels are operator-managed and not stable across
# gpu-operator versions, so delete by name pattern — anchored so the
# match is the two DaemonSets and nothing else.
kubectl -n cozy-gpu-operator get pods -o name \
  | grep -E '^pod/(nvidia-vfio-manager|nvidia-sandbox-validator)-' \
  | xargs -r kubectl -n cozy-gpu-operator delete
```

Within a couple of minutes `vfio-manager` should bind every target GPU to `vfio-pci` and the node's `allocatable` will gain the registered resource:

```bash
lspci -nnk -d 10de: | grep 'Kernel driver in use'
# Kernel driver in use: vfio-pci

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable}{"\n"}{end}' | grep nvidia.com
```

One label is **not** cleaned up automatically. The init container set `nvidia.com/gpu.deploy.driver=pre-installed`, and the operator's success path restores only the operand labels — it never resets this one (`rescheduleGPUOperatorComponents` in `k8s-driver-manager` v0.10.0 touches the validator / toolkit / device-plugin / etc. labels, not `deploy.driver`). So the node keeps `pre-installed` indefinitely: it is the same label the Symptom section uses as evidence, and it would disable the containerized-driver DaemonSet if the node later switches to the `container` workload. Clear it once recovery is confirmed:

```bash
# Nothing on the success path resets this label; clear it so it does not
# mislead future debugging or block a later container-workload switch.
kubectl label node <NODE> nvidia.com/gpu.deploy.driver-
```

### Known limitation

The skip-on-pre-installed behavior lives in the upstream [`NVIDIA/k8s-driver-manager`](https://github.com/NVIDIA/k8s-driver-manager) Go binary at `cmd/driver-manager/main.go`: the method `(*DriverManager).isHostDriver` is called from `(*DriverManager).uninstallDriver` and has no in-band opt-out in `:v0.10.0`. `FORCE_REINSTALL` exists as an env / CLI flag in the same binary but gates the later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit. Hosts that need to keep the NVIDIA host driver installed for non-Kubernetes workloads cannot currently share the same GPU with the passthrough variant. Two mitigation paths:

- **`container` variant** — a third variant of `cozystack.gpu-operator` that targets the apt-installed-driver host shape and exposes GPUs to pods (not VMs) without unbinding the host driver. Available now, and selectable from platform values with `bundles.iaas.gpuOperatorVariant: container`. It solves the host-driver conflict by giving up VM passthrough, so it is a mitigation only for workloads that can run in a pod.
- **Upstream override** — an env-var override of `isHostDriver` is the only structural fix that would let the passthrough variant coexist with a host driver. Not available today; requested in [NVIDIA/k8s-driver-manager#191](https://github.com/NVIDIA/k8s-driver-manager/issues/191).

## Building the vGPU Manager image

The proprietary vGPU Manager driver must be obtained from NVIDIA and packaged into a container image that the gpu-operator chart pulls — it is not installed from a raw `.run` at runtime. NVIDIA owns this build path; their [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) repository ships per-OS Dockerfiles under `vgpu-manager/<os>/` and is the source of truth for build args, base images, and supported OS releases. Follow the README in that repository.

The proprietary `.run` is the **Linux KVM** variant (not the Ubuntu KVM `.deb`, which ships pre-built modules for stock kernels only). It comes from the [NVIDIA Licensing Portal](https://ui.licensing.nvidia.com) under an NVIDIA AI Enterprise / vGPU subscription.

> **EULA:** never push the resulting image to a publicly readable registry. Use a private registry (in-cluster Harbor works well as a non-proxy project).

## Deploying with the vgpu variant

The platform's `iaas` bundle deploys the gpu-operator Package CR when `cozystack.gpu-operator` is in `bundles.enabledPackages` and `bundles.iaas.gpuOperatorVariant: vgpu` is set. The vGPU Manager image is proprietary and not redistributable, so the bundle does not ship a default tag — the operator builds the container per the upstream [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) recipe and supplies the private-registry coordinates through platform values:

```yaml
bundles:
  iaas:
    enabled: true
    gpuOperatorVariant: vgpu
  enabledPackages:
  - cozystack.gpu-operator

gpu:
  vgpuManager:
    repository: registry.example.com/nvidia
    image: vgpu-manager
    version: "595.58.02-ubuntu24.04"
    # imagePullSecrets lives per-component (vgpuManager, driver,
    # validator, dcgmExporter, …). The value is a list of strings,
    # not [{name: ...}].
    imagePullSecrets:
    - nvidia-registry-secret
```

The platform forwards `gpu.vgpuManager` into the emitted gpu-operator Package CR's `components.gpu-operator.values.gpu-operator.vgpuManager`, so the bundle handles the variant + image coordinates in one place. If you need to override anything else on the gpu-operator chart (driver, validator, dcgmExporter, custom node selectors), hand-craft a `Package` CR named `cozystack.gpu-operator` with the full `components.gpu-operator.values` block and put that name in `bundles.disabledPackages`. A hand-written Package CR does not take precedence over the bundle: as long as the platform emits its own copy of that package, the next platform render replaces whatever you wrote.

The `nvidia-registry-secret` should be a docker-registry Secret created beforehand in `cozy-gpu-operator`.

Verify the DaemonSet is running and `nvidia.ko` loads on every GPU node:

```bash
kubectl -n cozy-gpu-operator get pods -l app=nvidia-vgpu-manager-daemonset
kubectl -n cozy-gpu-operator exec -it <pod> -- nvidia-smi
```

`nvidia-smi` should enumerate the physical GPUs and report `Host VGPU Mode : SR-IOV`.

## Profile assignment (SR-IOV path)

> **The `vgpu` variant is experimental on Ada+ and ships without a profile-assignment loop.** NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does not exist on Ada+ — the DaemonSet errors with "no parent devices found for GPU at index '0'" and is therefore disabled by default in `values-vgpu.yaml`. Until an SR-IOV-aware controller is shipped, profile assignment is an out-of-band step that must be re-applied after every node reboot (`current_vgpu_type` resets to 0 on PCIe re-enumeration). Without this step, `permittedHostDevices.pciHostDevices` will report zero allocatable resources and no VM can request the vGPU. **Do not deploy the `vgpu` variant in production until you have an automated profile-assignment mechanism in place** — typically a small DaemonSet that reads a ConfigMap (`<bus-id> = <profile-id>`) and writes the corresponding `current_vgpu_type` files at boot.

Once `nvidia.ko` is loaded the driver enables SR-IOV (16 VFs per L40S by default). Each VF needs a vGPU profile written to its sysfs:

```bash
# from inside the nvidia-vgpu-manager-daemonset pod (privileged, hostPID)
echo 1155 > /sys/bus/pci/devices/0000:02:00.5/nvidia/current_vgpu_type
```

The numeric profile ID can be discovered per-VF:

```bash
cat /sys/bus/pci/devices/0000:02:00.5/nvidia/creatable_vgpu_types
```

For Pascal–Ampere GPUs (V100, T4, A100, A30) the mdev model still applies. Flip `vgpuDeviceManager.enabled: true` in your Package CR overrides — NVIDIA's device manager works correctly there.

## KubeVirt configuration

When `cozystack.gpu-operator` is in `bundles.enabledPackages` (and not also in `bundles.disabledPackages`), the platform mirrors the chosen GPU variant into the `KubeVirt` CR automatically. There is no manual `kubectl patch` step. This applies to the two VM variants only: `container` serves GPUs to pods and never to VMs, so selecting it emits the gpu-operator Package with no host-device wiring on the KubeVirt side.

If you opt out of bundle management and hand-craft a `cozystack.gpu-operator` Package CR directly — typically to apply overrides the bundle does not expose (driver settings, custom node selectors, validator / dcgmExporter tweaks, etc.) — the platform does NOT auto-wire `HostDevices` or `permittedHostDevices` into the KubeVirt CR. In that flow you also hand-craft a `cozystack.kubevirt` Package CR with `components.kubevirt.values.extraFeatureGates: [HostDevices]` and the appropriate `permittedHostDevices` block. The escape-hatch values shape under `.gpu` (below) is intentionally documented in the bundle-managed flow only. Both hand-crafted Package CRs need their names in `bundles.disabledPackages`: the iaas bundle emits `cozystack.kubevirt` on every render and `cozystack.gpu-operator` on every render that has it in `bundles.enabledPackages`, and that render replaces the hand-written spec rather than merging with it.

- `developerConfiguration.featureGates` gets `HostDevices` appended (current KubeVirt splits this from the `GPU` gate; the admission webhook rejects `spec.template.spec.domain.devices.hostDevices` without it).
- `permittedHostDevices.pciHostDevices` is filled from `packages/core/platform/files/gpu-passthrough-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: default` (the package default) and `gpu.replaceDefaults` is left at `false`. The table covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4), Volta (V100/V100S). All entries carry `externalResourceProvider: true` because the resource names come from `nvidia-sandbox-device-plugin`, not from KubeVirt's in-tree device plugin.
- `permittedHostDevices.mediatedDevices` is filled from `packages/core/platform/files/gpu-vgpu-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: vgpu` **and** `gpu.vgpuDeviceManager.enabled: true` — that knob defaults to `false`, so a stock vgpu cluster ships no mdev table at all (see the Ada Lovelace / Blackwell note below for why). `gpu.replaceDefaults: true` also suppresses it. This list only EXPOSES, by profile name (`mdevNameSelector`), mdevs that the GPU Operator's vGPU Device Manager CREATES on the node; the platform does not ship a numeric `mediatedDevicesConfiguration` default (those `nvidia-NNN` type ids are per-SKU/driver sysfs indices with no portable value — set `.gpu.mediatedDevicesConfiguration` yourself, with host-verified ids, only if you want KubeVirt rather than the Device Manager to create mdevs). The starter set covers Pascal–Ampere mdev profiles (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family range as the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace / Blackwell SR-IOV vGPU is out of scope for the chart's default list; advertise those VFs via the user-override hook below.

### Extending or replacing the default table

The platform exposes three knobs under `.gpu`:

```yaml
gpu:
  # Extend the platform defaults with cluster-specific entries. Read by
  # the two VM variants only; the container variant reads neither key.
  # pciHostDevices is read in both of them — it feeds the passthrough
  # (vfio-pci) path AND the post-kubevirt#16890 SR-IOV vGPU VF path on
  # Ada Lovelace / Blackwell. mediatedDevices is read in vgpu alone, for
  # the pre-#16890 mdev path on Pascal–Ampere; the passthrough variant
  # ignores it. Both render into the same KubeVirt CR.
  permittedHostDevices:
    pciHostDevices:
    - pciVendorSelector: "10DE:26B9"   # L40S, advertised as a VF for SR-IOV vGPU
      resourceName: nvidia.com/L40S-24Q
      # externalResourceProvider is intentionally omitted here: after
      # kubevirt/kubevirt#16890, virt-handler's in-tree device plugin
      # advertises the resource directly, no sandbox plugin in the loop.
    mediatedDevices: []
  # mediatedDevicesConfiguration makes KubeVirt itself create mdevs (vgpu mode). No platform default: mdev creation is normally delegated to the vGPU Device Manager (name-based), and these mediatedDeviceTypes are host/driver-specific nvidia-NNN sysfs indices (look yours up via /sys/bus/pci/devices/<BDF>/mdev_supported_types/*/name). Set this only to opt into KubeVirt-driven creation; mergeOverwrite REPLACES a supplied top-level key wholesale.
  mediatedDevicesConfiguration: {}
  # Wipe the platform defaults entirely and ship only the cluster's
  # curated lists. Useful for non-NVIDIA-only clusters and strict
  # allowlist requirements.
  replaceDefaults: false
```

`replaceDefaults: false` (the default) appends user entries to the NVIDIA defaults. `replaceDefaults: true` drops the NVIDIA table entirely — if you don't then supply your own pciHostDevices / mediatedDevices list, the rendered KubeVirt CR has no `permittedHostDevices` block and the admission webhook will reject every GPU VM.

### Notes on `nvidia-sandbox-device-plugin` resource names

The `resourceName` strings in `gpu-passthrough-defaults.yaml` are what `nvidia-sandbox-device-plugin` (`nvcr.io/nvidia/kubevirt-gpu-device-plugin`) advertises: it derives each slug mechanically from the device's PCI-IDs database name by uppercasing it, turning `/`, `.` and whitespace into `_`, and stripping the remaining non-alphanumerics (the `[` / `]`). So `TU104GL [Tesla T4]` becomes `nvidia.com/TU104GL_TESLA_T4` and `GA100GL [A30 PCIe]` becomes `nvidia.com/GA100GL_A30_PCIE` — the slug carries every token the PCI-IDs string holds (the `GL` die suffix, the `Tesla` brand on Turing/Volta, form factor, memory), not a tidy `<arch>_<model>`. The names track the pci.ids snapshot bundled in the plugin image, so a different plugin build can publish a different string — check with `kubectl describe node <node> | grep nvidia.com/` and override via `.gpu.permittedHostDevices.pciHostDevices` (or wipe the table with `replaceDefaults: true` and curate it yourself). PCI vendor:device IDs themselves are stable across driver versions.

### SR-IOV PF vs VF (Ada Lovelace and newer)

On L40S (and other Ada-Lovelace cards) the SR-IOV VFs report the same PCI device ID as the PF — `lspci -nn -d 10de:` on the host shows both as `[10de:26b9]`. `virt-handler` distinguishes them by `is-VF + has-vGPU-profile`, so a single `pciVendorSelector` matches the right set. Verify on your specific GPU before assuming this — some other generations split PF/VF IDs.

`externalResourceProvider: true` is **not** required when the resource is advertised by `virt-handler`'s in-tree device plugin (the SR-IOV path post kubevirt#16890). The platform passthrough defaults include the flag because that path is driven by the external sandbox plugin.

### Verifying allocatable capacity

```bash
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable.nvidia\.com/L40S-24Q}{"\n"}{end}'
```

## Licensing (DLS)

vGPU 17/20 uses the NVIDIA Delegated License Service. The legacy `ServerAddress=` / `ServerPort=7070` lines in `gridd.conf` are no longer authoritative — `nvidia-gridd` (running **inside the guest**) reads the DLS endpoint from the ClientConfigToken file directly.

The host vGPU Manager DaemonSet does not request a license — it only enables SR-IOV and loads `nvidia.ko`. Licensing is consumed entirely by the guest. The gpu-operator chart's `driver.licensingConfig.secretName` would mount the Secret into the **driver pod on the host**, where it has no effect for SR-IOV vGPU; do not wire the licensing Secret through it.

Instead, deliver the token and `gridd.conf` to the guest via cloud-init or a containerDisk overlay:

```yaml
# inside the VirtualMachine cloudInitNoCloud userData
write_files:
- path: /etc/nvidia/ClientConfigToken/client_configuration_token.tok
  # 0744 follows NVIDIA's recommendation in the Virtual GPU Software
  # Licensing User Guide ("Configuring a Licensed Client on Linux"):
  # nvidia-gridd does not necessarily run as the file owner.
  # https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/
  permissions: '0744'
  encoding: b64
  content: <base64 token>
- path: /etc/nvidia/gridd.conf
  permissions: '0644'
  content: |
    # FeatureType selects which vGPU Software license the guest requests.
    # 0 — unlicensed state (no license requested; Q profiles run in
    #     reduced mode after the grace period).
    # 1 — NVIDIA vGPU. The driver auto-selects the correct license
    #     type from the configured vGPU profile (Q → vWS, B → vPC,
    #     A → vCS / Compute). Use this for SR-IOV vGPU profiles.
    # 2 — explicitly NVIDIA RTX Virtual Workstation.
    # 4 — explicitly NVIDIA Virtual Compute Server.
    FeatureType=1
```

Verify activation inside the guest:

```bash
nvidia-smi -q | grep 'License Status'
# License Status   : Licensed
```

If the guest reports `Unlicensed (Unrestricted)` for more than a couple of minutes, check `journalctl _COMM=nvidia-gridd` for handshake errors against the DLS endpoint baked into the token.

### Migrating from chart v25.x

Operators upgrading from the previous Cozystack release (gpu-operator chart v25.3.0) should also note that the upstream chart deprecated `driver.licensingConfig.configMapName` in favour of `driver.licensingConfig.secretName`. The old key still works but emits a deprecation warning at render time. If your existing `Package` CR set the licensing reference via `configMapName`, switch it to `secretName` on this upgrade — the Secret content (`gridd.conf` and the ClientConfigToken) does not need to change. This applies to passthrough deployments that drove host-side licensing through the gpu-operator chart; SR-IOV vGPU does not consume the host-side licensing knob at all (see "Licensing (DLS)" above).

## Sample VirtualMachine

Either `hostDevices` or `gpus` accepts the resource (the upstream KubeVirt API resolves both PCI and mediated-device pools), but the convention is to use `hostDevices` for VF-style PCI passthrough:

```yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vgpu-smoke
  namespace: tenant-example
spec:
  runStrategy: Always
  template:
    spec:
      domain:
        cpu:
          cores: 4
        memory:
          guest: 8Gi
        devices:
          disks:
          - name: rootdisk
            disk:
              bus: virtio
          interfaces:
          - name: default
            masquerade: {}
          hostDevices:
          - name: gpu0
            deviceName: nvidia.com/L40S-24Q
      networks:
      - name: default
        pod: {}
      volumes:
      - name: rootdisk
        # A 2.4 GiB containerDisk overlay is too small to install
        # the GRID guest driver in-place. Use a CDI DataVolume of
        # 20 GiB+ in production.
        containerDisk:
          image: quay.io/containerdisks/ubuntu:24.04
```

Inside the guest, install the GRID driver from the `.run` (the GUEST `.run`, distinct from the host `vgpu-kvm` package), then `nvidia-smi` should report the configured profile:

```text
| 0  NVIDIA L40S-24Q                Off |   00000000:0E:00.0 Off |                    0 |
|        17 MiB / 24576 MiB    P0    Default                                                |
```

## Profile reference (L40S)

L40S supports the full Q (RTX vWS), B (vPC), A (vCS / Compute) profile families. The numeric IDs come from the driver and are visible in `creatable_vgpu_types`:

| Profile | Frame Buffer | Max instances per L40S | Use case |
| --- | --- | --- | --- |
| L40S-1Q | 1 GB | 48 | Light 3D / VDI |
| L40S-2Q | 2 GB | 24 | Medium 3D / VDI |
| L40S-4Q | 4 GB | 12 | Heavy 3D / VDI |
| L40S-6Q | 6 GB | 8 | Professional 3D |
| L40S-8Q | 8 GB | 6 | AI / ML inference |
| L40S-12Q | 12 GB | 4 | AI / ML training |
| L40S-24Q | 24 GB | 2 | Large AI workloads |
| L40S-48Q | 48 GB | 1 | Full GPU equivalent |

Other GPU families have analogous tables in the [NVIDIA Virtual GPU Software Documentation](https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/).

## OS support summary

The `container` variant column assumes the host already ships the NVIDIA driver and `nvidia-container-toolkit` via the distro package manager, with the `nvidia` runtime registered in containerd (`nvidia-ctk runtime configure --runtime=containerd`). With `driver.enabled=false` the operator uses the pre-installed host driver at its standard location, so a stock apt install needs no `hostPaths.driverInstallDir` override. Talos installs the driver under a non-standard prefix, so the operator does not find it at the default location — see `packages/system/gpu-operator/examples/` for the Talos-specific path with a compat DaemonSet and an explicit `hostPaths.driverInstallDir` override.

| Host OS | passthrough (`default`) | vGPU (`vgpu`) | container (`container`) |
| --- | --- | --- | --- |
| Ubuntu 24.04 | ⚠️ supported upstream, but the host must be clean of any apt-installed NVIDIA driver — see ["Passthrough variant: host preparation"](#passthrough-variant-host-preparation) | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | ✅ apt-installed driver + nvidia-container-toolkit |
| Ubuntu 22.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ |
| Ubuntu 20.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ |
| Ubuntu 26.04 | ⚠️ same clean-host requirement as 24.04, plus a `nvidia-driver` patch for usr-merge (details pending) | ⚠️ same patch + own Dockerfile fork | ✅ |
| Talos Linux | ✅ (open `vfio-pci`; the Talos image ships no host NVIDIA stack, so the clean-host check passes trivially) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked | ⚠️ host driver lands in a non-standard prefix — use `examples/values-native-talos.yaml` (compat DaemonSet + `hostPaths.driverInstallDir` override) as a starting point instead |