Free-floating 'NVIDIA recommends' claims are flagged as unverifiable
in review. Add the doc URL inline so future readers can verify the
permission model decision.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pin a concrete reference set so future readers can tell whether the
guide is still accurate against the cluster they're running:
KubeVirt main nightly 20260429_74d7c52588, this PR's vgpu variant,
NVIDIA vGPU 20.0 host driver 595.58.02, GRID guest driver 595.58.03.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
NVIDIA's gpu-driver-container repository owns the build path —
keeping a parallel set of build instructions in our docs duplicates
upstream documentation and goes stale every time the upstream
Dockerfile or build args change. Replace the docker build / docker
push snippet with a paragraph pointing at the upstream repo and the
NVIDIA Licensing Portal as the sources of truth, keep only the
items that are Cozystack-specific (private-registry / EULA note).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Apple Silicon / arm64 build hosts produce arm64 images native; GPU
nodes are amd64 and the kubelet pull fails with 'no matching
manifest' when the image is built without --platform. Document the
flag and add a one-line docker login note for private registries.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
NVIDIA's License System User Guide recommends 744 on the
ClientConfigToken file so nvidia-gridd (which does not necessarily
run as the file owner) can read it. Previous example used 0600,
which can silently break license activation when nvidia-gridd
runs as a service user.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Mirroring the corresponding cleanup in cozystack/website#467 review:
* KubeVirt v1.9.0 ETA: drop the 'July 2026' date — KubeVirt does not
publish hard release dates and the in-page wording was inconsistent
across the file. Replace with 'Targeted at the next minor release
(v1.9.0); track the PR for the actual release tag', matching the
website doc.
* Kernel-headers timing: 'The build downloads kernel headers at
container start time' conflated docker build with pod start; the
Dockerfile's entrypoint downloads them at runtime. Reword.
* Replace guillemets with ASCII quotes for consistency with the rest
of the file and the upstream chart docs.
* nvidia-smi -q license check: drop the unnecessary -A 1 — License
Status value is on the same line as the field name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two technical inaccuracies caught during review:
* The claim 'gpus: expects an mdev resource and will not match a
pciHostDevices entry' is wrong. Upstream KubeVirt
(pkg/virt-launcher/virtwrap/device/hostdevice/gpu/hostdev.go) iterates
both the PCI pool (v1.PCIResourcePrefix) and the mdev pool
(v1.MDevResourcePrefix) and accepts whichever resolves. Reword to
describe gpus: vs hostDevices: as a style convention rather than a
hard technical constraint.
* siderolabs/extensions#461 was closed with stateReason=COMPLETED, not
not_planned. Quoting the closure as 'won't fix' is editorial
recasting that does not match the actual closing comment. Replace
with the substantive reason from rothgar's closing comment: NVIDIA
does not publicly distribute the vGPU guest driver, requires
Enterprise Portal access, and Sidero cannot support vGPU until
NVIDIA changes its licensing terms.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds hack/check-gpu-operator-variants.bats. The test renders both
package variants with helm template and asserts the resulting
ClusterPolicy spec for the values that this PR cares about:
* ccManager.enabled=false (pinned in values.yaml; upstream v26 default
was flipped to true)
* vgpuDeviceManager.enabled=false (same — and on Ada+/Blackwell the
DaemonSet would crashloop because /sys/class/mdev_bus is empty)
* sandboxWorkloads.defaultWorkload — vm-passthrough on the default
variant, vm-vgpu on the vgpu variant; without these the upstream
'container' default makes either variant a silent no-op on
unlabelled GPU nodes
* vgpuManager.enabled=true on the vgpu variant; driver / devicePlugin
enabled=false on both variants (proprietary host driver path is
out of scope for both passthrough and SR-IOV vGPU)
Negative-tested locally: deleting any of these settings makes the
corresponding test fail, so it actually guards regressions on the
next chart bump.
Also addresses last-pass review comments:
* values-passthrough.yaml: set sandboxWorkloads.defaultWorkload=
vm-passthrough so the default variant is symmetric with the vgpu
variant — neither relies on per-node labels for the variant to
activate. Without this the upstream default 'container' would
apply on unlabelled GPU nodes and vfio-manager would never start.
* values.yaml: rewrite the vgpuDeviceManager comment to lead with
the load-bearing reason (mdev path missing on Ada+/Blackwell)
instead of the secondary 'stays at zero pods on passthrough'
rationalisation.
* docs/gpu-vgpu.md: add a 'Migrating from chart v25.x' subsection
in the Licensing section pointing at the configMapName ->
secretName rename. Old key still works upstream but emits a
deprecation warning; PR body release note already calls this out
but the docs were missing the migration breadcrumb.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
* examples/README.md and examples/values-native-talos.yaml header
comment still pointed at values-talos.yaml after the rename to
values-passthrough.yaml. Update both to the new name.
* values-vgpu.yaml: drop the leftover vgpuDeviceManager.config block.
The chart only renders the config when vgpuDeviceManager.enabled is
true, and we keep it false on Ada+; the orphan config.default key
was scaffolding without effect.
* docs/gpu-vgpu.md and values.yaml comment: stop grouping B100 under
Ada Lovelace. B100 is Blackwell; the SR-IOV vGPU mechanism applies
to both architectures but they should be named correctly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Addresses second review pass. Three blockers + one polish:
* values-vgpu.yaml: set sandboxWorkloads.defaultWorkload=vm-vgpu.
Without this the upstream default 'container' applies on any node
without the per-node nvidia.com/gpu.workload.config label, the
operator selects the container workload (driver/devicePlugin
DaemonSets), and those are disabled in this variant — so nothing
runs on the node and vgpu-manager never starts. The cluster-wide
default makes the variant functional out of the box; per-node
labels still override it for mixed clusters.
* docs/gpu-vgpu.md: imagePullSecrets is per-component on this chart
(vgpuManager.imagePullSecrets, driver.imagePullSecrets, …). The
Package CR example previously placed the field at gpu-operator: root
where it would be silently ignored at render. Move it under
vgpuManager: where the chart actually reads it.
* docs/gpu-vgpu.md, release note: mark the vgpu variant as
experimental and call out the missing profile-assignment loop
explicitly. Until an SR-IOV-aware controller writes
current_vgpu_type per VF (and re-applies after every reboot —
PCIe re-enumeration resets the value), the variant is not safe
for production. The docs now lead with that warning instead of
burying it in a caveat.
* Rename packages/system/gpu-operator/values-talos.yaml to
values-passthrough.yaml. The overrides (sandboxWorkloads.enabled,
driver and devicePlugin disabled) apply to any host OS using the
vfio-pci passthrough path — not Talos specifically. Update
PackageSource ref accordingly. Comment in the file documents the
rename so future readers understand the prior name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Addresses follow-up review of the vGPU variant + chart bump.
* values.yaml: pin ccManager.enabled=false and vgpuDeviceManager.enabled=false.
Upstream chart v26.x flipped both to true. ccManager auto-enables CC
mode on Hopper-class hardware after upgrade — a non-trivial behavior
change that should be opt-in. vgpuDeviceManager walks
/sys/class/mdev_bus/, which does not exist on Ada+ (L4/L40/L40S/B100
with vGPU 17/20). Pinning both off keeps the default footprint as it
was on v25.x; users explicitly opt in.
* hack/dcgm-default-counters.csv: refresh from dcgm-exporter tag
4.5.1-4.8.0 (matches the image bumped earlier in this PR). The
previous snapshot was pinned to 4.1.1-4.0.4. Adds DCGM_FI_DEV_FB_RESERVED
and other fields introduced upstream. Header comment updated.
hack/check-gpu-recording-rules.bats passes against the refreshed
fixture.
* docs/gpu-vgpu.md (Licensing section):
- Drop the wrong recommendation to wire the licensing Secret through
gpu-operator's driver.licensingConfig.secretName. That mounts into
the host driver pod, where SR-IOV vGPU does not consume a license.
Document guest-side delivery (cloud-init write_files for
/etc/nvidia/ClientConfigToken/ and /etc/nvidia/gridd.conf) end to
end.
- Correct FeatureType comment. Per the NVIDIA Virtual GPU Software
Licensing User Guide, FeatureType=0 is the unlicensed state, not
auto-detect; FeatureType=1 is the value that asks the driver to
auto-select the license type from the configured vGPU profile.
Switch the example to FeatureType=1 and list all four valid values
with their meaning.
- Add nvidia-smi verification snippet inside the guest.
* docs/gpu-vgpu.md (KubeVirt section): note that L40S SR-IOV VFs share
the PF device ID (10de:26b9), so a single pciVendorSelector matches
both. Recommend lspci -nn -d 10de: as a sanity check on other
generations where PF/VF IDs may differ.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Rewrites docs/gpu-vgpu.md to match the actual NVIDIA driver model on
current data-centre GPUs. Key changes:
- Drop the implicit assumption that vGPU uses mediated devices. Ada
Lovelace and newer (L4, L40, L40S, B100) on the vGPU 17/20 branch
use SR-IOV with per-VF NVIDIA sysfs (current_vgpu_type); KubeVirt
advertises them via permittedHostDevices.pciHostDevices.
- Document KubeVirt PR #16890 as a hard dependency for the SR-IOV
path. The PR is in main since 2026-04-10 and will ship in v1.9.0
(ETA July 2026); released tags v1.6.x/1.7.x/1.8.x do not include
it and backports are not planned.
- Replace the legacy NLS configuration block (ServerAddress=,
ServerPort=7070, FeatureType=1) with the current DLS flow:
ClientConfigToken file in /etc/nvidia/ClientConfigToken/, gridd.conf
with FeatureType=0 (auto-detect).
- Switch the build instructions from the archived
gitlab.com/nvidia/container-images/driver to the upstream
github.com/NVIDIA/gpu-driver-container.
- Switch sample VirtualMachine spec from gpus: (mdev resource) to
hostDevices: (PCI resource).
- Note explicitly that Talos is not recommended for the vGPU variant
— the proprietary .run cannot be redistributed in a Talos system
extension, and siderolabs/extensions#461 was closed won't-fix.
Passthrough on Talos is unaffected.
Rename packages/system/gpu-operator/values-talos-vgpu.yaml to
values-vgpu.yaml — the overrides (sandbox workloads, pod-side driver
and device plugin disabled) are not Talos-specific, they apply to
any VM-only GPU node. Update PackageSource accordingly.
Default vgpuDeviceManager.enabled: false. The DaemonSet walks
/sys/class/mdev_bus/, which does not exist on Ada+ — keeping it on
by default would crashloop modern hardware. Operators of older GPUs
(Pascal/Volta/Turing/Ampere) can flip the flag back on.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
ConfigMap-based licensing configuration (configMapName) is deprecated
in GPU Operator v26.x. Switch to Secret-based configuration (secretName)
as recommended by the updated CRD schema.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace simplified Containerfile with NVIDIA's Makefile-based build
system from gitlab.com/nvidia/container-images/driver. The GPU Operator
expects pre-compiled kernel modules, not a raw .run file.
Add EULA notice about public redistribution of vGPU driver images.
Add note about NLS ServerPort being deployment-dependent.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add vGPU mode as a new variant alongside the existing passthrough (default) variant.
This enables sharing a single physical GPU across multiple VMs using NVIDIA mediated devices.
Changes:
- Add values-talos-vgpu.yaml with vGPU Manager and vGPU Device Manager enabled
- Add 'vgpu' variant to PackageSource
- Add documentation for vGPU setup (driver image, NLS licensing, profiles, KubeVirt)
Users select the vgpu variant via Package CR and provide their proprietary
vGPU Manager image coordinates through component values.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin