Adds hack/check-gpu-operator-variants.bats. The test renders both
package variants with helm template and asserts the resulting
ClusterPolicy spec for the values that this PR cares about:
* ccManager.enabled=false (pinned in values.yaml; upstream v26 default
was flipped to true)
* vgpuDeviceManager.enabled=false (same — and on Ada+/Blackwell the
DaemonSet would crashloop because /sys/class/mdev_bus is empty)
* sandboxWorkloads.defaultWorkload — vm-passthrough on the default
variant, vm-vgpu on the vgpu variant; without these the upstream
'container' default makes either variant a silent no-op on
unlabelled GPU nodes
* vgpuManager.enabled=true on the vgpu variant; driver / devicePlugin
enabled=false on both variants (proprietary host driver path is
out of scope for both passthrough and SR-IOV vGPU)
Negative-tested locally: deleting any of these settings makes the
corresponding test fail, so it actually guards regressions on the
next chart bump.
Also addresses last-pass review comments:
* values-passthrough.yaml: set sandboxWorkloads.defaultWorkload=
vm-passthrough so the default variant is symmetric with the vgpu
variant — neither relies on per-node labels for the variant to
activate. Without this the upstream default 'container' would
apply on unlabelled GPU nodes and vfio-manager would never start.
* values.yaml: rewrite the vgpuDeviceManager comment to lead with
the load-bearing reason (mdev path missing on Ada+/Blackwell)
instead of the secondary 'stays at zero pods on passthrough'
rationalisation.
* docs/gpu-vgpu.md: add a 'Migrating from chart v25.x' subsection
in the Licensing section pointing at the configMapName ->
secretName rename. Old key still works upstream but emits a
deprecation warning; PR body release note already calls this out
but the docs were missing the migration breadcrumb.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
* examples/README.md and examples/values-native-talos.yaml header
comment still pointed at values-talos.yaml after the rename to
values-passthrough.yaml. Update both to the new name.
* values-vgpu.yaml: drop the leftover vgpuDeviceManager.config block.
The chart only renders the config when vgpuDeviceManager.enabled is
true, and we keep it false on Ada+; the orphan config.default key
was scaffolding without effect.
* docs/gpu-vgpu.md and values.yaml comment: stop grouping B100 under
Ada Lovelace. B100 is Blackwell; the SR-IOV vGPU mechanism applies
to both architectures but they should be named correctly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Addresses second review pass. Three blockers + one polish:
* values-vgpu.yaml: set sandboxWorkloads.defaultWorkload=vm-vgpu.
Without this the upstream default 'container' applies on any node
without the per-node nvidia.com/gpu.workload.config label, the
operator selects the container workload (driver/devicePlugin
DaemonSets), and those are disabled in this variant — so nothing
runs on the node and vgpu-manager never starts. The cluster-wide
default makes the variant functional out of the box; per-node
labels still override it for mixed clusters.
* docs/gpu-vgpu.md: imagePullSecrets is per-component on this chart
(vgpuManager.imagePullSecrets, driver.imagePullSecrets, …). The
Package CR example previously placed the field at gpu-operator: root
where it would be silently ignored at render. Move it under
vgpuManager: where the chart actually reads it.
* docs/gpu-vgpu.md, release note: mark the vgpu variant as
experimental and call out the missing profile-assignment loop
explicitly. Until an SR-IOV-aware controller writes
current_vgpu_type per VF (and re-applies after every reboot —
PCIe re-enumeration resets the value), the variant is not safe
for production. The docs now lead with that warning instead of
burying it in a caveat.
* Rename packages/system/gpu-operator/values-talos.yaml to
values-passthrough.yaml. The overrides (sandboxWorkloads.enabled,
driver and devicePlugin disabled) apply to any host OS using the
vfio-pci passthrough path — not Talos specifically. Update
PackageSource ref accordingly. Comment in the file documents the
rename so future readers understand the prior name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Addresses follow-up review of the vGPU variant + chart bump.
* values.yaml: pin ccManager.enabled=false and vgpuDeviceManager.enabled=false.
Upstream chart v26.x flipped both to true. ccManager auto-enables CC
mode on Hopper-class hardware after upgrade — a non-trivial behavior
change that should be opt-in. vgpuDeviceManager walks
/sys/class/mdev_bus/, which does not exist on Ada+ (L4/L40/L40S/B100
with vGPU 17/20). Pinning both off keeps the default footprint as it
was on v25.x; users explicitly opt in.
* hack/dcgm-default-counters.csv: refresh from dcgm-exporter tag
4.5.1-4.8.0 (matches the image bumped earlier in this PR). The
previous snapshot was pinned to 4.1.1-4.0.4. Adds DCGM_FI_DEV_FB_RESERVED
and other fields introduced upstream. Header comment updated.
hack/check-gpu-recording-rules.bats passes against the refreshed
fixture.
* docs/gpu-vgpu.md (Licensing section):
- Drop the wrong recommendation to wire the licensing Secret through
gpu-operator's driver.licensingConfig.secretName. That mounts into
the host driver pod, where SR-IOV vGPU does not consume a license.
Document guest-side delivery (cloud-init write_files for
/etc/nvidia/ClientConfigToken/ and /etc/nvidia/gridd.conf) end to
end.
- Correct FeatureType comment. Per the NVIDIA Virtual GPU Software
Licensing User Guide, FeatureType=0 is the unlicensed state, not
auto-detect; FeatureType=1 is the value that asks the driver to
auto-select the license type from the configured vGPU profile.
Switch the example to FeatureType=1 and list all four valid values
with their meaning.
- Add nvidia-smi verification snippet inside the guest.
* docs/gpu-vgpu.md (KubeVirt section): note that L40S SR-IOV VFs share
the PF device ID (10de:26b9), so a single pciVendorSelector matches
both. Recommend lspci -nn -d 10de: as a sanity check on other
generations where PF/VF IDs may differ.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin