Aleksei Sviridkin
6e8566a94c
The cozystack.gpu-operator PackageSource ships three variants: default, vgpu and container. The iaas bundle's variant guard accepted only the first two, so selecting container failed the render on a value the package ships, values-container.yaml configures deliberately, and the variant bats suite already covers. The exclusion was coherent when it was written and stopped being so immediately. The container variant and the guard landed within minutes of each other from parallel branches, neither an ancestor of the other, variant first. Until the guard, the bundle hardcoded the default variant and container was selected by hand-crafting a Package CR, so leaving it out of a bundle-side list cost nothing. The guard then made the bundle the variant owner and closed that route without noticing it was one; its comment describes itself as typo protection and names passthrough as the example typo. The KubeVirt side is not a bare allowlist edit. The container variant serves GPUs to pods through the device plugin and runs no VM-facing GPU stack: sandboxWorkloads and vfioManager are both off and the host driver stays bound. Both KubeVirt-side pieces the bundle wires are VM-facing — the HostDevices feature gate advertises a capability such a node cannot serve, and permittedHostDevices lists resources no sandbox device plugin registers there. Both are now emitted only for default and vgpu, and container yields the gpu-operator Package with the kubevirt component untouched. Comments and docs that described the KubeVirt mirroring as unconditional on variant, or the container variant as unmerged and unavailable, are corrected to match. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
GPU operator — native pod workload on Talos (reference)
The files in this directory are not templates. They are reference artifacts that document one working configuration for running GPU workloads directly in pods on a Talos-based Cozystack cluster, together with the DCGM metrics needed by the gpu/gpu-performance Grafana dashboard.
For non-Talos Linux hosts that already provide the NVIDIA driver and nvidia-container-toolkit via the distro package manager (the typical apt-installed Ubuntu/Debian shape), pick the first-class container variant instead — set bundles.iaas.gpuOperatorVariant: container in platform values and put cozystack.gpu-operator in bundles.enabledPackages, or hand-craft a Package CR with variant: container if you need overrides the bundle does not expose. Two host prerequisites still apply: containerd must have the nvidia runtime registered (nvidia-ctk runtime configure --runtime=containerd + restart; apt install lays down binaries only, it does not wire containerd), and because the variant runs with driver.enabled=false the operator uses the pre-installed host driver at its standard location — so a stock apt install needs no hostPaths.driverInstallDir override. The files here remain the right starting point on Talos, where the driver lands under a non-standard prefix and the operator does not find it at the default location without the override below.
The out-of-the-box values-passthrough.yaml for this package targets the sandbox (VFIO passthrough to KubeVirt VMs) scenario. The files here illustrate an alternative — running CUDA workloads in regular pods with the NVIDIA device plugin — and the workarounds it currently requires on Talos.
Files
values-native-talos.yaml— CozystackPackagevalues that disable sandbox workloads, enable the device plugin, pointhostPaths.driverInstallDirat the staging location used by the compat DaemonSet, and wire DCGM to the custom metrics ConfigMap.dcgm-custom-metrics.yaml—ConfigMapwith a DCGM metrics CSV that adds profiling, ECC, throttling and energy counters on top of the upstream defaults. The CSV is a superset needed for full coverage of thegpu/gpu-performancedashboard. Which parts are actually required depends on which dashboards you ship — see the table below.nvidia-driver-compat.yaml— DaemonSet that stageslibnvidia-ml.so.1andnvidia-smifrom the Talos glibc tree into a path where the NVIDIA GPU Operator validator expects them. See the "Why the compat DaemonSet exists" section below.
Why these are reference, not templates
Shipping these as first-class templates would silently impose assumptions that do not hold for every user:
- Whether the NVIDIA Talos system extension is installed on the nodes.
- Whether GPUs are exposed directly to pods or passed through to VMs.
- The exact path the installed driver ends up at (depends on the extension version and Talos release).
The sandbox-oriented values-passthrough.yaml remains the default. Operators
who want native pod GPU workloads can start from this directory and
adapt as needed.
Why the compat DaemonSet exists
The NVIDIA GPU Operator validator checks for libnvidia-ml.so.1 and
bin/nvidia-smi in the path given by hostPaths.driverInstallDir.
Talos installs them under /usr/local/glibc/usr/lib/ and
/usr/local/bin/, which the validator does not look at. Until upstream
addresses NVIDIA/gpu-operator#1687, the DaemonSet copies those
files into a directory the validator does inspect and creates the
.driver-ctr-ready flag file so the validator proceeds.
The compat DaemonSet runs privileged and bind-mounts host paths, so
the target namespace must allow privileged pods. On clusters that
enforce the Kubernetes Pod Security Standards at baseline or
restricted, label the namespace with
pod-security.kubernetes.io/enforce: privileged (and the matching
audit/warn labels if the admission webhook is configured to
surface violations) before applying the manifest.
Dashboards and what DCGM metrics they need
Five GPU dashboards live under gpu/* in
packages/system/monitoring/dashboards-infra.list. All of them share
packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml as
their source of aggregated series. The recording rules are safe to
ship on any cluster — they evaluate to empty series when DCGM is not
scraped, or when optional counters are missing.
What each dashboard needs on top of the upstream DCGM Exporter
default-counters.csv:
| Dashboard | Scope | Needs beyond defaults |
|---|---|---|
gpu-performance |
Per-node, per-GPU deep dive | DCGM_FI_DEV_POWER_VIOLATION, DCGM_FI_DEV_THERMAL_VIOLATION |
gpu-efficiency |
Per-workload util vs tensor active | DCGM_FI_DEV_POWER_VIOLATION, DCGM_FI_DEV_THERMAL_VIOLATION (via gpu:*_throttle_fraction:rate5m recording rules) |
gpu-fleet |
Cluster-wide admin inventory | DCGM_FI_DEV_POWER_MGMT_LIMIT (for the TDP vs draw panel) |
gpu-quotas |
Kube-quota vs live usage | kube_pod_container_resource_requests, kube_pod_status_phase, kube_node_status_allocatable (via namespace:gpu_count:allocated / cluster:gpu_count:* recording rules) |
gpu-tenants |
Per-namespace tenant view | nothing (works on default counters) |
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE and DCGM_FI_PROF_GR_ENGINE_ACTIVE
are already in the upstream default set for the pinned DCGM Exporter
version, so the tensor-saturation and engine-active panels work without
any CSV override. The three counters listed in the table — throttling
violations and the power management limit — are the only extras the
tracked dashboards need. The recording rules in
gpu-recording.rules.yaml consume utilization, FB used, power,
temperature and the tensor-active profiling counter from the default
set, plus DCGM_FI_DEV_POWER_VIOLATION and
DCGM_FI_DEV_THERMAL_VIOLATION — used by the
gpu.recording.efficiency.1m group to derive the
gpu:power_throttle_fraction:rate5m and
gpu:thermal_throttle_fraction:rate5m series consumed by the
throttling panels on the efficiency and fleet dashboards.
The gpu.recording.throttle.validation.5m group additionally ships the
GPUThrottleFractionOverOne alert (severity warning) as a regression
detector: it fires when either throttle-fraction series exceeds 1.0,
which would indicate that DCGM changed the scale/divisor of the
underlying violation counters and the recording rules need to be
re-derived.
Verification status
The minimum-CSV claims above are verified by
hack/check-gpu-recording-rules.bats, which cross-checks every
DCGM_FI_* reference in the tracked GPU dashboards and recording rules
against the union of the upstream default set (snapshotted at
hack/dcgm-default-counters.csv for the pinned DCGM Exporter version)
and the custom CSV in dcgm-custom-metrics.yaml. When the DCGM
Exporter image in packages/system/gpu-operator/charts/gpu-operator/values.yaml
is bumped, refresh the snapshot from the matching tag of the
NVIDIA/dcgm-exporter repository and rerun the test.