cozystack/packages/system/gpu-operator/examples

GPU operator — native pod workload on Talos (reference)

The files in this directory are not templates. They are reference artifacts that document one working configuration for running GPU workloads directly in pods on a Talos-based Cozystack cluster, together with the DCGM metrics needed by the gpu/gpu-performance Grafana dashboard.

For non-Talos Linux hosts that already provide the NVIDIA driver and nvidia-container-toolkit via the distro package manager (the typical apt-installed Ubuntu/Debian shape), pick the first-class container variant instead — variant: container in your Package CR is the starting point. Two host prerequisites still apply: containerd must have the nvidia runtime registered (nvidia-ctk runtime configure --runtime=containerd + restart; apt install lays down binaries only, it does not wire containerd), and because the variant runs with driver.enabled=false the operator uses the pre-installed host driver at its standard location — so a stock apt install needs no hostPaths.driverInstallDir override. The files here remain the right starting point on Talos, where the driver lands under a non-standard prefix and the operator does not find it at the default location without the override below.

The out-of-the-box values-passthrough.yaml for this package targets the sandbox (VFIO passthrough to KubeVirt VMs) scenario. The files here illustrate an alternative — running CUDA workloads in regular pods with the NVIDIA device plugin — and the workarounds it currently requires on Talos.

Files

• values-native-talos.yaml — Cozystack Package values that disable sandbox workloads, enable the device plugin, point hostPaths.driverInstallDir at the staging location used by the compat DaemonSet, and wire DCGM to the custom metrics ConfigMap.
• dcgm-custom-metrics.yaml — ConfigMap with a DCGM metrics CSV that adds profiling, ECC, throttling and energy counters on top of the upstream defaults. The CSV is a superset needed for full coverage of the gpu/gpu-performance dashboard. Which parts are actually required depends on which dashboards you ship — see the table below.
• nvidia-driver-compat.yaml — DaemonSet that stages libnvidia-ml.so.1 and nvidia-smi from the Talos glibc tree into a path where the NVIDIA GPU Operator validator expects them. See the "Why the compat DaemonSet exists" section below.

Why these are reference, not templates

Shipping these as first-class templates would silently impose assumptions that do not hold for every user:

• Whether the NVIDIA Talos system extension is installed on the nodes.
• Whether GPUs are exposed directly to pods or passed through to VMs.
• The exact path the installed driver ends up at (depends on the extension version and Talos release).

The sandbox-oriented values-passthrough.yaml remains the default. Operators who want native pod GPU workloads can start from this directory and adapt as needed.

Why the compat DaemonSet exists

The NVIDIA GPU Operator validator checks for libnvidia-ml.so.1 and bin/nvidia-smi in the path given by hostPaths.driverInstallDir. Talos installs them under /usr/local/glibc/usr/lib/ and /usr/local/bin/, which the validator does not look at. Until upstream addresses NVIDIA/gpu-operator#1687, the DaemonSet copies those files into a directory the validator does inspect and creates the .driver-ctr-ready flag file so the validator proceeds.

The compat DaemonSet runs privileged and bind-mounts host paths, so the target namespace must allow privileged pods. On clusters that enforce the Kubernetes Pod Security Standards at baseline or restricted, label the namespace with pod-security.kubernetes.io/enforce: privileged (and the matching audit/warn labels if the admission webhook is configured to surface violations) before applying the manifest.

Dashboards and what DCGM metrics they need

Five GPU dashboards live under gpu/* in packages/system/monitoring/dashboards-infra.list. All of them share packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml as their source of aggregated series. The recording rules are safe to ship on any cluster — they evaluate to empty series when DCGM is not scraped, or when optional counters are missing.

What each dashboard needs on top of the upstream DCGM Exporter default-counters.csv:

Dashboard	Scope	Needs beyond defaults
gpu-performance	Per-node, per-GPU deep dive	DCGM_FI_DEV_POWER_VIOLATION, DCGM_FI_DEV_THERMAL_VIOLATION
gpu-efficiency	Per-workload util vs tensor active	DCGM_FI_DEV_POWER_VIOLATION, DCGM_FI_DEV_THERMAL_VIOLATION (via gpu:*_throttle_fraction:rate5m recording rules)
gpu-fleet	Cluster-wide admin inventory	DCGM_FI_DEV_POWER_MGMT_LIMIT (for the TDP vs draw panel)
gpu-quotas	Kube-quota vs live usage	kube_pod_container_resource_requests, kube_pod_status_phase, kube_node_status_allocatable (via namespace:gpu_count:allocated / cluster:gpu_count:* recording rules)
gpu-tenants	Per-namespace tenant view	nothing (works on default counters)

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE and DCGM_FI_PROF_GR_ENGINE_ACTIVE are already in the upstream default set for the pinned DCGM Exporter version, so the tensor-saturation and engine-active panels work without any CSV override. The three counters listed in the table — throttling violations and the power management limit — are the only extras the tracked dashboards need. The recording rules in gpu-recording.rules.yaml consume utilization, FB used, power, temperature and the tensor-active profiling counter from the default set, plus DCGM_FI_DEV_POWER_VIOLATION and DCGM_FI_DEV_THERMAL_VIOLATION — used by the gpu.recording.efficiency.1m group to derive the gpu:power_throttle_fraction:rate5m and gpu:thermal_throttle_fraction:rate5m series consumed by the throttling panels on the efficiency and fleet dashboards.

The gpu.recording.throttle.validation.5m group additionally ships the GPUThrottleFractionOverOne alert (severity warning) as a regression detector: it fires when either throttle-fraction series exceeds 1.0, which would indicate that DCGM changed the scale/divisor of the underlying violation counters and the recording rules need to be re-derived.

Verification status

The minimum-CSV claims above are verified by hack/check-gpu-recording-rules.bats, which cross-checks every DCGM_FI_* reference in the tracked GPU dashboards and recording rules against the union of the upstream default set (snapshotted at hack/dcgm-default-counters.csv for the pinned DCGM Exporter version) and the custom CSV in dcgm-custom-metrics.yaml. When the DCGM Exporter image in packages/system/gpu-operator/charts/gpu-operator/values.yaml is bumped, refresh the snapshot from the matching tag of the NVIDIA/dcgm-exporter repository and rerun the test.