mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
Aleksei Sviridkin
9352d3ee0e
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.
The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.
- driver.enabled=false use the host driver
- toolkit.enabled=false host already has nvidia-container-toolkit;
enabling overwrites the containerd
config.toml the host configured via
nvidia-ctk runtime configure
- devicePlugin.enabled=true publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false do not unbind the host driver
- cdi.enabled=false CDI spec generation is serviced by the
disabled toolkit; an apt host uses the
legacy nvidia-container-runtime hook, so
pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false regular pod workloads, not VMs
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
116 lines
7.4 KiB
Markdown
116 lines
7.4 KiB
Markdown
# GPU operator — native pod workload on Talos (reference)
|
|
|
|
The files in this directory are **not** templates. They are reference artifacts that document one working configuration for running GPU workloads directly in pods on a Talos-based Cozystack cluster, together with the DCGM metrics needed by the `gpu/gpu-performance` Grafana dashboard.
|
|
|
|
For non-Talos Linux hosts that already provide the NVIDIA driver and `nvidia-container-toolkit` via the distro package manager (the typical apt-installed Ubuntu/Debian shape), pick the first-class `container` variant instead — `variant: container` in your Package CR is the starting point. Two host prerequisites still apply: containerd must have the `nvidia` runtime registered (`nvidia-ctk runtime configure --runtime=containerd` + restart; `apt install` lays down binaries only, it does not wire containerd), and because the variant runs with `driver.enabled=false` the operator uses the pre-installed host driver at its standard location — so a stock apt install needs no `hostPaths.driverInstallDir` override. The files here remain the right starting point on Talos, where the driver lands under a non-standard prefix and the operator does not find it at the default location without the override below.
|
|
|
|
The out-of-the-box `values-passthrough.yaml` for this package targets the sandbox (VFIO passthrough to KubeVirt VMs) scenario. The files here illustrate an alternative — running CUDA workloads in regular pods with the NVIDIA device plugin — and the workarounds it currently requires on Talos.
|
|
|
|
## Files
|
|
|
|
- [`values-native-talos.yaml`](./values-native-talos.yaml) — Cozystack
|
|
`Package` values that disable sandbox workloads, enable the device
|
|
plugin, point `hostPaths.driverInstallDir` at the staging location
|
|
used by the compat DaemonSet, and wire DCGM to the custom metrics
|
|
ConfigMap.
|
|
- [`dcgm-custom-metrics.yaml`](./dcgm-custom-metrics.yaml) — `ConfigMap`
|
|
with a DCGM metrics CSV that adds profiling, ECC, throttling and
|
|
energy counters on top of the upstream defaults. The CSV is a
|
|
superset needed for full coverage of the `gpu/gpu-performance`
|
|
dashboard. Which parts are actually required depends on which
|
|
dashboards you ship — see the table below.
|
|
- [`nvidia-driver-compat.yaml`](./nvidia-driver-compat.yaml) — DaemonSet
|
|
that stages `libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc
|
|
tree into a path where the NVIDIA GPU Operator validator expects
|
|
them. See the "Why the compat DaemonSet exists" section below.
|
|
|
|
## Why these are reference, not templates
|
|
|
|
Shipping these as first-class templates would silently impose
|
|
assumptions that do not hold for every user:
|
|
|
|
- Whether the NVIDIA Talos system extension is installed on the nodes.
|
|
- Whether GPUs are exposed directly to pods or passed through to VMs.
|
|
- The exact path the installed driver ends up at (depends on the
|
|
extension version and Talos release).
|
|
|
|
The sandbox-oriented `values-passthrough.yaml` remains the default. Operators
|
|
who want native pod GPU workloads can start from this directory and
|
|
adapt as needed.
|
|
|
|
## Why the compat DaemonSet exists
|
|
|
|
The NVIDIA GPU Operator validator checks for `libnvidia-ml.so.1` and
|
|
`bin/nvidia-smi` in the path given by `hostPaths.driverInstallDir`.
|
|
Talos installs them under `/usr/local/glibc/usr/lib/` and
|
|
`/usr/local/bin/`, which the validator does not look at. Until upstream
|
|
addresses [NVIDIA/gpu-operator#1687][1], the DaemonSet copies those
|
|
files into a directory the validator does inspect and creates the
|
|
`.driver-ctr-ready` flag file so the validator proceeds.
|
|
|
|
[1]: https://github.com/NVIDIA/gpu-operator/issues/1687
|
|
|
|
The compat DaemonSet runs privileged and bind-mounts host paths, so
|
|
the target namespace must allow privileged pods. On clusters that
|
|
enforce the Kubernetes Pod Security Standards at `baseline` or
|
|
`restricted`, label the namespace with
|
|
`pod-security.kubernetes.io/enforce: privileged` (and the matching
|
|
`audit`/`warn` labels if the admission webhook is configured to
|
|
surface violations) before applying the manifest.
|
|
|
|
## Dashboards and what DCGM metrics they need
|
|
|
|
Five GPU dashboards live under `gpu/*` in
|
|
`packages/system/monitoring/dashboards-infra.list`. All of them share
|
|
`packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml` as
|
|
their source of aggregated series. The recording rules are safe to
|
|
ship on any cluster — they evaluate to empty series when DCGM is not
|
|
scraped, or when optional counters are missing.
|
|
|
|
What each dashboard needs on top of the upstream DCGM Exporter
|
|
[`default-counters.csv`][default-csv]:
|
|
|
|
| Dashboard | Scope | Needs beyond defaults |
|
|
| ----------------- | ---------------------------------- | ----------------------------------------------------------------------- |
|
|
| `gpu-performance` | Per-node, per-GPU deep dive | `DCGM_FI_DEV_POWER_VIOLATION`, `DCGM_FI_DEV_THERMAL_VIOLATION` |
|
|
| `gpu-efficiency` | Per-workload util vs tensor active | `DCGM_FI_DEV_POWER_VIOLATION`, `DCGM_FI_DEV_THERMAL_VIOLATION` (via `gpu:*_throttle_fraction:rate5m` recording rules) |
|
|
| `gpu-fleet` | Cluster-wide admin inventory | `DCGM_FI_DEV_POWER_MGMT_LIMIT` (for the TDP vs draw panel) |
|
|
| `gpu-quotas` | Kube-quota vs live usage | `kube_pod_container_resource_requests`, `kube_pod_status_phase`, `kube_node_status_allocatable` (via `namespace:gpu_count:allocated` / `cluster:gpu_count:*` recording rules) |
|
|
| `gpu-tenants` | Per-namespace tenant view | nothing (works on default counters) |
|
|
|
|
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` and `DCGM_FI_PROF_GR_ENGINE_ACTIVE`
|
|
are already in the upstream default set for the pinned DCGM Exporter
|
|
version, so the tensor-saturation and engine-active panels work without
|
|
any CSV override. The three counters listed in the table — throttling
|
|
violations and the power management limit — are the only extras the
|
|
tracked dashboards need. The recording rules in
|
|
`gpu-recording.rules.yaml` consume utilization, FB used, power,
|
|
temperature and the tensor-active profiling counter from the default
|
|
set, plus `DCGM_FI_DEV_POWER_VIOLATION` and
|
|
`DCGM_FI_DEV_THERMAL_VIOLATION` — used by the
|
|
`gpu.recording.efficiency.1m` group to derive the
|
|
`gpu:power_throttle_fraction:rate5m` and
|
|
`gpu:thermal_throttle_fraction:rate5m` series consumed by the
|
|
throttling panels on the efficiency and fleet dashboards.
|
|
|
|
The `gpu.recording.throttle.validation.5m` group additionally ships the
|
|
`GPUThrottleFractionOverOne` alert (severity `warning`) as a regression
|
|
detector: it fires when either throttle-fraction series exceeds 1.0,
|
|
which would indicate that DCGM changed the scale/divisor of the
|
|
underlying violation counters and the recording rules need to be
|
|
re-derived.
|
|
|
|
## Verification status
|
|
|
|
The minimum-CSV claims above are verified by
|
|
`hack/check-gpu-recording-rules.bats`, which cross-checks every
|
|
`DCGM_FI_*` reference in the tracked GPU dashboards and recording rules
|
|
against the union of the upstream default set (snapshotted at
|
|
`hack/dcgm-default-counters.csv` for the pinned DCGM Exporter version)
|
|
and the custom CSV in `dcgm-custom-metrics.yaml`. When the DCGM
|
|
Exporter image in `packages/system/gpu-operator/charts/gpu-operator/values.yaml`
|
|
is bumped, refresh the snapshot from the matching tag of the
|
|
[`NVIDIA/dcgm-exporter`][default-csv] repository and rerun the test.
|
|
|
|
[default-csv]: https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/default-counters.csv
|