mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 05:25:14 +09:00
Aleksei Sviridkin
47a67761aa
* examples/README.md and examples/values-native-talos.yaml header comment still pointed at values-talos.yaml after the rename to values-passthrough.yaml. Update both to the new name. * values-vgpu.yaml: drop the leftover vgpuDeviceManager.config block. The chart only renders the config when vgpuDeviceManager.enabled is true, and we keep it false on Ada+; the orphan config.default key was scaffolding without effect. * docs/gpu-vgpu.md and values.yaml comment: stop grouping B100 under Ada Lovelace. B100 is Blackwell; the SR-IOV vGPU mechanism applies to both architectures but they should be named correctly. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
122 lines
6.5 KiB
Markdown
122 lines
6.5 KiB
Markdown
# GPU operator — native pod workload on Talos (reference)
|
|
|
|
The files in this directory are **not** templates. They are reference
|
|
artifacts that document one working configuration for running GPU
|
|
workloads directly in pods on a Talos-based Cozystack cluster, together
|
|
with the DCGM metrics needed by the `gpu/gpu-performance` Grafana
|
|
dashboard.
|
|
|
|
The out-of-the-box `values-passthrough.yaml` for this package targets the
|
|
sandbox (VFIO passthrough to KubeVirt VMs) scenario. The files here
|
|
illustrate an alternative — running CUDA workloads in regular pods with
|
|
the NVIDIA device plugin — and the workarounds it currently requires on
|
|
Talos.
|
|
|
|
## Files
|
|
|
|
- [`values-native-talos.yaml`](./values-native-talos.yaml) — Cozystack
|
|
`Package` values that disable sandbox workloads, enable the device
|
|
plugin, point `hostPaths.driverInstallDir` at the staging location
|
|
used by the compat DaemonSet, and wire DCGM to the custom metrics
|
|
ConfigMap.
|
|
- [`dcgm-custom-metrics.yaml`](./dcgm-custom-metrics.yaml) — `ConfigMap`
|
|
with a DCGM metrics CSV that adds profiling, ECC, throttling and
|
|
energy counters on top of the upstream defaults. The CSV is a
|
|
superset needed for full coverage of the `gpu/gpu-performance`
|
|
dashboard. Which parts are actually required depends on which
|
|
dashboards you ship — see the table below.
|
|
- [`nvidia-driver-compat.yaml`](./nvidia-driver-compat.yaml) — DaemonSet
|
|
that stages `libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc
|
|
tree into a path where the NVIDIA GPU Operator validator expects
|
|
them. See the "Why the compat DaemonSet exists" section below.
|
|
|
|
## Why these are reference, not templates
|
|
|
|
Shipping these as first-class templates would silently impose
|
|
assumptions that do not hold for every user:
|
|
|
|
- Whether the NVIDIA Talos system extension is installed on the nodes.
|
|
- Whether GPUs are exposed directly to pods or passed through to VMs.
|
|
- The exact path the installed driver ends up at (depends on the
|
|
extension version and Talos release).
|
|
|
|
The sandbox-oriented `values-passthrough.yaml` remains the default. Operators
|
|
who want native pod GPU workloads can start from this directory and
|
|
adapt as needed.
|
|
|
|
## Why the compat DaemonSet exists
|
|
|
|
The NVIDIA GPU Operator validator checks for `libnvidia-ml.so.1` and
|
|
`bin/nvidia-smi` in the path given by `hostPaths.driverInstallDir`.
|
|
Talos installs them under `/usr/local/glibc/usr/lib/` and
|
|
`/usr/local/bin/`, which the validator does not look at. Until upstream
|
|
addresses [NVIDIA/gpu-operator#1687][1], the DaemonSet copies those
|
|
files into a directory the validator does inspect and creates the
|
|
`.driver-ctr-ready` flag file so the validator proceeds.
|
|
|
|
[1]: https://github.com/NVIDIA/gpu-operator/issues/1687
|
|
|
|
The compat DaemonSet runs privileged and bind-mounts host paths, so
|
|
the target namespace must allow privileged pods. On clusters that
|
|
enforce the Kubernetes Pod Security Standards at `baseline` or
|
|
`restricted`, label the namespace with
|
|
`pod-security.kubernetes.io/enforce: privileged` (and the matching
|
|
`audit`/`warn` labels if the admission webhook is configured to
|
|
surface violations) before applying the manifest.
|
|
|
|
## Dashboards and what DCGM metrics they need
|
|
|
|
Five GPU dashboards live under `gpu/*` in
|
|
`packages/system/monitoring/dashboards-infra.list`. All of them share
|
|
`packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml` as
|
|
their source of aggregated series. The recording rules are safe to
|
|
ship on any cluster — they evaluate to empty series when DCGM is not
|
|
scraped, or when optional counters are missing.
|
|
|
|
What each dashboard needs on top of the upstream DCGM Exporter
|
|
[`default-counters.csv`][default-csv]:
|
|
|
|
| Dashboard | Scope | Needs beyond defaults |
|
|
| ----------------- | ---------------------------------- | ----------------------------------------------------------------------- |
|
|
| `gpu-performance` | Per-node, per-GPU deep dive | `DCGM_FI_DEV_POWER_VIOLATION`, `DCGM_FI_DEV_THERMAL_VIOLATION` |
|
|
| `gpu-efficiency` | Per-workload util vs tensor active | `DCGM_FI_DEV_POWER_VIOLATION`, `DCGM_FI_DEV_THERMAL_VIOLATION` (via `gpu:*_throttle_fraction:rate5m` recording rules) |
|
|
| `gpu-fleet` | Cluster-wide admin inventory | `DCGM_FI_DEV_POWER_MGMT_LIMIT` (for the TDP vs draw panel) |
|
|
| `gpu-quotas` | Kube-quota vs live usage | `kube_pod_container_resource_requests`, `kube_pod_status_phase`, `kube_node_status_allocatable` (via `namespace:gpu_count:allocated` / `cluster:gpu_count:*` recording rules) |
|
|
| `gpu-tenants` | Per-namespace tenant view | nothing (works on default counters) |
|
|
|
|
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` and `DCGM_FI_PROF_GR_ENGINE_ACTIVE`
|
|
are already in the upstream default set for the pinned DCGM Exporter
|
|
version, so the tensor-saturation and engine-active panels work without
|
|
any CSV override. The three counters listed in the table — throttling
|
|
violations and the power management limit — are the only extras the
|
|
tracked dashboards need. The recording rules in
|
|
`gpu-recording.rules.yaml` consume utilization, FB used, power,
|
|
temperature and the tensor-active profiling counter from the default
|
|
set, plus `DCGM_FI_DEV_POWER_VIOLATION` and
|
|
`DCGM_FI_DEV_THERMAL_VIOLATION` — used by the
|
|
`gpu.recording.efficiency.1m` group to derive the
|
|
`gpu:power_throttle_fraction:rate5m` and
|
|
`gpu:thermal_throttle_fraction:rate5m` series consumed by the
|
|
throttling panels on the efficiency and fleet dashboards.
|
|
|
|
The `gpu.recording.throttle.validation.5m` group additionally ships the
|
|
`GPUThrottleFractionOverOne` alert (severity `warning`) as a regression
|
|
detector: it fires when either throttle-fraction series exceeds 1.0,
|
|
which would indicate that DCGM changed the scale/divisor of the
|
|
underlying violation counters and the recording rules need to be
|
|
re-derived.
|
|
|
|
## Verification status
|
|
|
|
The minimum-CSV claims above are verified by
|
|
`hack/check-gpu-recording-rules.bats`, which cross-checks every
|
|
`DCGM_FI_*` reference in the tracked GPU dashboards and recording rules
|
|
against the union of the upstream default set (snapshotted at
|
|
`hack/dcgm-default-counters.csv` for the pinned DCGM Exporter version)
|
|
and the custom CSV in `dcgm-custom-metrics.yaml`. When the DCGM
|
|
Exporter image in `packages/system/gpu-operator/charts/gpu-operator/values.yaml`
|
|
is bumped, refresh the snapshot from the matching tag of the
|
|
[`NVIDIA/dcgm-exporter`][default-csv] repository and rerun the test.
|
|
|
|
[default-csv]: https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/default-counters.csv
|