mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
Aleksei Sviridkin
6e8566a94c
The cozystack.gpu-operator PackageSource ships three variants: default, vgpu and container. The iaas bundle's variant guard accepted only the first two, so selecting container failed the render on a value the package ships, values-container.yaml configures deliberately, and the variant bats suite already covers. The exclusion was coherent when it was written and stopped being so immediately. The container variant and the guard landed within minutes of each other from parallel branches, neither an ancestor of the other, variant first. Until the guard, the bundle hardcoded the default variant and container was selected by hand-crafting a Package CR, so leaving it out of a bundle-side list cost nothing. The guard then made the bundle the variant owner and closed that route without noticing it was one; its comment describes itself as typo protection and names passthrough as the example typo. The KubeVirt side is not a bare allowlist edit. The container variant serves GPUs to pods through the device plugin and runs no VM-facing GPU stack: sandboxWorkloads and vfioManager are both off and the host driver stays bound. Both KubeVirt-side pieces the bundle wires are VM-facing — the HostDevices feature gate advertises a capability such a node cannot serve, and permittedHostDevices lists resources no sandbox device plugin registers there. Both are now emitted only for default and vgpu, and container yields the gpu-operator Package with the kubevirt component untouched. Comments and docs that described the KubeVirt mirroring as unconditional on variant, or the container variant as unmerged and unavailable, are corrected to match. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
116 lines
7.5 KiB
Markdown
116 lines
7.5 KiB
Markdown
# GPU operator — native pod workload on Talos (reference)
|
|
|
|
The files in this directory are **not** templates. They are reference artifacts that document one working configuration for running GPU workloads directly in pods on a Talos-based Cozystack cluster, together with the DCGM metrics needed by the `gpu/gpu-performance` Grafana dashboard.
|
|
|
|
For non-Talos Linux hosts that already provide the NVIDIA driver and `nvidia-container-toolkit` via the distro package manager (the typical apt-installed Ubuntu/Debian shape), pick the first-class `container` variant instead — set `bundles.iaas.gpuOperatorVariant: container` in platform values and put `cozystack.gpu-operator` in `bundles.enabledPackages`, or hand-craft a Package CR with `variant: container` if you need overrides the bundle does not expose. Two host prerequisites still apply: containerd must have the `nvidia` runtime registered (`nvidia-ctk runtime configure --runtime=containerd` + restart; `apt install` lays down binaries only, it does not wire containerd), and because the variant runs with `driver.enabled=false` the operator uses the pre-installed host driver at its standard location — so a stock apt install needs no `hostPaths.driverInstallDir` override. The files here remain the right starting point on Talos, where the driver lands under a non-standard prefix and the operator does not find it at the default location without the override below.
|
|
|
|
The out-of-the-box `values-passthrough.yaml` for this package targets the sandbox (VFIO passthrough to KubeVirt VMs) scenario. The files here illustrate an alternative — running CUDA workloads in regular pods with the NVIDIA device plugin — and the workarounds it currently requires on Talos.
|
|
|
|
## Files
|
|
|
|
- [`values-native-talos.yaml`](./values-native-talos.yaml) — Cozystack
|
|
`Package` values that disable sandbox workloads, enable the device
|
|
plugin, point `hostPaths.driverInstallDir` at the staging location
|
|
used by the compat DaemonSet, and wire DCGM to the custom metrics
|
|
ConfigMap.
|
|
- [`dcgm-custom-metrics.yaml`](./dcgm-custom-metrics.yaml) — `ConfigMap`
|
|
with a DCGM metrics CSV that adds profiling, ECC, throttling and
|
|
energy counters on top of the upstream defaults. The CSV is a
|
|
superset needed for full coverage of the `gpu/gpu-performance`
|
|
dashboard. Which parts are actually required depends on which
|
|
dashboards you ship — see the table below.
|
|
- [`nvidia-driver-compat.yaml`](./nvidia-driver-compat.yaml) — DaemonSet
|
|
that stages `libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc
|
|
tree into a path where the NVIDIA GPU Operator validator expects
|
|
them. See the "Why the compat DaemonSet exists" section below.
|
|
|
|
## Why these are reference, not templates
|
|
|
|
Shipping these as first-class templates would silently impose
|
|
assumptions that do not hold for every user:
|
|
|
|
- Whether the NVIDIA Talos system extension is installed on the nodes.
|
|
- Whether GPUs are exposed directly to pods or passed through to VMs.
|
|
- The exact path the installed driver ends up at (depends on the
|
|
extension version and Talos release).
|
|
|
|
The sandbox-oriented `values-passthrough.yaml` remains the default. Operators
|
|
who want native pod GPU workloads can start from this directory and
|
|
adapt as needed.
|
|
|
|
## Why the compat DaemonSet exists
|
|
|
|
The NVIDIA GPU Operator validator checks for `libnvidia-ml.so.1` and
|
|
`bin/nvidia-smi` in the path given by `hostPaths.driverInstallDir`.
|
|
Talos installs them under `/usr/local/glibc/usr/lib/` and
|
|
`/usr/local/bin/`, which the validator does not look at. Until upstream
|
|
addresses [NVIDIA/gpu-operator#1687][1], the DaemonSet copies those
|
|
files into a directory the validator does inspect and creates the
|
|
`.driver-ctr-ready` flag file so the validator proceeds.
|
|
|
|
[1]: https://github.com/NVIDIA/gpu-operator/issues/1687
|
|
|
|
The compat DaemonSet runs privileged and bind-mounts host paths, so
|
|
the target namespace must allow privileged pods. On clusters that
|
|
enforce the Kubernetes Pod Security Standards at `baseline` or
|
|
`restricted`, label the namespace with
|
|
`pod-security.kubernetes.io/enforce: privileged` (and the matching
|
|
`audit`/`warn` labels if the admission webhook is configured to
|
|
surface violations) before applying the manifest.
|
|
|
|
## Dashboards and what DCGM metrics they need
|
|
|
|
Five GPU dashboards live under `gpu/*` in
|
|
`packages/system/monitoring/dashboards-infra.list`. All of them share
|
|
`packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml` as
|
|
their source of aggregated series. The recording rules are safe to
|
|
ship on any cluster — they evaluate to empty series when DCGM is not
|
|
scraped, or when optional counters are missing.
|
|
|
|
What each dashboard needs on top of the upstream DCGM Exporter
|
|
[`default-counters.csv`][default-csv]:
|
|
|
|
| Dashboard | Scope | Needs beyond defaults |
|
|
| ----------------- | ---------------------------------- | ----------------------------------------------------------------------- |
|
|
| `gpu-performance` | Per-node, per-GPU deep dive | `DCGM_FI_DEV_POWER_VIOLATION`, `DCGM_FI_DEV_THERMAL_VIOLATION` |
|
|
| `gpu-efficiency` | Per-workload util vs tensor active | `DCGM_FI_DEV_POWER_VIOLATION`, `DCGM_FI_DEV_THERMAL_VIOLATION` (via `gpu:*_throttle_fraction:rate5m` recording rules) |
|
|
| `gpu-fleet` | Cluster-wide admin inventory | `DCGM_FI_DEV_POWER_MGMT_LIMIT` (for the TDP vs draw panel) |
|
|
| `gpu-quotas` | Kube-quota vs live usage | `kube_pod_container_resource_requests`, `kube_pod_status_phase`, `kube_node_status_allocatable` (via `namespace:gpu_count:allocated` / `cluster:gpu_count:*` recording rules) |
|
|
| `gpu-tenants` | Per-namespace tenant view | nothing (works on default counters) |
|
|
|
|
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` and `DCGM_FI_PROF_GR_ENGINE_ACTIVE`
|
|
are already in the upstream default set for the pinned DCGM Exporter
|
|
version, so the tensor-saturation and engine-active panels work without
|
|
any CSV override. The three counters listed in the table — throttling
|
|
violations and the power management limit — are the only extras the
|
|
tracked dashboards need. The recording rules in
|
|
`gpu-recording.rules.yaml` consume utilization, FB used, power,
|
|
temperature and the tensor-active profiling counter from the default
|
|
set, plus `DCGM_FI_DEV_POWER_VIOLATION` and
|
|
`DCGM_FI_DEV_THERMAL_VIOLATION` — used by the
|
|
`gpu.recording.efficiency.1m` group to derive the
|
|
`gpu:power_throttle_fraction:rate5m` and
|
|
`gpu:thermal_throttle_fraction:rate5m` series consumed by the
|
|
throttling panels on the efficiency and fleet dashboards.
|
|
|
|
The `gpu.recording.throttle.validation.5m` group additionally ships the
|
|
`GPUThrottleFractionOverOne` alert (severity `warning`) as a regression
|
|
detector: it fires when either throttle-fraction series exceeds 1.0,
|
|
which would indicate that DCGM changed the scale/divisor of the
|
|
underlying violation counters and the recording rules need to be
|
|
re-derived.
|
|
|
|
## Verification status
|
|
|
|
The minimum-CSV claims above are verified by
|
|
`hack/check-gpu-recording-rules.bats`, which cross-checks every
|
|
`DCGM_FI_*` reference in the tracked GPU dashboards and recording rules
|
|
against the union of the upstream default set (snapshotted at
|
|
`hack/dcgm-default-counters.csv` for the pinned DCGM Exporter version)
|
|
and the custom CSV in `dcgm-custom-metrics.yaml`. When the DCGM
|
|
Exporter image in `packages/system/gpu-operator/charts/gpu-operator/values.yaml`
|
|
is bumped, refresh the snapshot from the matching tag of the
|
|
[`NVIDIA/dcgm-exporter`][default-csv] repository and rerun the test.
|
|
|
|
[default-csv]: https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/default-counters.csv
|