mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-04 11:07:03 +09:00
Aleksei Sviridkin
3977aef509
## What this PR does Adds GPU observability to Cozystack: five Grafana dashboards, a set of VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests for wiring the NVIDIA GPU Operator and DCGM Exporter on Talos when running GPU workloads directly in pods. ### Components 1. **Dashboards** — `dashboards/gpu/*.json`, registered in `packages/system/monitoring/dashboards-infra.list`. All follow the project convention: the Prometheus data source is selected through the `$ds_prometheus` template variable. - `gpu-performance` — per-GPU utilization (NVML, tensor pipe, graphics engine, memory copy), VRAM, power, temperature, and health (XID errors, power/thermal throttling). All metrics are shown per-GPU, filterable by the `$Hostname` selector. - `gpu-fleet` — cluster-wide inventory, capacity, utilization, and per-node power draw as percentage of TDP. - `gpu-tenants` — per-namespace allocation and 24-hour GPU-hour/kWh accounting; utilization and power are shown per-node (DCGM cannot attribute hardware metrics to namespaces). - `gpu-quotas` — allocated vs. requested GPUs, pending pods, and a per-namespace requests/limits table. All counts derive from the `namespace:gpu_count:allocated` recording rule, which excludes `Failed`/`Succeeded` pod carryover. - `gpu-efficiency` — cluster and per-GPU efficiency: tensor saturation, util-per-watt, and per-GPU power/thermal throttle fractions. 2. **Recording rules and alert** — `packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`. Four rule groups: - `gpu.recording.cluster.1m` — cluster-wide totals consumed by overview panels. - `gpu.recording.node.1m` — per-node hardware aggregates (grouped by `Hostname`) plus per-namespace allocation counts from kube resource requests; filters `cozy-*`/`kube-*`. - `gpu.recording.efficiency.1m` — per-GPU tensor saturation and util-per-watt, plus per-GPU power/thermal throttle fractions bounded via `clamp_max(..., 1)`. - `gpu.recording.throttle.validation.5m` — houses the `GPUThrottleFractionOverOne` warning alert that fires if the pre-clamp throttle fraction exceeds 1.0. This guards against DCGM counter-unit drift between GPU families; the `/1e9` divisor is verified on NVIDIA A10 with DCGM 3.x. Rules are safe on clusters without DCGM — they evaluate to empty series when no matching metrics are scraped. 3. **Reference manifests** under `packages/system/gpu-operator/examples/`. These are **not** templates; they document one working configuration: - `values-native-talos.yaml` — Cozystack Package values for the native-pod scenario. - `dcgm-custom-metrics.yaml` — ConfigMap extending the DCGM CSV with profiling, ECC, throttling, and energy counters used by the dashboards and recording rules. - `nvidia-driver-compat.yaml` — DaemonSet that stages `libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc tree into a location the GPU Operator validator inspects. Workaround for [NVIDIA/gpu-operator#1687](https://github.com/NVIDIA/gpu-operator/issues/1687). Requires `pod-security.kubernetes.io/enforce: privileged` on the target namespace when the cluster enforces baseline/restricted PodSecurityStandards. - `README.md` — explains the two deployment paths (sandbox vs. native), which DCGM metrics each dashboard depends on, and how the pieces fit together. 4. **Cross-validation test** — `hack/check-gpu-recording-rules.bats` enforces that every recording-rule reference in tracked dashboards resolves, and that every DCGM metric referenced by the rules is declared in either the upstream default CSV snapshot (`hack/dcgm-default-counters.csv`) or the custom CSV shipped under `examples/`. ### Why reference manifests, not templates The out-of-the-box `packages/system/gpu-operator/values-talos.yaml` targets **sandbox workloads** — GPUs passed through to KubeVirt VMs via VFIO, driver disabled on the host, device plugin disabled. That is the sensible default for Cozystack today. Running GPU workloads directly in pods on Talos is also possible but requires a different configuration that makes assumptions not every user shares: - the NVIDIA Talos system extension must be installed on GPU nodes, - the operator's own driver and toolkit components must be disabled because the extension provides them, - the validator's hardcoded paths need a workaround until NVIDIA/gpu-operator#1687 lands upstream. Shipping those as active templates would silently impose those assumptions. Shipping them as reference files next to the package lets operators opt in with eyes open and understand the moving parts before applying them. ### Scope note The dashboards and recording rules are useful independently of the reference manifests — any cluster that scrapes DCGM Exporter (however it is installed) will populate them. The reference manifests are self-contained and do not change any default behavior of the `gpu-operator` package. ### Screenshots <img width="2824" height="1055" alt="2026-04-27_12-36-52" src="https://github.com/user-attachments/assets/71f85de7-1c80-4817-b36c-aeb1c2bdd39a" /> <img width="2913" height="1255" alt="2026-04-27_12-36-39" src="https://github.com/user-attachments/assets/7a54daca-0727-4510-938c-080d31bdc229" /> <img width="2932" height="1251" alt="2026-04-27_12-36-30" src="https://github.com/user-attachments/assets/f6c6c215-3d05-400c-b21c-29f267c93f23" /> <img width="2864" height="1222" alt="2026-04-27_12-36-00" src="https://github.com/user-attachments/assets/e9d34bab-ade5-434b-92c3-f36712cf68d9" /> <img width="2958" height="1226" alt="image" src="https://github.com/user-attachments/assets/0047a5f8-12a5-4cda-8676-041de139cb76" /> ### Release note ```release-note feat(monitoring): add GPU observability — five Grafana dashboards (performance, fleet, tenants, quotas, efficiency), VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests under packages/system/gpu-operator/examples/ for running GPU workloads natively in pods on Talos. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added five GPU dashboards for performance, efficiency, fleet overview, quotas/allocation, and tenant usage (utilization, power, throttling, allocation, billing). * Added cluster recording rules and a throttling validation alert; new Prometheus-derived metrics and per-GPU rankings/timeseries. * **Documentation** * Added GPU Operator examples and README documenting native Talos setup and required DCGM counters. * **Tests** * Added validation tests ensuring dashboard ↔ recording-rule and DCGM metric consistency. * **Chores** * Registered new GPU dashboards in infra listings. <!-- end of auto-generated comment: release notes by coderabbit.ai -->