Aleksei Sviridkin
3977aef509
## What this PR does Adds GPU observability to Cozystack: five Grafana dashboards, a set of VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests for wiring the NVIDIA GPU Operator and DCGM Exporter on Talos when running GPU workloads directly in pods. ### Components 1. **Dashboards** — `dashboards/gpu/*.json`, registered in `packages/system/monitoring/dashboards-infra.list`. All follow the project convention: the Prometheus data source is selected through the `$ds_prometheus` template variable. - `gpu-performance` — per-GPU utilization (NVML, tensor pipe, graphics engine, memory copy), VRAM, power, temperature, and health (XID errors, power/thermal throttling). All metrics are shown per-GPU, filterable by the `$Hostname` selector. - `gpu-fleet` — cluster-wide inventory, capacity, utilization, and per-node power draw as percentage of TDP. - `gpu-tenants` — per-namespace allocation and 24-hour GPU-hour/kWh accounting; utilization and power are shown per-node (DCGM cannot attribute hardware metrics to namespaces). - `gpu-quotas` — allocated vs. requested GPUs, pending pods, and a per-namespace requests/limits table. All counts derive from the `namespace:gpu_count:allocated` recording rule, which excludes `Failed`/`Succeeded` pod carryover. - `gpu-efficiency` — cluster and per-GPU efficiency: tensor saturation, util-per-watt, and per-GPU power/thermal throttle fractions. 2. **Recording rules and alert** — `packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`. Four rule groups: - `gpu.recording.cluster.1m` — cluster-wide totals consumed by overview panels. - `gpu.recording.node.1m` — per-node hardware aggregates (grouped by `Hostname`) plus per-namespace allocation counts from kube resource requests; filters `cozy-*`/`kube-*`. - `gpu.recording.efficiency.1m` — per-GPU tensor saturation and util-per-watt, plus per-GPU power/thermal throttle fractions bounded via `clamp_max(..., 1)`. - `gpu.recording.throttle.validation.5m` — houses the `GPUThrottleFractionOverOne` warning alert that fires if the pre-clamp throttle fraction exceeds 1.0. This guards against DCGM counter-unit drift between GPU families; the `/1e9` divisor is verified on NVIDIA A10 with DCGM 3.x. Rules are safe on clusters without DCGM — they evaluate to empty series when no matching metrics are scraped. 3. **Reference manifests** under `packages/system/gpu-operator/examples/`. These are **not** templates; they document one working configuration: - `values-native-talos.yaml` — Cozystack Package values for the native-pod scenario. - `dcgm-custom-metrics.yaml` — ConfigMap extending the DCGM CSV with profiling, ECC, throttling, and energy counters used by the dashboards and recording rules. - `nvidia-driver-compat.yaml` — DaemonSet that stages `libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc tree into a location the GPU Operator validator inspects. Workaround for [NVIDIA/gpu-operator#1687](https://github.com/NVIDIA/gpu-operator/issues/1687). Requires `pod-security.kubernetes.io/enforce: privileged` on the target namespace when the cluster enforces baseline/restricted PodSecurityStandards. - `README.md` — explains the two deployment paths (sandbox vs. native), which DCGM metrics each dashboard depends on, and how the pieces fit together. 4. **Cross-validation test** — `hack/check-gpu-recording-rules.bats` enforces that every recording-rule reference in tracked dashboards resolves, and that every DCGM metric referenced by the rules is declared in either the upstream default CSV snapshot (`hack/dcgm-default-counters.csv`) or the custom CSV shipped under `examples/`. ### Why reference manifests, not templates The out-of-the-box `packages/system/gpu-operator/values-talos.yaml` targets **sandbox workloads** — GPUs passed through to KubeVirt VMs via VFIO, driver disabled on the host, device plugin disabled. That is the sensible default for Cozystack today. Running GPU workloads directly in pods on Talos is also possible but requires a different configuration that makes assumptions not every user shares: - the NVIDIA Talos system extension must be installed on GPU nodes, - the operator's own driver and toolkit components must be disabled because the extension provides them, - the validator's hardcoded paths need a workaround until NVIDIA/gpu-operator#1687 lands upstream. Shipping those as active templates would silently impose those assumptions. Shipping them as reference files next to the package lets operators opt in with eyes open and understand the moving parts before applying them. ### Scope note The dashboards and recording rules are useful independently of the reference manifests — any cluster that scrapes DCGM Exporter (however it is installed) will populate them. The reference manifests are self-contained and do not change any default behavior of the `gpu-operator` package. ### Screenshots <img width="2824" height="1055" alt="2026-04-27_12-36-52" src="https://github.com/user-attachments/assets/71f85de7-1c80-4817-b36c-aeb1c2bdd39a" /> <img width="2913" height="1255" alt="2026-04-27_12-36-39" src="https://github.com/user-attachments/assets/7a54daca-0727-4510-938c-080d31bdc229" /> <img width="2932" height="1251" alt="2026-04-27_12-36-30" src="https://github.com/user-attachments/assets/f6c6c215-3d05-400c-b21c-29f267c93f23" /> <img width="2864" height="1222" alt="2026-04-27_12-36-00" src="https://github.com/user-attachments/assets/e9d34bab-ade5-434b-92c3-f36712cf68d9" /> <img width="2958" height="1226" alt="image" src="https://github.com/user-attachments/assets/0047a5f8-12a5-4cda-8676-041de139cb76" /> ### Release note ```release-note feat(monitoring): add GPU observability — five Grafana dashboards (performance, fleet, tenants, quotas, efficiency), VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests under packages/system/gpu-operator/examples/ for running GPU workloads natively in pods on Talos. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added five GPU dashboards for performance, efficiency, fleet overview, quotas/allocation, and tenant usage (utilization, power, throttling, allocation, billing). * Added cluster recording rules and a throttling validation alert; new Prometheus-derived metrics and per-GPU rankings/timeseries. * **Documentation** * Added GPU Operator examples and README documenting native Talos setup and required DCGM counters. * **Tests** * Added validation tests ensuring dashboard ↔ recording-rule and DCGM metric consistency. * **Chores** * Registered new GPU dashboards in infra listings. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have Feature Requests please use the Discussion's Feature Request section.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.