The Pending GPU pods counter on gpu-quotas joined raw
kube_pod_container_resource_requests (per-container series) against
kube_pod_status_phase (per-pod series). Multi-container pods were
counted once per requesting container instead of once per pod, so the
widget over-reported whenever a Pending pod had more than one GPU
container. Collapse the requests to pod level before the join.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-efficiency: scope Tensor Saturation, Util-per-Watt and Power
Throttle stats to the $namespace selector. Cluster-wide means were
misleading when a user had narrowed the dashboard to specific
tenants — the headline numbers lied relative to the panels below.
- gpu-fleet: show per-node power draw as % of combined TDP cap
(DCGM_FI_DEV_POWER_MGMT_LIMIT) instead of raw watts. Thresholds
(60 / 80 %) generalize across GPU SKUs without per-model tuning.
- gpu-quotas: read cluster:gpu_count:allocated from the recording
rules instead of recomputing sum(kube_pod_container_resource_requests)
inline. Keeps the dashboard aligned with the canonical definition
in gpu-recording.rules.yaml so the two can't drift.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Strip Grafana export boilerplate (__inputs, __elements, __requires,
default annotations, embedded datasource inputs) and tighten panel
layouts across the three GPU dashboards. All three continue to use
the $ds_prometheus template variable.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Revise gpu-performance and add two new dashboards, registered in
dashboards-infra.list:
- gpu-efficiency (GPU Efficiency Score) — utilization vs. capacity
and workload efficiency signals.
- gpu-quotas (GPU Quotas & Allocation) — per-namespace requested vs.
used GPUs for tenant capacity planning.
All three dashboards use the $ds_prometheus template variable, per
the project convention.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Arsolitt