The controller commit-latency panel selected controller pods with
pod=~"$strimzi_cluster_name-controller-.*", but the controller node pool
is named c-<hash> (kafka.controllerPoolName), so Strimzi names controller
pods <cluster>-c-<hash>-<id>. The literal -controller- never matched, so
the panel was permanently empty. Select the pool by its -c- token instead.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The strimzi-kafka Grafana dashboard still had a 'ZK request latency' panel
(kafka_server_zookeeperclientmetrics_zookeeperrequestlatencyms), which goes
permanently empty after the ZooKeeper->KRaft migration. Replace it with a
KRaft controller commit-latency panel (kafka_server_raftmetrics_commit_latency_avg,
scraped from the controller pods). Metric names verified against a live KRaft
cluster. The PVC-exclusion filter that references zookeeper stays valid and is
left as is.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Address review findings on the KEDA read-replica autoscaler where the
implementation diverged from the merged design proposal.
Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG
instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or
replica — so the query is empty ONLY when no CNPG pod labels exist at all (a
broken kube-state-metrics pipeline), not during initial provisioning when the
sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin
`ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it
true, turning an EMPTY result into metric 0 (not an error) — the HPA would then
desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster
DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and
DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy
cluster (including a long basebackup) from tripping that alert.
Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips
are a merge no-op with no dip and never undercut the quorum floor.
Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently
raised both bounds, unbounded for a large maxSyncReplicas).
Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting
full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the
recommendation is read from the read-load metric, not a diverging desired count.
Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the
ScalingLimited condition (which is also true at the floor, so it fired on every
idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels
are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization
and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped
to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they
do not ship into the PrometheusRule.
Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection.
Regenerated README/schema/types.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The database-autoscaling dashboard hardcoded "datasource": "VictoriaMetrics" on
its namespace template variable, but cozystack provisions VictoriaMetrics
datasources as vm-shortterm / vm-longterm — there is no datasource named
"VictoriaMetrics", so on import the variable query failed ("datasource not
found"), the namespace variable never populated, and every panel (which filters
on namespace=~"$namespace") showed no data. Switch to the repo-standard pattern
used by db/cloudnativepg and the other dashboards: a `datasource`-typed template
variable (ds_prometheus) referenced as { type: prometheus, uid: ${ds_prometheus} }
from the namespace variable and every panel/target (the panels previously carried
no datasource at all and only worked by falling back to the default).
Also reframe the disable-wedge guard message to lead with the invariant (replicas
must exceed quorum.maxSyncReplicas) so a pure-quorum user who never enabled
autoscaling isn't told to "disable autoscaling"; the disable path is kept as a
parenthetical cause.
Addresses the adversarial branch-review findings on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The database-autoscaling dashboard shipped from the keda package as a ConfigMap
labeled grafana_dashboard: "1" for a Grafana sidecar that cozystack does not run
(no k8s-sidecar anywhere; every dashboard is provisioned by grafana-operator from
the grafana-dashboards image via GrafanaDashboard CRs). It was enabled by default
and never appeared in any Grafana. Move the JSON to dashboards/db/database-autoscaling.json
and register db/database-autoscaling in packages/system/monitoring/dashboards.list,
exactly how db/cloudnativepg and the other managed-db dashboards ship, and drop the
keda ConfigMap, its default-on toggle, and its wrong-mechanism test.
Addresses IvanHunters [MAJOR] review feedback on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Add a Grafana dashboard for KubeVirt virtual machines (CPU, memory,
network, disk I/O per VM) to the tenant-visible dashboards list.
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
## Problem
The **Capacity Planning** and **Namespaces** dashboards join
request/limit
metrics to `kube_pod_status_phase` via `* on (uid) group_left(phase)`.
Some pods (e.g. static pods like `ssh-proxy`) carry no `uid` label, so
every
such series collapses to the same empty match key and collides. The
query
backend rejects this with a 422:
> duplicate time series on the right side of `* on(uid)
group_left(phase)`
The affected panels render an error instead of data.
## Fix
Switch the join key to the always-unique `(namespace, pod)` and filter
`kube_pod_status_phase{phase="Running"} == 1` — the same pattern already
used
in `dashboards/gpu/gpu-quotas.json`. Sum results are unchanged (the old
multiply-by-0/1 is equivalent to filtering on `== 1`).
8 queries updated across two dashboards (CPU + memory, requests +
limits):
- `dashboards/main/capacity-planning.json` (6)
- `dashboards/main/namespaces.json` (2)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Corrected Capacity Planning dashboard CPU/Memory percentages and
unrequested-resource calculations so they consider only running pods.
* Fixed the “% Requested” and “Unrequested” panels for both CPU and
Memory to use consistent running-pod filtering and metric matching.
* Updated the Namespaces dashboard CPU and Memory repeated panels so
Requests and Limits reflect only running-pod metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
### Release note
```release-note
fix(dashboard): fix Capacity Planning and Namespaces dashboards failing with "duplicate time series" errors when static pods are present
```
The per-namespace Memory request/limit panels summed
kube_pod_container_resource_{requests,limits} over all pods regardless of
phase, while the sibling CPU panels were updated to multiply by
* on (namespace, pod) group_left(phase) (kube_pod_status_phase{phase="Running"} == 1).
Apply the same join so the paired CPU/Memory rows count the same pod population.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The == 1 phase filter drops the requests vector for nodes that have no
running pods, so the allocatable - requests subtraction silently omits
those nodes from the Unrequested CPU/Memory panels and totals. Default
the right-hand side to 0 per node via 'or sum by (node) (allocatable * 0)'.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Add a cross-tenant overview dashboard for the root/infra Grafana that
surfaces which tenants consume what resources, how many apps (Flux
HelmReleases) each runs, and per-tenant health.
Panels:
- Fleet summary stats (tenants, apps, pods, CPU/mem/storage)
- Tenant leaderboard table (CPU/mem usage vs requests, storage, pods,
PVCs, app count, not-ready apps)
- Top-N bar gauges by CPU, memory, storage and app count
- CPU/memory usage trends per tenant
- Pod restart rate and not-ready HelmReleases per tenant
Registered in dashboards-infra.list so it deploys only to the
root/cozy-monitoring Grafana, keeping cross-tenant data off per-tenant
instances. Uses standard cadvisor, kube-state-metrics and
gotk_resource_info metrics.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The capacity-planning and namespaces dashboards joined request/limit
metrics to kube_pod_status_phase via `* on (uid) group_left(phase)`.
Some pods (e.g. static pods like ssh-proxy) carry no `uid` label, so
they all collapse to the empty match key and collide, producing:
duplicate time series on the right side of `* on(uid) group_left(phase)`
and a 422 from the query backend.
Switch the join key to the always-unique `(namespace, pod)` and filter
`kube_pod_status_phase{phase="Running"} == 1`, matching the pattern
already used in dashboards/gpu/gpu-quotas.json. Sum results are
unchanged (the old multiply-by-0/1 is equivalent to filtering on == 1).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
DCGM exporter metrics carry the exporter's own namespace
(cozy-gpu-operator), not the workload namespace. Recording rules that
filtered namespace!~"cozy-.*" silently dropped all DCGM series,
producing empty dashboard panels.
Replace namespace-level hardware aggregations with node-level
equivalents (grouped by Hostname), keep namespace-level allocation
rules that use kube_pod_container_resource_requests (which carries the
real workload namespace), and rename pod-level efficiency rules to
gpu-level since DCGM cannot attribute hardware metrics to individual
pods.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The Pending GPU pods counter on gpu-quotas joined raw
kube_pod_container_resource_requests (per-container series) against
kube_pod_status_phase (per-pod series). Multi-container pods were
counted once per requesting container instead of once per pod, so the
widget over-reported whenever a Pending pod had more than one GPU
container. Collapse the requests to pod level before the join.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Pod-level panels on the efficiency dashboard and DCGM-level panels on
the performance dashboard ignored the $namespace template variable, so
changing it left the visualizations unchanged. Add the filter to each
query. Performance-side queries use the `$namespace|` empty-tolerant
form so host-level DCGM series without a namespace label remain
visible when a specific namespace is selected.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Clarify that the "Average utilization" panel on gpu-fleet reflects the
legacy NVML view (DCGM_FI_DEV_GPU_UTIL) rather than engine-active
profiling. For AI/LLM workloads the NVML number is optimistic; the
gpu-efficiency dashboard carries the profiling-based view.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Align pod:tensor_saturation:avg5m with namespace:tensor_active:avg and
DCGM's native 0..1 range by dropping the * 100 from the recording rule
and multiplying at display time in gpu-efficiency.json. Also scope
pod:util_per_watt:avg5m with avg by (Hostname, gpu, UUID, namespace,
pod) so the series mirrors pod:tensor_saturation's grouping and stays
usable in topk queries.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-efficiency: scope Tensor Saturation, Util-per-Watt and Power
Throttle stats to the $namespace selector. Cluster-wide means were
misleading when a user had narrowed the dashboard to specific
tenants — the headline numbers lied relative to the panels below.
- gpu-fleet: show per-node power draw as % of combined TDP cap
(DCGM_FI_DEV_POWER_MGMT_LIMIT) instead of raw watts. Thresholds
(60 / 80 %) generalize across GPU SKUs without per-model tuning.
- gpu-quotas: read cluster:gpu_count:allocated from the recording
rules instead of recomputing sum(kube_pod_container_resource_requests)
inline. Keeps the dashboard aligned with the canonical definition
in gpu-recording.rules.yaml so the two can't drift.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-fleet: cluster-wide admin view — inventory, capacity (total /
allocated / free), per-node utilization and power, throttling,
temperatures, XID errors.
- gpu-tenants: per-namespace view — live allocation, utilization,
tensor saturation, power, and 24h GPU-hours / kWh integrations for
billing inputs.
Register both under gpu/* in dashboards-infra.list so they ship as
GrafanaDashboard CRs and fall under the bats cross-check introduced
earlier on this branch.
Update examples/README to spell out which DCGM counters each of the
five gpu/* dashboards actually needs on top of the upstream default
CSV — gpu-performance needs profiling and throttling counters,
gpu-efficiency needs profiling, gpu-tenants needs only
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE for its tensor panel, and gpu-fleet
and gpu-quotas work on the default counter set alone.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Strip Grafana export boilerplate (__inputs, __elements, __requires,
default annotations, embedded datasource inputs) and tighten panel
layouts across the three GPU dashboards. All three continue to use
the $ds_prometheus template variable.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Revise gpu-performance and add two new dashboards, registered in
dashboards-infra.list:
- gpu-efficiency (GPU Efficiency Score) — utilization vs. capacity
and workload efficiency signals.
- gpu-quotas (GPU Quotas & Allocation) — per-namespace requested vs.
used GPUs for tenant capacity planning.
All three dashboards use the $ds_prometheus template variable, per
the project convention.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add the gpu/gpu-performance dashboard and register it in the infra
dashboard list. The dashboard provides:
- Cluster overview: total/allocated GPUs, average utilization,
aggregate power draw.
- Utilization: GPU util (NVML), tensor pipe active (realistic load
for LLM/AI workloads), graphics engine active, memory copy util.
- Memory: VRAM used/free per GPU.
- Power and temperature per GPU.
- Health: XID errors, power and thermal throttling.
The dashboard relies on DCGM_FI_* metrics plus the cluster:gpu_* and
namespace:gpu_* recording rules added to monitoring-agents.
The JSON follows the cozystack convention — Prometheus data source is
selected via the $ds_prometheus template variable.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add documentation explaining how to enable Hubble for network
observability in Grafana. Include four pre-built Hubble dashboards
(overview, dns-namespace, l7-http-metrics, network-overview) and
register them in the monitoring hub's dashboard list.
Closes#749
Signed-off-by: majiayu000 <1835304752@qq.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Expanded the monitored dashboards with a new storage dashboard entry.
- Introduced proactive alert configurations that cover key storage
components.
- Added templated alert management to streamline dynamic configuration.
- Enhanced metric collection by integrating monitoring endpoints for
storage components.
- Delivered a comprehensive dashboard offering real-time insights into
storage performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Co-authored-by: Andrei Kvapil <kvapss@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a comprehensive Grafana dashboard for Goldpinger, offering
real-time insights into node health, error occurrences, and response
times with intuitive filtering.
- Expanded deployment configurations to include Goldpinger across
environments, streamlining release management and dependency handling.
- Launched a dedicated deployment package featuring customizable
templates for secure, efficient Kubernetes deployments—including
workloads, services, ingress, and monitoring integrations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Andrey Kolkov