## What this PR does
Makes Fluent Bit keep the pod lineage labels
`apps.cozystack.io/application.{group,kind,name}` on container log
lines, as the stream fields
`apps_cozystack_io_application_{group,kind,name}`, for both the default
container-log output and the per-tenant routed output.
Why: the per-application logs scope in the design proposal [Tenant
application logs](https://github.com/cozystack/community/pull/95)
(section 6) filters by these stream fields. Today the kubernetes filter
fetches pod labels, but they are dropped together with the rest of the
`kubernetes_*` keys before output.
What changes in the pipeline:
- A second `nest` lift of the pod labels, then a rename of the three
lineage keys to their final names before the existing `Remove_wildcard
kubernetes_` runs. All other labels are still removed. Every lifted key
is prefixed, so no other label can collide into the identity names.
- The three keys are added to `_stream_fields` of both container-log
outputs. Events and audit pipelines are unchanged.
- A parsed JSON log line is kept under `log_processed` (already on
`main`), so a line's content cannot create the identity keys.
- Guest Kubernetes clusters are not affected: their monitoring-agents
release replaces the filters and outputs wholesale. A regression test in
`packages/apps/kubernetes` pins that guest streams carry no identity
fields.
Effect on existing streams: VictoriaLogs treats a missing stream field
like an empty one, and the rename writes nothing when the label is
absent. Pods without lineage labels keep exactly the same stream. Pods
with lineage labels get a new stream once after the rollout; their
earlier lines stay reachable by namespace.
#### How it was tested
- helm-unittest for `monitoring-agents` (lift order, rename before
`Remove_wildcard`, stream fields on both outputs, events and audit
untouched) and for `kubernetes` (guest streams unchanged). The
events/audit guard was mutation-checked.
- Verified on a test cluster: real pod log lines arrive, Redis pods'
lines carry the three `apps_cozystack_io_application_*` stream fields,
and Fluent Bit logs show no errors.
#### Known limitation
Guest clusters write to the parent log store without authentication, so
a guest administrator can forge these stream fields as well as the
namespace field, within the store their tenant shares. This is the
write-path trust risk already documented in the proposal.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Downstream: no trigger matched. Candidate: none affected, because the
diff changes `packages/system/monitoring-agents/values.yaml` (a system
package with no schema) and adds tests; the file under
`packages/apps/kubernetes` is a test only, so the app's `values.yaml`
and `values.schema.json` are unchanged.
### Release note
```release-note
feat(monitoring-agents): container log lines of pods that belong to a Cozystack application now carry `apps_cozystack_io_application_{group,kind,name}` stream fields in VictoriaLogs. Streams of pods without application labels are unchanged; streams of application pods start a new stream once after the upgrade.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Container-log streams now include application group, kind, and name
metadata for root and tenant stores.
* Guest monitoring streams include pod name, container name, and
namespace fields.
* **Bug Fixes**
* Application metadata is added to container-log streams without
changing the existing stream fields for event and audit outputs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
A reader that scopes logs to one application needs the owning
application on every line, and the lineage webhook already puts it on
the pod as apps.cozystack.io/application.{group,kind,name}. Fluent Bit
fetched those labels and then dropped them with the rest of the
kubernetes map, so the store could only be scoped by namespace.
Keep the three labels as apps_cozystack_io_application_{group,kind,name},
the names the metrics side uses, and make them stream fields on every
container-log output so the scope is an indexed stream filter. A pod
without the labels produces no such fields, and VictoriaLogs builds the
stream from the fields present, so existing streams keep their
identity; a labelled pod starts a new stream once.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The tenant metrics API scopes a query to one application by looking up
its pods and PVCs through their apps.cozystack.io/application.* lineage
labels. kube-state-metrics emits kube_persistentvolumeclaim_labels only
for resources on --metric-labels-allowlist, so without PVCs there the
PVC half of that lookup always comes back empty and volume series of
an application fall out of its scope.
All labels are allowed, as for pods and deployments: kube-state-metrics
emits one series per PVC whatever keys are listed, and PVCs are far
fewer than pods, which already export every label.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
## What this PR does
Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.
```
docs/operations/backup-classes.md -> operations/services/backup-classes
docs/operations/resource-presets.md -> guides/resource-management
docs/storage-immutability.md -> operations/storage-class-immutability
docs/operations/system-memory-limits.md -> operations/configuration/system-memory-limits
docs/gateway-classes.md -> networking/gateway-classes
docs/observability/distributed-tracing.md -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md -> operations/services/object-storage/capacity-for-backups
```
`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.
References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.
Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.
### Downstream repositories
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763
Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.
Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.
### Release note
```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
Fluent Bit's kubernetes filter runs with Merge_Log On but without
Merge_Log_Key, so a parsed JSON log line is merged into the record root.
An application log line can thereby forge the service fields that later
pipeline stages rely on for routing and tenant isolation
(kubernetes_namespace_name, kubernetes.labels, tenant, cluster,
log_source). Setting Merge_Log_Key log_processed nests the parsed
payload under a dedicated key so application content can no longer
override top-level service fields.
Applies to both the platform (system/monitoring-agents) and guest
(apps/kubernetes) Fluent Bit configurations.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.
docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The sizing comment named the measured peak without saying how much
room 2Gi leaves above it, which left open why the chart pins 2Gi when
the default LimitRange limit proposed in #3494 is 4Gi.
2Gi is 6.7x the measured 307Mi peak working set. Read as linear in
object count, that is roughly 2100 pods and 170k series against the
322 pods and ~25.6k series the peak was measured on, so the limit
stays a ceiling rather than a bound on a cluster several times larger
than the one behind the numbers.
Comment only. The requests, the limit and the deliberate absence of a
CPU limit are all unchanged.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
It ran BestEffort with no resources at all, so it set no memory.max on
its pod cgroup and the Talos userspace OOM handler kept it permanently
in the victim set: any tenant workload driving node memory pressure
could evict it, and it is the source of every kube-state metric we
alert on.
Numbers are measured rather than guessed. A 14d peak on a 3-node
cluster with 322 pods and ~25.6k series is 307Mi working set, 280Mi
RSS, 176Mi mean. The request sits above the mean and the limit carries
wide headroom, because this component scales with object count and
--metric-labels-allowlist widens every pod and deployment series
further, so a larger cluster lands well above what was measured.
No CPU limit, following the node DaemonSet sizing in c9ba7a3cc: CPU is
irrelevant to the OOM ranking, and a binding ceiling on a scraper turns
into scrape timeouts rather than backpressure.
Supersedes #2632, whose 256Mi limit sits below the measured peak.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A tenant that runs its own monitoring stack gets a VictoriaLogs instance,
but the node fluent-bit sends every container log to global.target, so
that instance stays empty. The tenant chart already names the stack a
namespace reports to in namespace.cozystack.io/monitoring, the label the
per-tenant vmagent selects on, so the log path keys off the same label.
The routed record leaves the default outputs (keep false): a tenant with
its own monitoring holds its logs, the root store no longer does. The
re-tagged record re-enters the filter chain, where the Match * "Add
tenant" cannot overwrite the key, so a per-target Set labels it with the
tenant it is sent to. One rewrite_tag carries every rule, keeping a
single emitter however many tenants are routed.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
The per-job limit replaces the global flag downward as well, so an
install that raised promscrape.maxScrapeSize past it through extraArgs
has to raise this value too; say so where the value is set. Pin that a
numeric value still renders as a string, which the CRD field requires,
and pin the alert severity. Name the cluster in the alert and say where
the apiserver value lives, since a guest cluster cannot set it.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
A target whose response exceeds the scrape size limit reports up == 0,
which is indistinguishable from the target being down: TargetDown and
KubeAPIDown fire and nothing names the real cause. Alert on
vm_promscrape_max_scrape_size_exceeded_errors_total so the first alert
says what is wrong. The counter has no target label, so the description
points at the vmagent targets page to find the target.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
kube-apiserver's /metrics grows with the number of resources and CRDs,
and on a full install it outgrows the global 32MB
-promscrape.maxScrapeSize. vmagent then refuses every scrape of it, so
up is 0 and KubeAPIDown and TargetDown fire against a healthy API
server while every rule over API server metrics goes blind.
Set max_scrape_size on the apiserver endpoint instead of raising the
global flag. vmagent uses it in place of the flag for that job only, so
the 32MB guard for every other target stays as it is. vmagent reads the
whole uncompressed body into memory even with streamParse, so the
per-job value is a memory ceiling and stays finite (128MB), exposed as
vmagent.apiserverMaxScrapeSize.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## Summary
- Adds a `modify` filter in fluent-bit that removes
`kubernetes_node_name` and `kubernetes_host` from all container logs
- Prevents infrastructure topology (node names, hosts) from leaking to
tenant users
## Problem
`kubernetes_node_name` is exposed in log metadata, leaking node topology
to end-users. Other sensitive fields like `kubernetes_host` are also
visible.
Relates to #2194
## Test plan
- [ ] `helm template` monitoring-agents and verify `Remove
kubernetes_node_name` filter is present
- [ ] Deploy and verify `kubernetes_node_name` is absent from logs in
tenant Grafana
```release-note
Strip infrastructure metadata (node name, host) from tenant logs
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Improved Kubernetes monitoring log processing by filtering out most
`kubernetes_*` fields for `kube.*` events, while preserving the key
Kubernetes identity fields (such as pod, namespace, and
container-related names) for clearer, more consistent log output.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The Talos userspace OOM handler (v1.12+) ranks cgroups with
memory_max.hasValue() ? 0.0 :
{Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current
and discards every cgroup scoring zero, so only a memory limit takes a
pod out of the victim set. Selection is decoupled from the trigger: a
tenant workload thrashing against its own limit drives node memory PSI,
and a limit-free system DaemonSet is killed for it. metallb-speaker and
linstor-satellite were the observed victims.
Set requests and limits on the node DaemonSets that had none. Requests
come from 14d peak working set measured on a 3-node dev cluster; limits
are deliberately loose, sized as a runaway backstop rather than a fitted
ceiling, because several of these scale with cluster size that a dev
cluster understates and because undersizing converts a rare
pressure-driven kill into a deterministic one. A limit costs nothing
until it is hit -- only the request reserves.
memory.max lands on the POD cgroup only when EVERY container in the pod
carries a limit, so sidecars and init containers are covered too; one
limit-free container returns the whole pod to the victim set.
Two DaemonSets are not chart-rendered and needed their own seam:
linstor-satellite via the LinstorSatelliteConfiguration podTemplate, and
virt-handler via KubeVirt customizeComponents.patches, since virt-operator
owns that DaemonSet. Note linstor-controller stays deliberately limit-free
-- it is a JVM that must not be OOM-killed mid-operation, and cluster_test
pins that.
Two things the multus manifest needed beyond the values change.
customize-deployment.patch is the source for that template, so the
limits are carried there rather than only in the generated file, which
`make update` would discard. And the reason kube-multus had no memory
limit was that this patch DELETED upstream's 50Mi rather than raising
it -- 50Mi being well under the 230Mi measured peak -- which is what
removed memory.max from the pod cgroup in the first place.
Not covered, because upstream hardcodes the containers with no resources
knob: the four frr-k8s cp-* init containers, the kube-ovn hostpath-init
and install-cni init containers, and cozy-proxy, whose vendored chart
exposes no resources value at all. hami and kilo are left alone as
optional components with no usage measurements behind them. All of these
still get a limit from the default LimitRange the operator maintains in
every system namespace.
Every limit is pinned by a helm-unittest assertion, and every assertion
was mutation-tested: changing the rendered value turns its suite red.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/helm-unit-tests.sh discovers work by probing each package for a
`test` Makefile target, so a package without one is skipped in silence
however many suites sit under tests/.
monitoring-agents had tests/fluent-bit_probes_test.yaml and no target,
so that suite had never run once since it was written. It passes; the
only thing missing was the two lines that reach it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Allow-list the kubernetes_ fields kept on kube.* records so that
infrastructure topology (node name, host) no longer leaks to tenant
users, while preserving pod/namespace/container identity fields.
Relates to #2194
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
The fluent-bit DaemonSet crash-loops on a loaded install, timing out its
HelmRelease and cascading into velero and backupstrategy-controller
DependencyNotReady failures. The subchart default liveness/readiness probes
(initialDelaySeconds 0, timeoutSeconds 1, failureThreshold 3) target the HTTP
server on :2020, but that server does not bind until AFTER the kubernetes
filter's 'testing connectivity with API server' step. Under a concurrent
install the kube-API can be unreachable for ~10s, delaying the :2020 bind,
while the single-threaded HTTP server can miss a 1s timeout when the engine is
busy. The probes then fire into a connection-refused port and liveness-kill
fluent-bit into a restart loop.
Give both probes load headroom (initialDelaySeconds 15, timeoutSeconds 5,
failureThreshold 6) so the API-gated late bind and busy-engine blips are
tolerated. The httpGet target is unchanged (path / on port http = :2020). Add
a helm-unittest pinning the relaxed timing and the preserved target.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Bumps the etcd metrics scrape proxy from the 2021-era v0.11.0 to v0.22.0 on the
same quay.io/brancz registry that etcd-operator already standardized on. The two
flags used here (--secure-listen-address, --upstream) are core and stable across
the version range.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
In Auto mode the VPA Updater evicts running pods to apply new recommendations. During install that eviction churn recreates VictoriaMetrics pods faster than Cilium releases their endpoint IPs, tripping the cluster-pool IPAM "IP already in use" (putEndpointIdInvalid) leak that wedges a pod and stalls the install. Initial sets requests at pod creation without ever evicting, removing the trigger (the cilium-leak healer remains as defense for non-VPA churn paths).
Also raise vmstorage/vmselect minAllowed floors to 200m/512Mi: under Initial a pod is sized once at creation and never re-sized at runtime, so the floor is the safety net against the heavy query/storage tiers being left under-provisioned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml:119
Signed-off-by: Arsolitt <arsolitt@gmail.com>
DCGM exporter metrics carry the exporter's own namespace
(cozy-gpu-operator), not the workload namespace. Recording rules that
filtered namespace!~"cozy-.*" silently dropped all DCGM series,
producing empty dashboard panels.
Replace namespace-level hardware aggregations with node-level
equivalents (grouped by Hostname), keep namespace-level allocation
rules that use kube_pod_container_resource_requests (which carries the
real workload namespace), and rename pod-level efficiency rules to
gpu-level since DCGM cannot attribute hardware metrics to individual
pods.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The /1e9 divisor in gpu:{power,thermal}_throttle_fraction:rate5m was
derived empirically against DCGM 3.x on A10 — the counter documents
itself as microseconds but ticks in nanoseconds in practice. If a
future exporter release honors the documented units, pre-clamp values
would exceed 1.0 while clamp_max(..., 1) silently masks the drift,
plateauing every throttle fraction at 100% and making the panels
lie in unison.
Add a validation group that fires when the raw max/1e9 value exceeds
1.0 for 15m, so we notice and rescale to /1e6 before dashboards
silently mislead operators.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
pod:util_per_watt:avg5m divided two DCGM metrics without an explicit
on(...) clause, so the match used the intersection of their label
sets. If dcgm-exporter relabeling ever diverges between
DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_POWER_USAGE (e.g. a pod-mapping
label appears on one but not the other after a config change), the
entire result drops to empty silently. Pin the match to the labels we
group by so divergence becomes a missing side, not a missing rule.
Throttle fractions had a related shape problem: dcgm-exporter emits
one series per GPU for each pod-mapping combination. On a shared GPU
(restart races, MIG/MPS) the same physical counter appears under
multiple pod labels and downstream avg(...) panels get diluted by the
pod count. Fold duplicates with max by (Hostname, gpu, UUID) before
clamp_max so the fraction is tied to the physical GPU.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
kube-state-metrics keeps kube_pod_container_resource_requests series
for Failed/Succeeded pods until the apiserver garbage-collects them,
which could inflate :allocated beyond what tenants actually hold and
drive cluster:gpu_count:free negative.
Join the request metric against kube_pod_status_phase filtered to
Pending|Running — the canonical pattern from Kubernetes' own
container_resource recording rules — on both the cluster and namespace
aggregates. Add clamp_min(..., 0) on cluster:gpu_count:free as a
second line of defence against transient label drift between
kube-state-metrics and DCGM.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Drop the hardcoded metadata.namespace so the rule inherits the chart's
release namespace, and add an explicit empty params field on every
group for schema consistency. No behavior change.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Align pod:tensor_saturation:avg5m with namespace:tensor_active:avg and
DCGM's native 0..1 range by dropping the * 100 from the recording rule
and multiplying at display time in gpu-efficiency.json. Also scope
pod:util_per_watt:avg5m with avg by (Hostname, gpu, UUID, namespace,
pod) so the series mirrors pod:tensor_saturation's grouping and stays
usable in topk queries.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-efficiency: scope Tensor Saturation, Util-per-Watt and Power
Throttle stats to the $namespace selector. Cluster-wide means were
misleading when a user had narrowed the dashboard to specific
tenants — the headline numbers lied relative to the panels below.
- gpu-fleet: show per-node power draw as % of combined TDP cap
(DCGM_FI_DEV_POWER_MGMT_LIMIT) instead of raw watts. Thresholds
(60 / 80 %) generalize across GPU SKUs without per-model tuning.
- gpu-quotas: read cluster:gpu_count:allocated from the recording
rules instead of recomputing sum(kube_pod_container_resource_requests)
inline. Keeps the dashboard aligned with the canonical definition
in gpu-recording.rules.yaml so the two can't drift.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- namespace:gpu_count:sum — never consumed by any tracked dashboard;
the billable view is already covered by namespace:gpu_count:allocated,
and the admin view by cluster:gpu_count:allocated.
- namespace:energy_joules:sum — no panel integrates joules; kWh
readings on the tenant dashboard compute their own integrations
from namespace:power_watts:sum.
- pod:tensor_to_nvml_ratio:avg5m — interesting tenant signal in
theory, but not wired into any panel and carrying it just burns
cardinality on large fleets.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Align namespace:gpu_count:allocated with every other namespace:* rule
by filtering out cozy-*/kube-*. All other per-namespace rules
(gpu_util, tensor_active, fb_used_bytes, power_watts) already exclude
system namespaces, so the label set produced by :allocated diverged
from them — any dashboard variable or join that reads across these
rules could end up with a different namespace list depending on which
rule supplied the :allocated column.
Trade-off: per-namespace GPU accounting for system workloads is no
longer available through this rule. If it's ever needed, add a
dedicated system:gpu_count:allocated rather than widening this one —
the "billable tenant view" invariant is what the filter is protecting.
Cluster-level cluster:gpu_count:allocated intentionally keeps system
pods so it stays aligned with cluster:gpu_count:total and
cluster:gpu_count:free remains meaningful. As a consequence,
sum(namespace:gpu_count:allocated) no longer equals
cluster:gpu_count:allocated; the delta is system-pod GPU usage, which
is fine for the cluster-admin view.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- Drop gpu.recording.30s group: per-GPU 30s aggregates had no consumers
in tracked dashboards, only burned cardinality.
- Drop namespace:gpu_allocated_count:gauge: identical expression to
namespace:gpu_count:sum under a different name.
- Reground :allocated on kube_pod_container_resource_requests so it
reflects what tenants requested (Pending+Running) rather than what
DCGM currently sees. namespace:gpu_count:sum stays DCGM-based and
represents actually-running pods; the gap between the two is the
signal admins want.
- Add namespace:gpu_count:allocated as the per-namespace counterpart.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add VMRule with recording rules for DCGM metrics at three levels:
- gpu.recording.30s: per-GPU aggregates over 30s windows
- gpu.recording.cluster.1m: cluster-wide totals for overview panels
- gpu.recording.namespace.1m: per-namespace aggregates for tenant
reporting and GPU-hour calculations
The rules are safe to ship on clusters without DCGM — they evaluate to
empty series when no matching metrics are scraped.
Used by dashboards/gpu/gpu-performance.json.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Replace deprecated single-node VLogs CR with VLCluster (cluster mode)
for reliability and horizontal scalability.
Changes:
- Replace VLogs (v1beta1) with VLCluster (v1) using vlinsert/vlselect/vlstorage
- Update fluent-bit outputs to vlinsert-generic:9481
- Update Grafana datasource to vlselect:9471
- Update ExternalName service from vlogs-generic to vlinsert-generic
- Add VPA for all VLCluster components
- Update WorkloadMonitors for three-component architecture
- Add migration 35 to preserve old VLogs resources with keep annotation
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
PR #2075 added _cluster.cluster-domain references to monitoring-agents
values.yaml for FQDN resolution. This broke fluent-bit because it is
a subchart where _cluster values are not accessible (only global
values are shared with subcharts), causing "index of untyped nil".
Revert values.yaml to short DNS names (.svc) for the management
cluster where they resolve locally, and add FQDN with cluster-domain
suffix in the tenant kubernetes HelmRelease where _cluster values are
available and cross-cluster DNS resolution is needed.
Closes#2084
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Monitoring agents in tenant workload clusters failed to deliver metrics
and logs because service addresses used short DNS names without the
cluster domain suffix. Tenant CoreDNS could not resolve these names
across cluster boundaries.
Append the configured cluster domain from _cluster.cluster-domain to
all vmagent remoteWrite URLs and fluent-bit output hosts, falling back
to cluster.local when not set.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Replace unrendered Helm template expressions in values.yaml with
global.target parameter. Pass tenant-root from platform bundle,
default to cozy-monitoring. Use tpl in vmagent template to render URLs.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
- Replace hardcoded tenant-root with {{ .Release.Namespace }} in vmagent
and fluent-bit configs
- Add ExternalName services in cozystack-basics to redirect monitoring
traffic from cozy-monitoring to tenant-root when engine is deployed
- Add missing components to monitoring PackageSource
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
More granular k8s app extensions dependencies.
As a part of the change, packages were updated:
- monitoring-agents: metrics-server moved to a separate package
- monitoring-agents: prometheus-operator-crds moved to a separate
package
Main Cozystack bundles were updated to reflect packages changes.
### Release note
```release-note
Granular k8s extensions dependencies.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a standalone metrics-server release and a
prometheus-operator-crds release.
* **Improvements**
* Expanded Prometheus-related CRDs with new configuration options and
status subresources.
* Adjusted monitoring orchestration so components (cilium, metrics,
CRDs) install in a clearer order.
* **Refactor**
* Split metrics-server out of the monitoring-agents bundle and moved
Prometheus CRDs into a dedicated package.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This patch introduces a whitelist-based label filtering mechanism in
cadvisor/kubelet metrics collection. By explicitly keeping only the
desired labels, we avoid noisy and high-cardinality dimensions while
retaining meaningful CPU metrics for analysis.
This improves the stability of the metrics pipeline and ensures
consistent visibility into application workloads.
```release-note
[platform] Introduce whitelist label filtering for cadvisor/kubelet
metrics to reduce noise and improve CPU metric reliability.
```
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
This patch delivers changes to the monitoring config of Kube-OVN
plunger, which were accidentally omitted in its release, leading to a
duplicate service, broken monitoring agents' helm release and not
actually scraping the plunger.
```release-note
[kubeovn-plunger] Fix the VMServiceScrape object for collecting the
plunger's metrics.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This patch implements external monitoring of the Kube-OVN cluster. A new
reconciler timed to run its reconcile loop at a fixed interval execs
into the ovn-central pods and collects their cluster info. If the
members' opinions about the cluster disagree, an alert is raised. Other
issues with the distributed consensus are also highlighted.
```release-note
[kubeovn,cozystack-controller] Implement the KubeOVN plunger, an
external monitoring agent for the ovn-central cluster.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>