Commit Graph
63 Commits
Author SHA1 Message Date
Aleksei Sviridkin c1ba05574f feat(monitoring-agents): keep application identity in container log streams (#5217)
## What this PR does

Makes Fluent Bit keep the pod lineage labels
`apps.cozystack.io/application.{group,kind,name}` on container log
lines, as the stream fields
`apps_cozystack_io_application_{group,kind,name}`, for both the default
container-log output and the per-tenant routed output.

Why: the per-application logs scope in the design proposal [Tenant
application logs](https://github.com/cozystack/community/pull/95)
(section 6) filters by these stream fields. Today the kubernetes filter
fetches pod labels, but they are dropped together with the rest of the
`kubernetes_*` keys before output.

What changes in the pipeline:

- A second `nest` lift of the pod labels, then a rename of the three
lineage keys to their final names before the existing `Remove_wildcard
kubernetes_` runs. All other labels are still removed. Every lifted key
is prefixed, so no other label can collide into the identity names.
- The three keys are added to `_stream_fields` of both container-log
outputs. Events and audit pipelines are unchanged.
- A parsed JSON log line is kept under `log_processed` (already on
`main`), so a line's content cannot create the identity keys.
- Guest Kubernetes clusters are not affected: their monitoring-agents
release replaces the filters and outputs wholesale. A regression test in
`packages/apps/kubernetes` pins that guest streams carry no identity
fields.

Effect on existing streams: VictoriaLogs treats a missing stream field
like an empty one, and the rename writes nothing when the label is
absent. Pods without lineage labels keep exactly the same stream. Pods
with lineage labels get a new stream once after the rollout; their
earlier lines stay reachable by namespace.

#### How it was tested

- helm-unittest for `monitoring-agents` (lift order, rename before
`Remove_wildcard`, stream fields on both outputs, events and audit
untouched) and for `kubernetes` (guest streams unchanged). The
events/audit guard was mutation-checked.
- Verified on a test cluster: real pod log lines arrive, Redis pods'
lines carry the three `apps_cozystack_io_application_*` stream fields,
and Fluent Bit logs show no errors.

#### Known limitation

Guest clusters write to the parent log store without authentication, so
a guest administrator can forge these stream fields as well as the
namespace field, within the store their tenant shares. This is the
write-path trust risk already documented in the proposal.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff changes `packages/system/monitoring-agents/values.yaml` (a system
package with no schema) and adds tests; the file under
`packages/apps/kubernetes` is a test only, so the app's `values.yaml`
and `values.schema.json` are unchanged.

### Release note

```release-note
feat(monitoring-agents): container log lines of pods that belong to a Cozystack application now carry `apps_cozystack_io_application_{group,kind,name}` stream fields in VictoriaLogs. Streams of pods without application labels are unchanged; streams of application pods start a new stream once after the upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Container-log streams now include application group, kind, and name
metadata for root and tenant stores.
* Guest monitoring streams include pod name, container name, and
namespace fields.
* **Bug Fixes**
* Application metadata is added to container-log streams without
changing the existing stream fields for event and audit outputs.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-04 18:27:20 +03:00
ZverGuy e5f1714836 feat(monitoring): keep application identity on container log lines
A reader that scopes logs to one application needs the owning
application on every line, and the lineage webhook already puts it on
the pod as apps.cozystack.io/application.{group,kind,name}. Fluent Bit
fetched those labels and then dropped them with the rest of the
kubernetes map, so the store could only be scoped by namespace.

Keep the three labels as apps_cozystack_io_application_{group,kind,name},
the names the metrics side uses, and make them stream fields on every
container-log output so the scope is an indexed stream filter. A pod
without the labels produces no such fields, and VictoriaLogs builds the
stream from the fields present, so existing streams keep their
identity; a labelled pod starts a new stream once.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:18:09 +07:00
ZverGuy 7de70ac2fc feat(monitoring-agents): export PVC labels from kube-state-metrics
The tenant metrics API scopes a query to one application by looking up
its pods and PVCs through their apps.cozystack.io/application.* lineage
labels. kube-state-metrics emits kube_persistentvolumeclaim_labels only
for resources on --metric-labels-allowlist, so without PVCs there the
PVC half of that lookup always comes back empty and volume series of
an application fall out of its scope.

All labels are allowed, as for pods and deployments: kube-state-metrics
emits one series per PVC whatever keys are listed, and PVCs are far
fewer than pods, which already export every label.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:12:15 +07:00
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034)
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
2026-11-03 23:09:46 +05:00
IvanHunters 8d96c46994 fix(monitoring-agents): isolate merged log payload under log_processed
Fluent Bit's kubernetes filter runs with Merge_Log On but without
Merge_Log_Key, so a parsed JSON log line is merged into the record root.
An application log line can thereby forge the service fields that later
pipeline stages rely on for routing and tenant isolation
(kubernetes_namespace_name, kubernetes.labels, tenant, cluster,
log_source). Setting Merge_Log_Key log_processed nests the parsed
payload under a dedicated key so application content can no longer
override top-level service fields.

Applies to both the platform (system/monitoring-agents) and guest
(apps/kubernetes) Fluent Bit configurations.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-29 17:08:49 +03:00
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
gireassen 525814d399 fix: typo in monitoring-agents values.yaml
Signed-off-by: gireassen <80508012+gireassen@users.noreply.github.com>
2026-10-10 22:38:07 +03:00
Myasnikov Daniil 361311c5c9 docs(monitoring-agents): note what object count the 2Gi limit covers
The sizing comment named the measured peak without saying how much
room 2Gi leaves above it, which left open why the chart pins 2Gi when
the default LimitRange limit proposed in #3494 is 4Gi.

2Gi is 6.7x the measured 307Mi peak working set. Read as linear in
object count, that is roughly 2100 pods and 170k series against the
322 pods and ~25.6k series the peak was measured on, so the limit
stays a ceiling rather than a bound on a cluster several times larger
than the one behind the numbers.

Comment only. The requests, the limit and the deliberate absence of a
CPU limit are all unchanged.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 04:56:09 +03:00
Myasnikov Daniil 2e533acf4e fix(monitoring-agents): give kube-state-metrics a sized memory limit
It ran BestEffort with no resources at all, so it set no memory.max on
its pod cgroup and the Talos userspace OOM handler kept it permanently
in the victim set: any tenant workload driving node memory pressure
could evict it, and it is the source of every kube-state metric we
alert on.

Numbers are measured rather than guessed. A 14d peak on a 3-node
cluster with 322 pods and ~25.6k series is 307Mi working set, 280Mi
RSS, 176Mi mean. The request sits above the mean and the limit carries
wide headroom, because this component scales with object count and
--metric-labels-allowlist widens every pod and deployment series
further, so a larger cluster lands well above what was measured.

No CPU limit, following the node DaemonSet sizing in c9ba7a3cc: CPU is
irrelevant to the OOM ranking, and a binding ceiling on a scraper turns
into scrape timeouts rather than backpressure.

Supersedes #2632, whose 256Mi limit sits below the measured peak.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 04:56:03 +03:00
Mattia Eleuteri cdff5ed380 feat(monitoring): route tenant logs to the tenant's own VictoriaLogs
A tenant that runs its own monitoring stack gets a VictoriaLogs instance,
but the node fluent-bit sends every container log to global.target, so
that instance stays empty. The tenant chart already names the stack a
namespace reports to in namespace.cozystack.io/monitoring, the label the
per-tenant vmagent selects on, so the log path keys off the same label.

The routed record leaves the default outputs (keep false): a tenant with
its own monitoring holds its logs, the root store no longer does. The
re-tagged record re-enters the filter chain, where the Match * "Add
tenant" cannot overwrite the key, so a per-target Set labels it with the
tenant it is sent to. One rewrite_tag carries every rule, keeping a
single emitter however many tenants are routed.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:10:54 +02:00
IvanHunters 380c446cda fix(monitoring): pin the scrape size rendering and sharpen the alert
The per-job limit replaces the global flag downward as well, so an
install that raised promscrape.maxScrapeSize past it through extraArgs
has to raise this value too; say so where the value is set. Pin that a
numeric value still renders as a string, which the CRD field requires,
and pin the alert severity. Name the cluster in the alert and say where
the apiserver value lives, since a guest cluster cannot set it.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:56:07 +03:00
IvanHunters a17c5b2667 feat(monitoring): alert when vmagent refuses an oversized scrape
A target whose response exceeds the scrape size limit reports up == 0,
which is indistinguishable from the target being down: TargetDown and
KubeAPIDown fire and nothing names the real cause. Alert on
vm_promscrape_max_scrape_size_exceeded_errors_total so the first alert
says what is wrong. The counter has no target label, so the description
points at the vmagent targets page to find the target.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:28:29 +03:00
IvanHunters 4a268f6c01 fix(monitoring): give the kube-apiserver scrape its own size limit
kube-apiserver's /metrics grows with the number of resources and CRDs,
and on a full install it outgrows the global 32MB
-promscrape.maxScrapeSize. vmagent then refuses every scrape of it, so
up is 0 and KubeAPIDown and TargetDown fire against a healthy API
server while every rule over API server metrics goes blind.

Set max_scrape_size on the apiserver endpoint instead of raising the
global flag. vmagent uses it in place of the flag for that job only, so
the 32MB guard for every other target stays as it is. vmagent reads the
whole uncompressed body into memory even with streamParse, so the
per-job value is a memory ceiling and stays finite (128MB), exposed as
vmagent.apiserverMaxScrapeSize.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:27:00 +03:00
myasnikovdaniil b7e5100e73 fix(monitoring): strip infrastructure metadata from tenant logs (#2196)
## Summary

- Adds a `modify` filter in fluent-bit that removes
`kubernetes_node_name` and `kubernetes_host` from all container logs
- Prevents infrastructure topology (node names, hosts) from leaking to
tenant users

## Problem

`kubernetes_node_name` is exposed in log metadata, leaking node topology
to end-users. Other sensitive fields like `kubernetes_host` are also
visible.

Relates to #2194

## Test plan

- [ ] `helm template` monitoring-agents and verify `Remove
kubernetes_node_name` filter is present
- [ ] Deploy and verify `kubernetes_node_name` is absent from logs in
tenant Grafana

```release-note
Strip infrastructure metadata (node name, host) from tenant logs
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Improved Kubernetes monitoring log processing by filtering out most
`kubernetes_*` fields for `kube.*` events, while preserving the key
Kubernetes identity fields (such as pod, namespace, and
container-related names) for clearer, more consistent log output.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-04 16:16:24 +05:00
Myasnikov Daniil c9ba7a3cca fix(platform): give node DaemonSets memory requests and limits
The Talos userspace OOM handler (v1.12+) ranks cgroups with

  memory_max.hasValue() ? 0.0 :
    {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current

and discards every cgroup scoring zero, so only a memory limit takes a
pod out of the victim set. Selection is decoupled from the trigger: a
tenant workload thrashing against its own limit drives node memory PSI,
and a limit-free system DaemonSet is killed for it. metallb-speaker and
linstor-satellite were the observed victims.

Set requests and limits on the node DaemonSets that had none. Requests
come from 14d peak working set measured on a 3-node dev cluster; limits
are deliberately loose, sized as a runaway backstop rather than a fitted
ceiling, because several of these scale with cluster size that a dev
cluster understates and because undersizing converts a rare
pressure-driven kill into a deterministic one. A limit costs nothing
until it is hit -- only the request reserves.

memory.max lands on the POD cgroup only when EVERY container in the pod
carries a limit, so sidecars and init containers are covered too; one
limit-free container returns the whole pod to the victim set.

Two DaemonSets are not chart-rendered and needed their own seam:
linstor-satellite via the LinstorSatelliteConfiguration podTemplate, and
virt-handler via KubeVirt customizeComponents.patches, since virt-operator
owns that DaemonSet. Note linstor-controller stays deliberately limit-free
-- it is a JVM that must not be OOM-killed mid-operation, and cluster_test
pins that.

Two things the multus manifest needed beyond the values change.
customize-deployment.patch is the source for that template, so the
limits are carried there rather than only in the generated file, which
`make update` would discard. And the reason kube-multus had no memory
limit was that this patch DELETED upstream's 50Mi rather than raising
it -- 50Mi being well under the 230Mi measured peak -- which is what
removed memory.max from the pod cgroup in the first place.

Not covered, because upstream hardcodes the containers with no resources
knob: the four frr-k8s cp-* init containers, the kube-ovn hostpath-init
and install-cni init containers, and cozy-proxy, whose vendored chart
exposes no resources value at all. hami and kilo are left alone as
optional components with no usage measurements behind them. All of these
still get a limit from the default LimitRange the operator maintains in
every system namespace.

Every limit is pinned by a helm-unittest assertion, and every assertion
was mutation-tested: changing the rendered value turns its suite red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-13 16:47:52 +05:00
Myasnikov Daniil cf87201099 test(monitoring-agents): run this package's helm suite
hack/helm-unit-tests.sh discovers work by probing each package for a
`test` Makefile target, so a package without one is skipped in silence
however many suites sit under tests/.

monitoring-agents had tests/fluent-bit_probes_test.yaml and no target,
so that suite had never run once since it was written. It passes; the
only thing missing was the two lines that reach it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-13 16:47:35 +05:00
Mattia Eleuteri 0356cde906 fix(monitoring): strip infrastructure metadata from tenant logs
Allow-list the kubernetes_ fields kept on kube.* records so that
infrastructure topology (node name, host) no longer leaks to tenant
users, while preserving pod/namespace/container identity fields.

Relates to #2194

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-07-09 09:19:57 +02:00
Aleksei Sviridkin 45a273eba5 fix(monitoring-agents): relax fluent-bit probes for loaded installs
The fluent-bit DaemonSet crash-loops on a loaded install, timing out its
HelmRelease and cascading into velero and backupstrategy-controller
DependencyNotReady failures. The subchart default liveness/readiness probes
(initialDelaySeconds 0, timeoutSeconds 1, failureThreshold 3) target the HTTP
server on :2020, but that server does not bind until AFTER the kubernetes
filter's 'testing connectivity with API server' step. Under a concurrent
install the kube-API can be unreachable for ~10s, delaying the :2020 bind,
while the single-threaded HTTP server can miss a 1s timeout when the engine is
busy. The probes then fire into a connection-refused port and liveness-kill
fluent-bit into a restart loop.

Give both probes load headroom (initialDelaySeconds 15, timeoutSeconds 5,
failureThreshold 6) so the API-gated late bind and busy-engine blips are
tolerated. The httpGet target is unchanged (path / on port http = :2020). Add
a helm-unittest pinning the relaxed timing and the preserved target.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-24 23:38:38 +03:00
Aleksei Sviridkin 95706ad0be chore(packages/system/monitoring-agents): bump etcd-proxy kube-rbac-proxy to v0.22.0
Bumps the etcd metrics scrape proxy from the 2021-era v0.11.0 to v0.22.0 on the
same quay.io/brancz registry that etcd-operator already standardized on. The two
flags used here (--secure-listen-address, --upstream) are core and stable across
the version range.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-22 19:12:33 +03:00
Myasnikov DaniilandClaude Opus 4.8 0997105b05 fix(monitoring): set VPA updateMode to Initial to stop install-time eviction churn
In Auto mode the VPA Updater evicts running pods to apply new recommendations. During install that eviction churn recreates VictoriaMetrics pods faster than Cilium releases their endpoint IPs, tripping the cluster-pool IPAM "IP already in use" (putEndpointIdInvalid) leak that wedges a pod and stalls the install. Initial sets requests at pod creation without ever evicting, removing the trigger (the cilium-leak healer remains as defense for non-VPA churn paths).

Also raise vmstorage/vmselect minAllowed floors to 200m/512Mi: under Initial a pod is sized once at creation and never re-sized at runtime, so the floor is the safety net against the heavy query/storage tiers being left under-provisioned.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 12:57:23 +05:00
Arsolitt aba5ae3fcd fix(monitoring): prevent many-to-many match in util-per-watt recording rule
Address review feedback from coderabbitai on packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml:119

Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-28 11:21:12 +03:00
Arsolitt 27225f9e83 fix(monitoring): use node-level GPU metrics instead of namespace-level
DCGM exporter metrics carry the exporter's own namespace
(cozy-gpu-operator), not the workload namespace. Recording rules that
filtered namespace!~"cozy-.*" silently dropped all DCGM series,
producing empty dashboard panels.

Replace namespace-level hardware aggregations with node-level
equivalents (grouped by Hostname), keep namespace-level allocation
rules that use kube_pod_container_resource_requests (which carries the
real workload namespace), and rename pod-level efficiency rules to
gpu-level since DCGM cannot attribute hardware metrics to individual
pods.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-27 12:46:20 +03:00
Arsolitt f5f083e841 feat(monitoring): annotate GPUThrottleFractionOverOne with verified hardware
Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-19 11:09:51 +03:00
Arsolitt 5e8194c850 docs(monitoring): explain cluster-layer filter asymmetry in GPU rules
Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-19 11:09:50 +03:00
Arsolitt 165e175d70 feat(monitoring): alert on DCGM throttle divisor drift
The /1e9 divisor in gpu:{power,thermal}_throttle_fraction:rate5m was
derived empirically against DCGM 3.x on A10 — the counter documents
itself as microseconds but ticks in nanoseconds in practice. If a
future exporter release honors the documented units, pre-clamp values
would exceed 1.0 while clamp_max(..., 1) silently masks the drift,
plateauing every throttle fraction at 100% and making the panels
lie in unison.

Add a validation group that fires when the raw max/1e9 value exceeds
1.0 for 15m, so we notice and rescale to /1e6 before dashboards
silently mislead operators.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-18 08:11:57 +03:00
Arsolitt 605bcd338c fix(monitoring): pin label matching on GPU efficiency and throttle rules
pod:util_per_watt:avg5m divided two DCGM metrics without an explicit
on(...) clause, so the match used the intersection of their label
sets. If dcgm-exporter relabeling ever diverges between
DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_POWER_USAGE (e.g. a pod-mapping
label appears on one but not the other after a config change), the
entire result drops to empty silently. Pin the match to the labels we
group by so divergence becomes a missing side, not a missing rule.

Throttle fractions had a related shape problem: dcgm-exporter emits
one series per GPU for each pod-mapping combination. On a shared GPU
(restart races, MIG/MPS) the same physical counter appears under
multiple pod labels and downstream avg(...) panels get diluted by the
pod count. Fold duplicates with max by (Hostname, gpu, UUID) before
clamp_max so the fraction is tied to the physical GPU.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-18 08:03:21 +03:00
Arsolitt 0634654d63 fix(monitoring): exclude terminated pods from GPU allocation count
kube-state-metrics keeps kube_pod_container_resource_requests series
for Failed/Succeeded pods until the apiserver garbage-collects them,
which could inflate :allocated beyond what tenants actually hold and
drive cluster:gpu_count:free negative.

Join the request metric against kube_pod_status_phase filtered to
Pending|Running — the canonical pattern from Kubernetes' own
container_resource recording rules — on both the cluster and namespace
aggregates. Add clamp_min(..., 0) on cluster:gpu_count:free as a
second line of defence against transient label drift between
kube-state-metrics and DCGM.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-18 07:53:48 +03:00
Arsolitt 51b0dedd08 chore(monitoring): tidy GPU VMRule top-level structure
Drop the hardcoded metadata.namespace so the rule inherits the chart's
release namespace, and add an explicit empty params field on every
group for schema consistency. No behavior change.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-18 07:45:02 +03:00
Arsolitt 2518e09d67 refactor(monitoring): store pod:tensor_saturation as unitless ratio
Align pod:tensor_saturation:avg5m with namespace:tensor_active:avg and
DCGM's native 0..1 range by dropping the * 100 from the recording rule
and multiplying at display time in gpu-efficiency.json. Also scope
pod:util_per_watt:avg5m with avg by (Hostname, gpu, UUID, namespace,
pod) so the series mirrors pod:tensor_saturation's grouping and stays
usable in topk queries.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-18 07:03:42 +03:00
Arsolitt 2fa4b3e31c refactor(monitoring): tighten GPU dashboard queries
- gpu-efficiency: scope Tensor Saturation, Util-per-Watt and Power
  Throttle stats to the $namespace selector. Cluster-wide means were
  misleading when a user had narrowed the dashboard to specific
  tenants — the headline numbers lied relative to the panels below.
- gpu-fleet: show per-node power draw as % of combined TDP cap
  (DCGM_FI_DEV_POWER_MGMT_LIMIT) instead of raw watts. Thresholds
  (60 / 80 %) generalize across GPU SKUs without per-model tuning.
- gpu-quotas: read cluster:gpu_count:allocated from the recording
  rules instead of recomputing sum(kube_pod_container_resource_requests)
  inline. Keeps the dashboard aligned with the canonical definition
  in gpu-recording.rules.yaml so the two can't drift.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-18 06:48:33 +03:00
Arsolitt 7eb9fe8ade refactor(monitoring): drop unused GPU recording rules
- namespace:gpu_count:sum — never consumed by any tracked dashboard;
  the billable view is already covered by namespace:gpu_count:allocated,
  and the admin view by cluster:gpu_count:allocated.
- namespace:energy_joules:sum — no panel integrates joules; kWh
  readings on the tenant dashboard compute their own integrations
  from namespace:power_watts:sum.
- pod:tensor_to_nvml_ratio:avg5m — interesting tenant signal in
  theory, but not wired into any panel and carrying it just burns
  cardinality on large fleets.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-18 06:41:45 +03:00
Arsolitt dbed4992b0 fix(monitoring): exclude system namespaces from namespace:gpu_count:allocated
Align namespace:gpu_count:allocated with every other namespace:* rule
by filtering out cozy-*/kube-*. All other per-namespace rules
(gpu_util, tensor_active, fb_used_bytes, power_watts) already exclude
system namespaces, so the label set produced by :allocated diverged
from them — any dashboard variable or join that reads across these
rules could end up with a different namespace list depending on which
rule supplied the :allocated column.

Trade-off: per-namespace GPU accounting for system workloads is no
longer available through this rule. If it's ever needed, add a
dedicated system:gpu_count:allocated rather than widening this one —
the "billable tenant view" invariant is what the filter is protecting.

Cluster-level cluster:gpu_count:allocated intentionally keeps system
pods so it stays aligned with cluster:gpu_count:total and
cluster:gpu_count:free remains meaningful. As a consequence,
sum(namespace:gpu_count:allocated) no longer equals
cluster:gpu_count:allocated; the delta is system-pod GPU usage, which
is fine for the cluster-admin view.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-18 06:28:49 +03:00
Arsolitt 4e8731b588 refactor(monitoring): rework GPU recording rules
- Drop gpu.recording.30s group: per-GPU 30s aggregates had no consumers
  in tracked dashboards, only burned cardinality.
- Drop namespace:gpu_allocated_count:gauge: identical expression to
  namespace:gpu_count:sum under a different name.
- Reground :allocated on kube_pod_container_resource_requests so it
  reflects what tenants requested (Pending+Running) rather than what
  DCGM currently sees. namespace:gpu_count:sum stays DCGM-based and
  represents actually-running pods; the gap between the two is the
  signal admins want.
- Add namespace:gpu_count:allocated as the per-namespace counterpart.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-18 06:02:17 +03:00
Arsolitt cb3e0adf64 feat(monitoring): add GPU recording rules
Add VMRule with recording rules for DCGM metrics at three levels:

- gpu.recording.30s: per-GPU aggregates over 30s windows
- gpu.recording.cluster.1m: cluster-wide totals for overview panels
- gpu.recording.namespace.1m: per-namespace aggregates for tenant
  reporting and GPU-hour calculations

The rules are safe to ship on clusters without DCGM — they evaluate to
empty series when no matching metrics are scraped.

Used by dashboards/gpu/gpu-performance.json.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
2026-04-17 16:55:42 +03:00
Kirill IlinandClaude 62d36ec4ee feat(monitoring): migrate VictoriaLogs from VLogs to VLCluster
Replace deprecated single-node VLogs CR with VLCluster (cluster mode)
for reliability and horizontal scalability.

Changes:
- Replace VLogs (v1beta1) with VLCluster (v1) using vlinsert/vlselect/vlstorage
- Update fluent-bit outputs to vlinsert-generic:9481
- Update Grafana datasource to vlselect:9471
- Update ExternalName service from vlogs-generic to vlinsert-generic
- Add VPA for all VLCluster components
- Update WorkloadMonitors for three-component architecture
- Add migration 35 to preserve old VLogs resources with keep annotation

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-03-10 12:04:12 +05:00
Andrei KvapilandClaude d8bb3527de fix(monitoring-agents): revert _cluster FQDN and use cluster-domain in tenant HelmRelease
PR #2075 added _cluster.cluster-domain references to monitoring-agents
values.yaml for FQDN resolution. This broke fluent-bit because it is
a subchart where _cluster values are not accessible (only global
values are shared with subcharts), causing "index of untyped nil".

Revert values.yaml to short DNS names (.svc) for the management
cluster where they resolve locally, and add FQDN with cluster-domain
suffix in the tenant kubernetes HelmRelease where _cluster values are
available and cross-cluster DNS resolution is needed.

Closes #2084

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-23 16:15:36 +01:00
IvanHunters 0738fae56d fix(monitoring-agents): use FQDN with cluster domain for metrics and logs endpoints
Monitoring agents in tenant workload clusters failed to deliver metrics
and logs because service addresses used short DNS names without the
cluster domain suffix. Tenant CoreDNS could not resolve these names
across cluster boundaries.

Append the configured cluster domain from _cluster.cluster-domain to
all vmagent remoteWrite URLs and fluent-bit output hosts, falling back
to cluster.local when not set.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-02-19 18:30:20 +03:00
Andrei KvapilandClaude b8ccdedbf8 [monitoring] Fix YAML parse error in monitoring-agents vmagent template
Replace unrendered Helm template expressions in values.yaml with
global.target parameter. Pass tenant-root from platform bundle,
default to cozy-monitoring. Use tpl in vmagent template to render URLs.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-12 00:22:18 +01:00
IvanHuntersandClaude bb638f3447 fix(monitoring): parametrize namespace for monitoring-agents
- Replace hardcoded tenant-root with {{ .Release.Namespace }} in vmagent
  and fluent-bit configs
- Add ExternalName services in cozystack-basics to redirect monitoring
  traffic from cozy-monitoring to tenant-root when engine is deployed
- Add missing components to monitoring PackageSource

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-02-06 18:33:13 +03:00
Kirill Ilin 207a5171f0 [monitoring-agents] Set minReplicas to 1 for VPA for VMAgent
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-01-20 18:00:13 +05:00
Andrei KvapilandClaude 3618abed62 refactor: move scripts to hack directory
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-15 16:06:56 +01:00
Andrei Kvapil b1a55f5a38 Granular kubernetes application extensions dependencies (#1683)
## What this PR does
More granular k8s app extensions dependencies.

As a part of the change, packages were updated:
- monitoring-agents: metrics-server moved to a separate package
- monitoring-agents: prometheus-operator-crds moved to a separate
package

Main Cozystack bundles were updated to reflect packages changes. 

### Release note
```release-note
Granular k8s extensions dependencies.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a standalone metrics-server release and a
prometheus-operator-crds release.

* **Improvements**
* Expanded Prometheus-related CRDs with new configuration options and
status subresources.
* Adjusted monitoring orchestration so components (cilium, metrics,
CRDs) install in a clearer order.

* **Refactor**
* Split metrics-server out of the monitoring-agents bundle and moved
Prometheus CRDs into a dedicated package.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-05 14:44:53 +01:00
nbykov0 67e47256e2 [system:monitoring-agents] separate metrics-server
Signed-off-by: nbykov0 <166552198+nbykov0@users.noreply.github.com>
2025-12-01 20:51:26 +03:00
nbykov0 644d71eef7 [system:monitoring-agents] rename coredns metrics service
Signed-off-by: nbykov0 <166552198+nbykov0@users.noreply.github.com>
2025-12-01 17:50:02 +03:00
IvanHunters 8601299a91 [platform] Add whitelist for labels in cadvisor/kubelet metrics
This patch introduces a whitelist-based label filtering mechanism in
cadvisor/kubelet metrics collection. By explicitly keeping only the
desired labels, we avoid noisy and high-cardinality dimensions while
retaining meaningful CPU metrics for analysis.

This improves the stability of the metrics pipeline and ensures
consistent visibility into application workloads.

```release-note
[platform] Introduce whitelist label filtering for cadvisor/kubelet
metrics to reduce noise and improve CPU metric reliability.
```

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2025-09-30 11:04:19 +03:00
Timofei Larkin 08d2d61f1a [kubeovn] Fix service scrape for plunger
This patch delivers changes to the monitoring config of Kube-OVN
plunger, which were accidentally omitted in its release, leading to a
duplicate service, broken monitoring agents' helm release and not
actually scraping the plunger.

```release-note
[kubeovn-plunger] Fix the VMServiceScrape object for collecting the
plunger's metrics.
```

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2025-09-15 10:50:16 +03:00
Timofei Larkin 382a9787f4 [kubeovn] Implement the KubeOVN plunger
This patch implements external monitoring of the Kube-OVN cluster. A new
reconciler timed to run its reconcile loop at a fixed interval execs
into the ovn-central pods and collects their cluster info. If the
members' opinions about the cluster disagree, an alert is raised. Other
issues with the distributed consensus are also highlighted.

```release-note
[kubeovn,cozystack-controller] Implement the KubeOVN plunger, an
external monitoring agent for the ovn-central cluster.
```

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2025-09-11 02:11:58 +03:00
Timofei Larkin 62a6da0063 Make VMAgent extraArgs tunable
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2025-06-23 11:15:42 +03:00
Andrei Kvapil dfd01ff118 [platform] Fix deps for paas-hosted bundle
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-06-09 10:12:06 +02:00
kevin880202 fc8b52d73d reset and add audit/event monitoring in fluentbit values
Signed-off-by: kevin880202 <dytoponts11@gmail.com>
2025-05-21 22:07:27 +08:00