Commit Graph
60 Commits
Author SHA1 Message Date
Aleksei Sviridkin 4f19e038c9 Merge branch 'main' into fix/monitoring-vmalert-scrape
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:18 +03:00
Aleksei Sviridkin b6f5a4fa35 feat(monitoring): stamp application identity on scraped series (#5216)
## What this PR does

Adds a default scrape class `application-identity` to both VMAgents: the
cluster-wide one in `monitoring-agents` and the per-tenant one in
`monitoring`. Every series scraped from a pod that carries the lineage
labels `apps.cozystack.io/application.{group,kind,name}` gets them as
`apps_cozystack_io_application_{group,kind,name}`.

Why: the per-application metrics scope in the design proposal [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94) (Design 4a)
needs to tell which application an exporter series belongs to. Today
that is only possible indirectly, through pod names.

How the labels are protected against forgery by an exporter:

- At the target stage, the class always sets all three labels: to the
pod's real lineage value, or to a placeholder `-` when the pod has none
(`-` is not a valid Kubernetes label value). With `honorLabels: false`,
any copy the exporter publishes therefore collides with the target label
and is renamed to `exported_*`.
- At the metric stage, the class drops every `exported_*` copy.
- The placeholder is removed by the VMAgent's global remote-write
relabeling (`inlineRelabelConfig`), not at the metric stage: vmagent
adds `up` and `scrape_*` with the target labels after metric relabeling,
so only the remote-write stage sees those series. Targets without
lineage therefore keep exactly the label set they have today on every
series, including `up` and `scrape_*`.
- With `honorLabels: true` the exporter's value wins and no relabel rule
can tell it apart. ingress-nginx scrapes with `honorLabels: true`, and
its series describe Ingress objects in any namespace, so the
controller's identity would misattribute traffic; its scrape now drops
the identity labels explicitly, from the target (so `up` and `scrape_*`
stay unchanged) and from the scraped samples (so an exporter's copy
cannot survive). Switching it to `honorLabels: false` was not done,
because that renames `namespace` and breaks the existing ingress
dashboards.

#### How it was tested

- helm-unittest for `monitoring-agents`, `monitoring` and `ingress`.
These tests check the rendered config only; the relabeling behaviour was
verified offline by running vmagent v1.136.0, built from source, against
the rendered rules: `up` and `scrape_*` carry no identity for targets
without lineage, and the placeholder appears on no series.
- The rendered `scrapeClasses` validate against the vendored VMAgent CRD
schema.
- The rendered rules were exercised on a locally built vmagent v1.136.0
(the version the bundled operator deploys) writing to a local
VictoriaMetrics: with and without lineage, with and without a forging
exporter, `honorLabels` true and false, and the ingress-nginx endpoint
rules. Checked on exporter samples and on `up`, `scrape_samples_scraped`
and `scrape_series_added`; the previous rules reproduce the `-`
placeholder on the generated series of targets without lineage, the new
ones leave no placeholder on any series.
- Verified on a test cluster with the earlier revision that removed the
placeholder at the metric stage (the move to remote-write relabeling is
not yet verified on a cluster): series carry the application labels;
metrics read for a Redis instance through the per-application proxy root
contained only that instance's pods; vmagent logs show no errors.

#### Known limitations

- Scrapes with `honorLabels: true` other than ingress-nginx are not
protected against an exporter that publishes the identity labels itself.
Today these are the vendored qdrant and harbor ServiceMonitors; their
exporters are platform images with fixed label names, and their series
describe the application's own pods.
- A tenant's raw `vmagent.inlineScrapeConfig` bypasses scrape classes
entirely, and so do `additionalScrapeConfigs`. VMProbe takes only auth
settings from a scrape class (operator v0.68.4); no VMProbe exists in
the platform today.
- A scrape object's own relabel rules run after the class and can
overwrite the labels; tenants cannot create VictoriaMetrics operator
objects, so this is platform-controlled.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff changes templates and tests in `packages/system/monitoring-agents`,
`packages/system/monitoring` and `packages/extra/ingress` without
adding, renaming or removing a package and without touching any
`values.yaml` or schema. The telemetry metrics the telemetry server
reads (`cozy_cluster_info`, `cozy_application_count`) are produced by
the operator, not by these scrapes. The proposal text in
`cozystack/community` (#94, Design 4a) should be corrected to name
qdrant and harbor as further `honorLabels: true` scrapes; that is not a
trigger-map item.

### Release note

```release-note
feat(monitoring): VMAgent now adds `apps_cozystack_io_application_{group,kind,name}` labels to every series scraped from a pod that carries application lineage labels. Because the label set of those series changes, each of them becomes a new series after the upgrade: graphs and dashboards show a break at the rollout, queries over a range that spans it see two series per pod, and alert `for:` timers on those series restart. Series of targets without lineage labels, including node-level series (cAdvisor, kubelet) and kube-state-metrics, and all ingress-nginx series keep their current label set; this includes their `up` and `scrape_*` series.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* Monitoring metrics now use application identity labels derived from
Kubernetes pod metadata for covered scrape targets. Exporter-supplied
duplicates and placeholder values are removed, including from `up` and
`scrape_*` metrics.
* Ingress metrics no longer retain application identity labels at either
target or metric relabeling stages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 12:03:57 +03:00
Alexey Artamonov 275c840a8c fix(monitoring): leave Grafana's ingress out in a tenant without ingress
grafana-operator creates the Ingress from spec.ingress verbatim, so an
empty _namespace.ingress reached the API as ingressClassName "" and was
rejected. With no ingress-nginx in the tenant there is nothing for that
Ingress to attach to.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Marian Koreniuk e92bc8664b fix(tests): keep two unit tests green after the release image restamp
The release build rewrites packages/*/values.yaml: the kube-ovn image tag
becomes the Cozystack version and the traces collector becomes the
first-party otelcol-tracing build. Two unit tests read exactly those
fields, so every PR against release-1.7 has failed Unit & controller
tests since the first rc, and release-1.8 would repeat it.

Same change as #5179 on release-1.7.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-11-08 19:37:03 +01:00
ZverGuy 7f7496c8d3 fix(monitoring): remove the identity placeholder from scrape-health series
vmagent appends up and scrape_* to each scrape after metric relabeling:
in VictoriaMetrics v1.136.0, the version operator v0.68.4 deploys,
lib/promscrape/scrapework.go addAutoMetrics passes needRelabel=false to
writeRequestCtx.addRows, so addRow attaches the target labels but skips
MetricRelabelConfigs. The "-" placeholder that the scrape class sets on
targets without lineage was removed only in metricRelabelConfigs, so it
stayed on the scrape-health series of every such target, including
nodes and kube-state-metrics.

The removal now runs in the VMAgent's inlineRelabelConfig, which the
operator passes as -remoteWrite.relabelConfig. app/vmagent/remotewrite
TryPush applies it to every series before remote write, scraped and
generated alike, so targets without lineage keep the label set they had
before the class. The exported_* drop stays in the class: only scraped
samples can carry it.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-05 00:16:36 +07:00
ZverGuy 5a6086c7a1 test(monitoring): pin the endpoint count the ingress identity test relies on
The ingress assertion reads only the first endpoint of each scrape, so a
second endpoint added without the label drop would go unnoticed. Pinning
the count makes such an addition fail the suite.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:30:32 +07:00
ZverGuy 14f815a5cc feat(monitoring): stamp application identity on tenant vmagent scrapes
Application pods in a tenant's namespaces are also scraped by the
tenant's own vmagent, and its series need the same identity labels and
the same protection against an exporter choosing them, or the two stores
would disagree about which application a series belongs to. The class
is identical to the cluster-wide one in monitoring-agents.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:16:56 +07:00
Yan Bondarenko 7bda2bdbc0 fix(monitoring): select the vmalert Service the operator creates
The VMServiceScrape selected app.kubernetes.io/instance=vmalert, while the
chart names the VMAlert vmalert-<first metricsStorages name> and the
operator puts that name into the instance label of its Service. The scrape
matched nothing, so the metrics of vmalert itself were never collected.

The instance stays in the selector so the scrape keeps pointing at the one
VMAlert this release renders.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-25 16:30:14 +03:00
Yan Bondarenko 9d4221b890 fix(monitoring): send alerts to every Alertmanager replica
vmalert had a single notifier, the headless vmalertmanager-alertmanager
Service. It resolves to all three pods, but each alert reached one replica
at a time and moved to another when vmalert reconnected. On clusters running
1.6.4 all active alerts sat on one replica while the other two had none, and
on one of them the alert count of each replica changed about 170 times a
day. The Alertmanager cluster deduplicates notifications only when every
replica receives every alert. The replica an alert left let it expire and
sent a resolved notification, and the replica it reached sent a new firing
one.

vmalert now lists each Alertmanager pod as its own notifier, using the
<name>-<i>.<service>.<namespace>.svc address the operator itself gives
StatefulSet pods. The replica count comes from one helper that also sets
spec.replicaCount of the VMAlertmanager, so the two cannot drift.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-12 19:45:24 +03:00
youdie006 b7708fb285 docs(monitoring): correct the release name the OIDC comment, guide and test fixture encode
The naming block in _helpers.tpl, the guide's opening paragraph and the
oidc test fixture all describe the per-tenant Grafana as the release
`monitoring`. Both instances are released as `monitoring-system` by the
monitoring application and differ only by namespace: the platform one in
`tenant-root`, a per-tenant one in each tenant namespace.

Assisted-by: LLM
Signed-off-by: youdie006 <youdie006@users.noreply.github.com>
2026-10-11 18:32:33 +03:00
Aleksei Sviridkin 520af57905 chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out
internal mirror block, an internal product name, and a reviewer's
name in a comment do not belong in a public repository; drop them
and replace what needs a stand-in with neutral placeholders.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 04:38:06 +03:00
Alexey Artamonov e63b9e439c fix(monitoring): count a central request as the receiver admitted it
The limiter refuses for good any request it counts above burst, and
burst was the receiver's 4 MiB cap. A request grows on its way to the
limiter: the sampler adds a tracestate to every span it keeps, and the
limiter counts a request as the collector re-encodes it, which writes
the empty value of every attribute the wire may omit, so a valid
request of spans with empty-valued attributes is counted 1.4x. A
request the receiver admitted near its cap was therefore refused on
every retry.

The limiter now runs right after memory_limiter, so it counts the
request before sampling. The receiver caps a request at 3 MiB and
burst stays at 5 MiB, which holds 1.4x the cap; a larger burst would
also be what each pod start lets through above the rate. The rate is
counted before sampling, so the store keeps only the sampled share of
it. The texts say that a request above the cap gets a 400, which
senders do not retry, and that each collector rollout a tenant causes
lets that burst through again on every pod.

The replica bound that keeps that allowance small now applies only to
a central tenant's collector, whose render fails above ten replicas: a
collector writing to the tenant's own store has no rate, and the
replica counts it shipped with stay valid.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 65f0caae2b fix(monitoring): keep a central tenant's write login out of its Grafana
A tenant's Grafana pod may reach the shared traces store's vmauth, and
the Secret a tenant names in oidc.customConfig.secretRef was mounted
into it whole. The apps API does not prune keys the Monitoring schema
leaves out, so a tenant could also set grafana.image: an image of its
choosing, with the collector's Secret mounted, writes past the rate
limit the collector enforces.

Grafana's image is the platform's, like the collector's, and for a
central tenant only the auth.ini key Grafana reads is mounted from the
named Secret. Other tenants have no collector login to keep out and
keep the whole Secret, which an auth.ini reading a sibling key with
$__file{} relies on.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov cc51cac63d fix(monitoring): give a central tenant's Grafana a login that cannot write
The collector holds a central tenant to its write rate, but the one
login it writes with was also the login of the tenant's Grafana
datasource, and it could both read and write. vmauth cleans the request
path before it picks a route, so any user the tenant admits to its
Grafana, a Viewer included, could POST through the datasource proxy to
../insert and write into the shared store past the limiter.

Each central tenant now has two VMUsers on its one account: the
collector's, which may only write, and Grafana's, which may only read,
each with its own chart-owned Secret and password. A namespace name
holds no dot, so the reader's username never matches another tenant's.
On upgrade the datasource moves to the new login in the same release
that creates it, so reads may fail until vmauth loads the new user.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 3497855e15 fix(platform): read the tracing rate only as plain decimal digits
Sprig's int64 reads a string with a leading zero as octal and turns a
parse error into 0, so "0100" would reach the collectors as 64, and
"08" or digits beyond int64 would leave the default in place without
a word. A float beyond int64 converts differently on each CPU
architecture, and the collector adds its replica count to the rate
before splitting it.

A string rate must now be plain decimal digits, and any rate above
1 TiB/s fails the render; the collector also falls back to the default
on digits it cannot hold rather than start with a rate of zero, which
the processor refuses.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov fc0db49187 fix(monitoring): bound what a central tenant writes above its rate
Each collector pod starts with a full token bucket, and burst was at
least 64 MiB, the largest request vtinsert accepts. A tenant controls
when its collector pods start, through any Monitoring change or by
deleting them, and how many there are, so it could write tens of
mebibytes above its rate per pod start and gigabytes per rollout of
many replicas, which the two-day bound does not survive.

The limiter only sees requests the receiver admits, so in central mode
the receiver now caps them at 4 MiB, decompressed for HTTP, and burst
drops to that floor; a pod start lets at most 4 MiB through above the
rate. The replica count is capped at 10. The texts now say what the
default rate guarantees: one tenant writes at most a default store's
disk cap over two days counted uncompressed, rather than that it
cannot fill it, and they name the per-pod-start allowance and that a
sender pinned to one replica gets that replica's share.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 177b207a27 feat(monitoring): hold each shared-central tenant to a write rate
The design has the per-tenant collector rate-limit its tenant, so that
a noisy or hostile tenant cannot exhaust a shared backend. The disk cap
alone does not do that: VictoriaTraces always keeps the last two days,
so a tenant writing faster than the cap allows over two days fills the
disk for every tenant on the store.

The platform sets the rate with
monitoring.tracingCentralTenantBytesPerSecond, and it reaches every
tenant in _cluster through the cozystack-values Secret. A change there is a change in the values each
monitoring HelmRelease reads, so Flux upgrades them all; a rate read
with lookup at render time would never reach a tenant whose release is
already in sync. The platform writes the key only when it is set, since
every release in every tenant reads that Secret. Without it the
collector holds the tenant to the rate of a default 10Gi store.

Each central tenant's collector splits the rate across its replicas
and refuses data above it before the batch processor, so the sender
sees the refusal. An OTLP/HTTP sender gets a 429 and retries; the
limiter answers gRPC with RESOURCE_EXHAUSTED and no RetryInfo, which
the OTLP specification makes final, so central tenants should export
over HTTP.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov d5fc723208 feat(monitoring): bound every traces store by disk usage
The distributed tracing design requires each traces backend to be
disk-bounded, not only age-bounded, so a full volume cannot block
ingest. retentionDiskUsageBytes was optional, so a store without it
filled its volume and stopped taking spans; with a shared-central store
that stops ingest for every tenant at once.

A store without a byte cap now drops its oldest day once its
filesystem is 80% full. VictoriaTraces refuses to start with both
limits set, so a byte cap, when given, replaces the percentage one.
It always keeps the last two days, which the values text now says.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-30 23:29:41 +03:00
Alexey Artamonov 27b7376bb9 feat(monitoring): add a shared-central tracing topology
A per-tenant traces backend costs every tenant its own VictoriaTraces
stack. A tenant can now send new spans to one shared store in
tenant-root instead, switched on by _namespace.tracingCentral, which
the tenant chart publishes and the apps API does not let an
application set. tenant-root decides whether it hosts that store, with
tracingCentralHost next to a cluster-mode store named generic.

VictoriaTraces takes the tenant account from request headers and does
no authorization of its own, so other tenants reach the store only
through a vmauth in tenant-root; tenant-root itself, with whoever it
admits to its Grafana, stays inside the trust boundary. Each central
tenant gets a VMUser that pins an account derived from its namespace
name and UID for both writes and reads and caps its requests in
flight, which the collector's two export senders stay under; its
collector exports and its Grafana reads with credentials the chart
generates and keeps, and vmauth overrides any account a client sends.
The chart owns those credentials because the operator publishes a
VMUser's own Secret only once a VMAuth selects it; the collector rolls
when they change, since it reads them at start. The vmauth internal
routes, whose /metrics names every tenant's user, are served on a port
tenants cannot reach, and the vmauth is sized by a VPA, spread across
nodes under a disruption budget, has a WorkloadMonitor like the other
monitoring workloads, and may reach the apiserver, where its
config-reloader reads its config Secret.

Renders refuse the states in which spans would be dropped with every
object Ready: a central tenant while tenant-root hosts nothing,
tenant-root dropping the hosting while central tenants remain, and
the host flag without a store to host. Each refusal reads the other
side at render time, so a concurrent opt-in and stop-hosting can pass
both; the values text names that race and its recovery. At run time
tenant-root's release is not Ready while the vmauth has failed.
Only the collector's export moves: local stores the tenant still lists
keep rendering with their datasources, because the switch belongs to
the parent tenant and the stores to this one, and the one-store limit
of the collector holds in both modes, since turning central off makes
it export locally again. tenant-root never counts as central.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-30 22:23:47 +03:00
Yan Bondarenko 17aaf3e038 feat(monitoring): add optional email alert delivery
Platform alerts need an email delivery path alongside Alerta without
exposing arbitrary namespace Secrets through tenant-controlled values.
SMTP credentials therefore use a fixed Secret name.

Alertmanager parses the addresses and connects to the SMTP host only
when it sends a notification, so a malformed value there drops alerts
instead of failing the release. The chart validates these fields when
it renders. Addresses must be literal dot-atom mailboxes, and a domain
literal must hold a dotted-quad IPv4 address, because IPv6 literal
syntax differs across the supported Go mail parsers. A bracketed
smarthost must be a valid IPv4 or IPv6 address, and its port must be
between 1 and 65535 without leading zeros. The repeat interval follows
the Alertmanager duration grammar and range, so a value that renders is
one Alertmanager loads.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
2026-09-30 13:55:12 +03:00
IvanHunters 2fa51bf587 test(charts): isolate guard coverage for every guarded template
Add per-template guard suites so each guarded hostname site is exercised
on its own: harbor harbor.yaml and httproute.yaml, bootbox machines.yaml,
seaweedfs cosi-deployment.yaml, bucket deployment.yaml and httproute.yaml,
monitoring alerta.yaml, grafana httproute.yaml and the grafanaHost helper
via oidc-keycloak.yaml. Each suite renders only its target template, so
removing that file's guard reddens it. Also add a postgres positive case
that renders the Certificate with an apex set (external=true).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 18:32:49 +03:00
IvanHunters 0f4d39f1ef test(charts): add empty-apex guard tests for the remaining guarded charts
Add host_guard_test.yaml for postgres, vpn, bootbox, seaweedfs,
monitoring and bucket. Each renders its guarded template with a
non-empty injected _namespace whose host is empty and no per-app host
override, and asserts the ns-host-required failure, plus a positive
render case with the apex set. Completes guard coverage alongside the
existing harbor, qdrant and kubernetes tests.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 16:01:31 +03:00
scooby87 cffe509dcf feat(monitoring): per-tenant OTLP collector (tracing phase 2) (#3763)
## What this PR does

Phase 2 of distributed tracing (design cozystack/community#38, epic
#3761), built on the phase-1 backend (#3762, merged): a per-tenant
**OpenTelemetry Collector** — the app-facing OTLP ingest gateway. It
lives in the tenant namespace (like a per-tenant vmagent), so
app→collector traffic is intra-namespace and never crosses the tenant
boundary. No OTel operator exists in the platform, so this is a plain
Deployment + ConfigMap + Service (the fluent-bit pattern).

## Changes

- **`templates/vtraces/collector.yaml`** — OTLP receivers (gRPC `:4317`,
HTTP `:4318`) → `memory_limiter` + head sampling
(`probabilistic_sampler`, default 10%) + optional `attributes/redact`
(PII/secret control point) + `batch` → `otlphttp` exporter to the
`tracingStorages` backend (`vtinsert:10481` cluster / `vtsingle:10428`
single, endpoints per upstream docs). PSS-restricted securityContext,
digest-pinned image via `cozy-lib.image`, config-checksum roll.
- **app-facing Service `otel-traces`** (4317/4318).
- **`tracingCollector` values** — `enabled` / `replicas` /
`samplingPercentage` (schema-bounded 0-100) / `redactAttributes` /
`resources`. A **fail-guard** rejects an enabled collector fronting more
than one `tracingStorages` backend (it exports to a single backend by
design).
- **`otel-traces` WorkloadMonitor** and dashboard resource-map entry,
rendered under the collector's own gate. The same resource-map Role also
gets the pre-existing `alermanager` typo corrected to `alertmanager`, so
the alertmanager WorkloadMonitor is readable from the dashboard.
- helm-unittest coverage for the collector and its WorkloadMonitor;
regenerated `values.schema.json` / `README.md` / `monitoring-rd`
ApplicationDefinition.

## Verification

- `helm unittest` — `system/monitoring` 110, `extra/monitoring` 15, all
pass.
- `helm template` renders cleanly; `make generate` idempotent.
- Live check on a test cluster (single-mode backend): the collector
starts with `redactAttributes` containing a numeric, a colon-bearing and
an empty key; an OTLP/HTTP span sent to `otel-traces:4318` is accepted
and is queryable from the backend through the Jaeger API, with the
redacted attributes removed and the others kept.
- Local `/branch-review` — **LGTM** (one blocker + four recommendations
fixed, each with a test: multi-backend guard, memory_limiter cgroup
limit, configurable replicas, sampling bounds, trust-boundary wording).

## Scope / deferred (epic #3761)

`AccountID`/`ProjectID` header injection for the shared-central topology
(phase 4) and engine integrations (phase 3) are follow-up PRs.

### Screenshots

N/A, no UI change.

### Downstream repositories

The diff touches only `packages/extra/monitoring`,
`packages/system/monitoring` and `packages/system/monitoring-rd`. The
website's monitoring reference page is regenerated from the package
README by the release bot, and the Terraform provider models only the
tenant's `monitoring` flag, not these values, so no follow-up is needed.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): deploy a per-tenant OpenTelemetry (OTLP) collector for traces once a VictoriaTraces backend is configured. Applications send OTLP to the in-namespace `otel-traces` Service (gRPC `:4317`, HTTP `:4318`); the collector applies head sampling (`samplingPercentage`, default 10%) and optional attribute redaction (`redactAttributes`) before exporting to the backend. Off by default (no backend configured).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added configurable trace storage in cluster or single-instance modes,
with retention settings and Grafana datasources.
* Added an enabled-by-default OpenTelemetry collector with configurable
replicas, sampling, attribute redaction, and resource limits.
* Added workload monitoring and automatic resource recommendations for
configured trace components.
  * Added readiness health checks for both tracing backend modes.

* **Bug Fixes**
* Corrected dashboard permissions for alert monitoring and trace
collector resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 16:27:45 +03:00
Aleksei Sviridkin 368d00b5d4 test(e2e): remove unused monitoring suites and audit BATS runners (#4019)
## What this PR does

I removed the two monitoring OIDC BATS suites that had no runner after
the Chainsaw migration, and brought the runner audit and wrapper OIDC
test from [#3778](https://github.com/cozystack/cozystack/pull/3778).

Also added checks for audience mapper and `OIDC_MODE=System`. The named
`KeycloakRealmGroup` test documents coverage the document-count and kind
assertions already provided.

Runner audit reports unused BATS suites, missing files in runner paths,
and unresolved Make variables (including `$(E2E_SUITE)`).

Live Monitoring API admission, Grafana operator reconciliation and mode
transitions are not tested here.

### Validation

- `make -j4 unit-tests SHELL=/usr/bin/dash` passed.
- Monitoring Helm tests: 53 system-chart tests and 10 wrapper-chart
tests passed.
- Runner audit: 12 cases passed under dash and Bats 1.14.0. The full
shell syntax and BATS sweep passed under dash too.
- Three chart mutations and seven runner mutations failed as expected.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
NONE
```
2026-09-25 16:55:05 +04:00
Alexey Artamonov 60b1b05434 feat(monitoring): add per-tenant OTLP collector for distributed tracing
Applications need one in-namespace OTLP endpoint that also enforces
sampling and redaction before spans reach the VictoriaTraces backend.
No OpenTelemetry operator ships with the platform, so the collector is
a plain Deployment, ConfigMap and Service ('otel-traces', gRPC 4317,
HTTP 4318) in the tenant namespace; app-to-collector traffic therefore
never leaves the tenant, and isolation comes from the existing tenant
network policies.

The collector renders only when it is enabled and a tracingStorages
backend is configured. Tracing is off by default, and an empty backend
list is a clean off-state rather than a render failure; an app exporter
that finds no Service simply retries. It writes to the first backend
only and rejects a list with several, instead of silently dropping
spans for the others.

redactAttributes is the PII and secret control point, so its keys are
rendered quoted and empty keys are dropped: an unquoted key with YAML
structural characters, or an empty one, makes the collector fail at
startup, and a key YAML re-types (numbers, booleans, null) silently
escapes redaction. replicas has a lower bound of 1 so a bad value fails
schema validation instead of the whole release at apply time; a zeroed
collector is expressed with enabled: false.

The collector gets a WorkloadMonitor and a dashboard-resources Role
entry under the same render gate, so it is part of tenant workload
accounting like every other workload in the chart.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-25 13:35:34 +03:00
Aleksei Sviridkin 2d2186d1f0 feat(monitoring): VictoriaTraces backend + Grafana datasource (tracing phase 1) (#3762)
## What this PR does

Phase 1 of distributed tracing (design: cozystack/community#38; epic:
#3761): a **VictoriaTraces backend and Grafana traces datasource** in
the monitoring stack, mirroring the logs (`VLCluster`) implementation.
No new operator: `VTCluster`/`VTSingle` already ship in
victoria-metrics-operator.

- **`tracingStorages`** values in `extra/monitoring` (annotated,
schema-generating) and `system/monitoring` (defaults), per-entry `mode:
cluster|single`. **Empty by default, so tracing is off out of the box**:
nothing produces spans until the collector ships, so no backend is
provisioned until an operator adds an entry.
- **`templates/vtraces/vtraces.yaml`**: `VTCluster` (default) or
`VTSingle` (edge/dev/small); `retentionPeriod` plus optional
`retentionMaxDiskSpaceUsageBytes`; a release-scoped **PVC cleanup
label**, reused by the existing post-delete hook, so no hook change is
needed.
- **Render-time validation** (`_helpers.tpl`): the installed
VictoriaTraces CRDs are the trimmed ones, so a bad value would otherwise
reach the operator and fail after admission. Each `tracingStorages`
entry is checked before anything renders: `name` must be an RFC 1123
label of at most 42 characters (the `vtstorage-<name>-<hash>` pod
revision label is capped at 63) and unique across the list, `mode` must
be `cluster` or `single`, `storage` must be a positive quantity (a zero
size makes the operator mount an EmptyDir in single mode, and a negative
one fails when the PVC is created), and `retentionDiskUsageBytes` must
be a VictoriaMetrics `BytesString` (the operator rejects the
Kubernetes-quantity `8Gi` grammar that `storage` uses). The `storage`
field is also typed as a quantity in the schema, and the rendered name
is quoted so a numeric name stays a string.
- **`templates/vtraces/grafana-datasource.yaml`**: a Jaeger-typed traces
datasource per entry (built-in Grafana core datasource, no plugin).
- **VPA**: a `VerticalPodAutoscaler` for each trace component
(`vtinsert`/`vtselect` Deployments, `vtstorage` StatefulSet, `vtsingle`
Deployment), matching the metrics/logs sizing, with `updateMode:
Initial`.
- **Readiness gate**: `VTCluster`/`VTSingle` `healthCheckExprs` on
`status.updateStatus`, alongside the existing `VLCluster` gate.
- **VictoriaLogs datasource fix**
(`templates/vlogs/grafana-datasource.yaml`): the range over
`logsStorages` emitted no `---` between documents, so two or more
entries collapsed into one document and only the last datasource
survived. Each entry is now its own document.
- **Tests and schema**: helm-unittest suites for off-by-default, cluster
vs single, datasource type and both query URLs, retention, PVC label,
VPA targets, every validation guard with its boundaries, and per-entry
vlogs datasource documents, plus regenerated `values.schema.json`,
`README.md` and the `monitoring-rd` ApplicationDefinition.

#### Upgrade impact

With default values the only rendered object that changes is the
`monitoring-system` HelmRelease: it gains the `VTCluster`/`VTSingle`
`healthCheckExprs` and `tracingStorages: []`. Flux evaluates those
expressions only against objects in the release inventory, so they are
inert while tracing is off. No VictoriaTraces object is created on
upgrade. Installs with two or more `logsStorages` entries additionally
gain the `GrafanaDatasource` objects that were previously dropped (see
the VictoriaLogs fix above); nothing else under VictoriaLogs or
VictoriaMetrics changes.

#### Verification

- `helm unittest`: `system/monitoring` (89 tests) and `extra/monitoring`
(11 tests) pass. Each validation guard and each branch of the
zero-storage pattern was mutation-checked: reverting it turns its own
test red.
- `helm template` renders `VTCluster`/`VTSingle`/datasource/VPA cleanly.
`tracingStorages: []` (the default) renders nothing. An invalid,
duplicate or over-long `name`, a zero or negative `storage`, or a bad
`retentionDiskUsageBytes` (`8Gi`) is refused at render with an
actionable message; a numeric `name` renders as a string. `make
generate` is idempotent.
- **Datasource URLs verified against upstream docs** ([VictoriaTraces
and
Grafana](https://docs.victoriametrics.com/victoriatraces/querying/grafana/)):
cluster `vtselect:10471/select/jaeger`, single
`vtsingle:10428/select/jaeger`. `jaeger` is the stable, built-in Grafana
datasource; the Tempo Query API (`/select/tempo`, `type: tempo`,
VictoriaTraces v0.9.4+) has a richer TraceQL UX but is
upstream-experimental, so Jaeger is the default. Both URLs are pinned by
unit tests.
- The `retentionDiskUsageBytes` pattern, the zero-storage check and the
name limit were checked against victoria-metrics-operator v0.68.4
(`BytesString.UnmarshalJSON`, `isStorageEmpty`, the `vtstorage-<name>`
StatefulSet name) and the Kubernetes `ControllerRevisionName` naming.

#### Scope / deferred (tracked in #3761)

Per-tenant OTLP Collector, engine integrations, shared-central topology,
and docs are follow-up PRs. This PR intentionally does **not** enable
the OTLP/gRPC listener: apps can use `vtinsert`'s native OTLP/HTTP, and
the gRPC listener lands with the collector.

Builds on the now-merged design proposal cozystack/community#38.

### Screenshots

Not a UI change.

### Downstream repositories

Walked against the diff, no box ticked yet:

- cozystack/website: `monitoring` is in the website `Makefile`
`SERVICES` list, so its reference page is regenerated from
`packages/extra/monitoring/README.md` automatically on the next stable
tag. A user guide for tracing belongs to the docs follow-up in #3761; no
website issue or PR exists for it yet.
- cozystack/terraform-provider-cozystack: the provider has no
`Monitoring` resource (only the tenant's `monitoring` flag), so the new
schema field does not reach it.
- The remaining repositories in the trigger map are not reached by this
diff: no platform values, variants, installer values, `hack/` tooling,
node prerequisites, ApplicationDefinition semantics or telemetry names
change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): add a VictoriaTraces backend and a Grafana traces datasource to the monitoring stack (distributed tracing, phase 1). Tracing is off by default; enable it per tenant by adding a `tracingStorages` entry (`cluster` or `single` mode). Also fix the VictoriaLogs Grafana datasources: with two or more `logsStorages` entries, every entry except the last was silently dropped, so existing installs with several log stores gain the missing datasources on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added configurable trace storage with cluster or single-node
deployment modes.
* Supports retention periods, persistent storage sizing, storage
classes, and optional disk-usage limits.
* Automatically configures Grafana tracing data sources for each enabled
trace backend.
* Monitoring readiness reflects the health of configured trace storage
services.
  * Invalid trace storage settings are rejected during configuration.
* **Bug Fixes**
* Every VictoriaLogs storage now gets its own Grafana datasource; with
several log storages, all but the last were previously dropped.
* **Tests**
* Added coverage for storage rendering, retention settings, validation
guards, Grafana endpoints, health checks, and cleanup behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 14:21:26 +04:00
Myasnikov Daniil 16d23c8a53 test(monitoring): preserve OIDC coverage without live BATS suites
Rendering the inner chart alone cannot catch a wrapper dropping tenant
OIDC settings. The audience mapper and System-mode reconcile flag also
need assertions before the unused live suites are removed.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-24 17:08:47 +05:00
Alexey Artamonov 76445ba9b6 fix(monitoring): reject a negative tracingStorages storage
The storage guard only matched zero-shaped values, so -1Gi passed it and
the schema's quantity pattern alike, rendered into the VTCluster/VTSingle
spec, and left the release failing under retries: -1 once the API server
refused the PVC. Refuse a leading minus in the same guard, so the error
names the field before anything is applied and matches what the message
promises.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 14:39:39 +03:00
Myasnikov Daniil c55ab268e2 test(monitoring): remove orphaned OIDC BATS suites
The two pre-Chainsaw monitoring OIDC suites have had no invocation site since the app suite migrated to Chainsaw. Remove them instead of carrying dead tests that force the selector to run every live suite.

Keep the existing selector and capture guards for the historical path shape, and update their comments now that the directory is empty. Retain a named chart assertion for the no-KeycloakRealmGroup ownership invariant; the existing count and kind assertions already enforce the same behavior.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-24 16:28:00 +05:00
Alexey Artamonov cf7a5684d0 test(monitoring): make the single-mode VPA absence assertions able to fail
containsDocument with not: true but no any: true is scoped to a single
document, so the "no cluster VPA in single mode" and "no VPA when empty"
assertions passed regardless of what the render actually held. Add any: true
inside the matcher so they scan every document: a mutation that leaks the
cluster VPAs into single mode now turns the suite red, and it is green on HEAD.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 37789e1768 fix(monitoring): reject zero storage and over-long names in tracingStorages
Two tracingStorages values render cleanly and only fail once the operator
acts on them.

- A zero storage is silently destructive in single mode: the operator's
  isStorageEmpty gate checks IsZero() and mounts an EmptyDir instead of a
  PVC, the VTSingle reports operational and the release goes Ready, and every
  stored span is lost on the next reschedule. Type the storage field as a
  quantity so the schema refuses non-quantities at admission, and refuse
  every zero spelling ("", 0, 0Gi, 0.0, .0, 0e3, 0m, +0, -0Mi) in the render
  guard, since the quantity pattern accepts those.
- A name over 42 characters wedges cluster mode: the operator names the
  StatefulSet vtstorage-<name>, and each pod gets the label
  controller-revision-hash: vtstorage-<name>-<hash>, whose value is capped at
  63 with up to 10 characters of hash. Past that no pod is created, the
  VTCluster goes failed and the release stays not-Ready under retries: -1.
  Cap the length in the same guard.
- Quote the rendered name, so a numeric name such as 2024 stays a string
  instead of failing the apply on metadata.name.
- Rewrite the helper's rationale so it matches the trimmed CRDs and the
  pinned operator, and state in the mode field description that the API
  cannot enforce immutability on an array-item field, so the note is the only
  guard.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 30fcfe6877 fix(monitoring): emit each vlogs GrafanaDatasource as its own document
The range over logsStorages emitted its documents with no `---` separator, so
two or more entries concatenated into a single document with duplicate keys,
each later entry overwrote the earlier one, and every log store but the last
silently got no datasource. Add the separator inside the range, matching the
traces datasource template.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov f20bd7b334 fix(monitoring): validate tracingStorages entries and flag mode/retention hazards
The installed VictoriaTraces CRDs are the trimmed ones (crds.plain: false),
so their spec is x-kubernetes-preserve-unknown-fields and a bad value reaches
the operator instead of being refused by the API server. Catch the ones a
render guard can, and document the one it cannot.

- name is used verbatim as the VTCluster/VTSingle and GrafanaDatasource
  metadata.name, so an invalid name is a late admission error and a duplicate
  silently overwrites the first entry (last write wins in the range). Guard it
  against an RFC 1123 label and a uniqueness set.
- retentionDiskUsageBytes maps to the operator's BytesString field, whose
  UnmarshalJSON rejects anything outside
  ^[0-9]+(kb|mb|gb|tb|KB|MB|GB|TB|KiB|MiB|GiB|TiB)?$, notably not the Gi
  grammar of the sibling storage field. The operator then marks the CR failed
  and the release fails well after the value was written. Refuse it at render
  with the upstream pattern instead.
- changing mode re-identifies the object instead of reconfiguring it, so
  switching single to cluster prunes the VTSingle and the operator
  garbage-collects its PVC, and cluster to single (or dropping the entry)
  leaks the vtstorage PVCs
  until the release is deleted. A stateless template cannot see the prior
  mode, so record the loss and the one-way migration in the field description.

The name, uniqueness, mode and retention checks share one pass in a helper;
the inline mode guard in vtraces.yaml moves there. The retention description
also notes that the cap is per vtstorage replica in cluster mode.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 41aa2aad1d feat(monitoring): add VPA for VictoriaTraces backend components
vpa.yaml sized only metrics and logs components; the trace components got no
VerticalPodAutoscaler, so they sat at the operator's static defaults with no way
to size them (tracingStorages has no resource knobs and the template hardcodes
resources/replicaCount).

Add VPAs per tracingStorages entry, branching on mode: cluster mode gets vtinsert
and vtselect (Deployments) + vtstorage (StatefulSet), single mode gets
vtsingle (Deployment). Kinds verified against victoria-metrics-operator v0.68.4.
updateMode: Initial like the rest of the file (no eviction, because of the
Cilium IP-reuse race), which is also required for VTSingle's plain RWO PVC (an
evicting VPA would deadlock on multi-attach). Bounds are hardcoded, without the
dead min/maxAllowed conditionals the logs entries carry (tracingStorages
declares no such properties). Empty tracingStorages renders no trace VPA. Tests
cover the cluster three-VPA split, the single-mode single VPA with its
updateMode, and off-by-default.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 00ff19ec5c fix(monitoring): default tracing off and defer fail-on-empty guard
tracingStorages had two defaults (extra + system), so 'tracingStorages: null'
silently backfilled the inner default and a VTCluster rendered anyway, while
'tracingStorages: []' aborted the whole monitoring release. There was no clean
'off', and the default shipped a 6-pod VTCluster per tenant for a pipeline
that nothing writes spans into yet.

- Default tracingStorages to [] in both packages/system/monitoring/values.yaml
  and packages/extra/monitoring/values.yaml, so absence has nothing to fall
  through to and the generated schema advertises the shipped (off) state.
- Drop the fail-on-empty guard in templates/vtraces/vtraces.yaml. Nothing
  produces spans without a collector, so an absent backend drops no data. The
  guard belongs with the collector, where it becomes correct: an enabled
  producer with no backend really would drop spans, the logsStorages #3181
  rationale.
- healthCheckExprs stay unconditional (Flux evaluates them only against CRs
  present in the inventory, so they are inert when nothing renders).
- Tests: replace the empty-fails case with off-by-default (renders nothing);
  set tracingStorages explicitly where cases relied on the old default.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov dbe7a732ad test(monitoring): cover VTSingle PVC cleanup label + single-mode disk cap
Single mode stamps the release-scoping cleanup label via
spec.storageMetadata, a different field than the cluster path
(spec.storage.storage.volumeClaimTemplate), and it was untested, so a dropped
storageMetadata block would orphan the VTSingle PVC on tenant deletion with no
test to catch it. Add the VTSingle cleanup-label assertion and a single-mode
top-level retentionMaxDiskSpaceUsageBytes assertion.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 3774e09159 fix(monitoring): reject unsupported tracingStorages mode
tracingStorages[].mode accepted any string, so a typo (e.g. 'singel')
silently rendered a VTCluster instead of failing. Add a render guard that
fails on anything but cluster|single, restrict the schema property with a
TracingMode enum, and add a helm-unittest case asserting the rejected mode.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 32f170949e test(monitoring): pin verified VictoriaTraces query URLs; jaeger stable default
Datasource service/port/path verified against upstream docs
(https://docs.victoriametrics.com/victoriatraces/querying/grafana/):
cluster vtselect:10471/select/jaeger, single vtsingle:10428/select/jaeger.
Jaeger is the stable, built-in Grafana core datasource; the Tempo API
(/select/tempo, type: tempo, VictoriaTraces v0.9.4+) is upstream-experimental,
kept as a documented alternative. Add helm-unittest assertions pinning both
the cluster and single-mode query URLs.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov c2fce3d41d feat(monitoring): add VictoriaTraces backend + Grafana datasource for tracing
Phase 1 of the distributed-tracing design proposal (cozystack/community#38):
a per-tenant traces backend in the monitoring stack, mirroring the logs
(VLCluster) implementation. No new operator: VTCluster/VTSingle already ship
in victoria-metrics-operator.

- tracingStorages values (extra + system), per-entry mode cluster|single
- templates/vtraces/vtraces.yaml: VTCluster (default) / VTSingle (edge/dev),
  #3181-style empty-list render guard, retentionPeriod + optional disk cap
  (retentionMaxDiskSpaceUsageBytes), release-scoped PVC cleanup label
- templates/vtraces/grafana-datasource.yaml: Jaeger-typed traces datasource
- extra/monitoring readiness gate: VTCluster/VTSingle healthCheckExprs on
  status.updateStatus, alongside VLCluster
- helm-unittest suites (required/empty-guard, cluster-vs-single, datasource,
  PVC label) + regenerated values.schema.json / README / ApplicationDefinition

The per-tenant OTLP Collector, engine integrations, shared-central topology
and docs are out of scope and tracked in the tracing epic.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 44a9a8e735 feat(platform): make the Keycloak realm name configurable
The realm name "cozy" was hardcoded in keycloak-configure (which owns
the realm, its clients and the cozystack-cluster-admin roles), in the
dashboard and linstor-gui gatekeepers, in the info kubeconfig, and in the
Grafana and managed-Kubernetes apiserver OIDC issuers. Operators could
not rename the realm without patching vendored templates, which
`make update` reverts. The apiserver OIDC issuer is already configurable
via talm oidcIssuerUrl, so pointing it at a custom realm left every SSO
component on "cozy" and broke login with an issuer mismatch.

authentication.oidc.realmName (default "cozy") now rides the _cluster
channel so every consumer resolves the same realm, and the default
keycloakInternalUrl is derived from it so the realm is named in one place.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-21 12:15:10 +03:00
Aleksei Sviridkin eef5fe6908 fix(platform): declare the injected _cluster values in system charts
Ten system charts cannot be rendered on their own. They read the
platform-injected _cluster channel, and two of them (bucket,
monitoring) also read _namespace, without declaring either key in
values.yaml. On a cluster the platform supplies both through the
cozystack-values Secret, so installs are unaffected, but rendering a
chart by itself dies before a single template is emitted: on "index of
untyped nil" for a missing _cluster, and on "nil pointer evaluating
interface {}.host" for a missing _namespace.

Declare the keys with empty defaults. Helm merges chart defaults under
the supplied values, so an injected _cluster still wins and rendered
output does not change. Each of the ten also gets a suite that renders
with nothing injected; drop a declaration again and that chart's suite
goes red.

This makes `helm template` succeed for all ten. `helm lint` also
reports a missing cozy-lib dependency on keycloak and monitoring,
which is a separate pre-existing defect and survives this change.

etcd-operator reads _cluster too and stays undeclared here: its single
call site already guards with `| default dict`, so the chart renders
standalone as it is.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-07 02:23:47 +03:00
Aleksei Sviridkin d73c102945 test(system): anchor index-addressed negative asserts to what they name
notContains passes whenever its path does not resolve, and also
whenever it resolves to something that simply lacks the forbidden
content. A different container at the same index qualifies, so these
guards go green the moment a sidecar is placed ahead of the workload
they were written for, while the flag or env var they forbid comes
back on the real container.

Assert the identity at that index first, in each affected case: the
container name for the Deployment-shaped ones, and for the Cilium
policy its selector, since a deny rule has no name. The selector
identifies that rule only while there is one rule to identify, so it
is paired with a length assertion, and with a second one on the
rule's toPorts, because that path carries two indexes and anchoring
the outer one alone leaves the inner free.

The anchor goes only where nothing else holds the index. A positive
contains on the same path fails on its own once a sidecar arrives
ahead of the workload, and an equal on the whole argument list fails
the same way, so neither carries a separate anchor. A negative does,
including the per-element form below: a wildcard resolving to the
sidecar's own arguments matches nothing and passes.

Several of the guards could not fail for the reason they name, and
the anchor alone would have left that hidden behind a comment saying
they were covered. contains and notContains compare elements for
equality, so a mapping is compared whole: `{name: GF_...}` never
matched an entry carrying a value beside the name, and `{port: 7473}`
would not match that port re-added with a protocol beside it, which
is how the neighbouring entry in the same list is written. Those get
`any: true`.

Equality against a rendered argument is the same failure worn
differently, and it arrives in three disguises. A prefix that never
equals an element: `--pinned-tenants=` never equalled
`--pinned-tenants=tenant-a=shard0`, and `--target-host=` never
equalled a rendered `--target-host=<value>`. One value out of many
under a name that claims the flag: `--compatibility=cilium` left the
kilo default render free to carry `--compatibility=flannel` off the
same values key, and the two `--proxy-fqdn` negatives forbade only
the FQDN the default cluster-domain produces. And a form that holds
by luck: `--gateway-api` still fails today only because that template
renders it bare, so an upstream switch to `--gateway-api=true` would
retire the guard in silence.

All of those become notMatchRegex over args[*], which matches the
flag name per element and fails on any value and on the bare form
alike. A length assertion would not have covered the prefix cases,
since an upstream bump that adds the flag while consolidating another
argument keeps the count. A wildcard that resolves to nothing still
passes, so the index anchors stay.

The whole argument list is a different instrument, answering what the
default invocation is rather than whether one flag is absent. It
fails on an argument added, removed or swapped instead of only on the
ones someone thought to enumerate, which is worth having where the
template is first-party and a failure is a deliberate change asking
to be read. flux-shard-operator gets it in `renders the default
Deployment shape`, replacing eleven individual contains that listed
exactly the same eleven elements; `--shard-cpu-limit=` was dead for
the prefix reason and is dropped rather than repaired, because that
list now covers it. The vendored ouroboros templates get no such pin:
upstream churn arrives there on its own schedule, and a whole-list
assertion would fail a test named for one flag for a reason that has
nothing to do with that flag.

Anchor every index on a path, not the outermost one. The Cilium
assertion reads spec.ingressDeny[0].toPorts[0].ports, and pinning the
rule list alone would still let the port return on a second toPorts
entry of the same rule.

Each guard was checked by rendering what it forbids onto the element
it names, and each anchor by rendering the shift it exists to catch.

The form is already used in this tree, in opensearch-operator and
dashboard. Assertions of the same shape remain in files this commit
does not touch, including one in a package it does, and stay out of
scope here.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-12 15:28:37 +03:00
Andrei Kvapil 705e39d3d7 feat(monitoring): install Grafana plugins from the in-cluster mirror
Point GF_INSTALL_PLUGINS at the grafana-dashboards static file server
instead of the public Grafana catalog, so plugin installation needs no
internet egress at pod startup and keeps working on air-gapped installs.
Route the upstream Grafana image reference through cozy-lib.image so
mirrored-registry installs keep working, and add regression tests for
the image reference and the plugin install path.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-24 16:08:51 +02:00
Ivan Okhotnikov 0164a6afa4 fix(monitoring): set groups_attribute_path=groups so allowed_groups actually works on Grafana v11.x
Verified end-to-end on dev3 with Grafana v11.6.15: OAuth flow reaches
Grafana, id_token and userinfo response both carry the correct
`groups` claim (`[cozystack-cluster-admin, tenant-root-super-admin]`
for kvaps), `allowed_groups` in grafana.ini lists
`tenant-root-super-admin`, and login still fails with
`user not a member of one of the required groups`.

Grafana debug log shows `Attributes: map[]` — the extracted user's
Groups slice is empty. On v11.x+ Grafana does not auto-map the
`groups` claim onto user.Groups unless `groups_attribute_path` is
set explicitly (the field is documented as optional but the gate is
a silent no-op without it — a claim-name-string is not enough).

Fix: hard-wire `groups_attribute_path=groups` in the chart-forced
System-mode block alongside `allowed_groups`. "groups" points at
the top-level `groups` array from userinfo, matching the shape
Keycloak's oidc-group-membership-mapper emits (verified on the same
dev3 run).

Two helm-unittest cases updated to assert `groups_attribute_path`
lives next to `allowed_groups` in both the users-map-active and
empty-users branches (rejecting the silent no-op regression). Docs
also updated to name the requirement.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
2026-07-15 15:31:45 +03:00
IvanHunters b002c967ac fix(monitoring): close cross-tenant Grafana OIDC hole via tenant-group gate
Round-3 review from lllamnyp: any user in the shared `cozy` Keycloak
realm could sign into any tenant's Grafana. The audience mapper stops
replay of a minted token across releases, but does not gate who may
mint one — tenant-alice could complete the OAuth flow against
tenant-bob's Grafana. With `spec.oidc.users` non-empty the earlier
`allow_sign_up=false` fix mostly closed the door (though tenant-alice
still hit `error 401` after landing in Grafana, revealing the release
exists); with empty users under mode: System — a config the schema
permits and the docs describe as hands-off — the login succeeded and
minted a Viewer account for every cozy-realm identity.

The fix uses the KeycloakRealmGroups the tenant chart already
provisions (`<namespace>-{view,use,admin,super-admin}` in
packages/apps/tenant/templates/keycloakgroups.yaml) — no new
directory objects owned by the Monitoring release. Two chart edits:

1. oidc-keycloak.yaml: add the platform-managed realm-default `groups`
   scope to `defaultClientScopes` explicitly. EDP's KeycloakClient
   does NOT merge realm-level defaults into `defaultClientScopes`, so
   the scope must be listed here or the `groups` claim would be
   missing from the token.

2. grafana.yaml: in System mode, unconditionally emit
   `allowed_groups: <ns>-view <ns>-use <ns>-admin <ns>-super-admin`
   in `auth.generic_oauth` and add `groups` to `scopes`. Grafana
   rejects any login whose token does not carry at least one of the
   groups.

The gate is chart-owned and does not depend on the users-map — with
`users: []` tenant members still get a self-service Viewer through
`auto_assign_org_role`, cross-tenant users get 401 in Keycloak's OAuth
callback. Docs updated to reflect the two-layered contract
(tenant-membership gate + optional users-map).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-15 15:31:45 +03:00
IvanHunters ff4ef2c189 fix(monitoring): force allow_sign_up=false and gate isolation dict on users-map
Address PR#3176 review round-2 gap-1: the docs promise "users not in
spec.oidc.users get nothing", but Grafana's default allow_sign_up=true
combined with the chart-forced skip_org_role_sync=true was minting a
Viewer account for every cozy-realm identity that completed the OAuth
flow, then reaping it on the next helm-upgrade prune pass.

The chart-forced isolation dict (now three keys: skip_org_role_sync,
oauth_allow_insecure_email_lookup, allow_sign_up=false) is applied only
when spec.oidc.users is non-empty. A BYO-IdP operator who does not use
the users-map gets none of the three forced on top of their inline
config, matching the review's minor: forcing insecure_email_lookup
against an issuer that may not verify emails is a needless account-
linking surface, and allow_sign_up=false with no pre-provisioned
accounts would lock everyone out.

Tests split the two branches: users-active asserts all three keys are
present and the operator cannot override them via CustomConfig-inline;
users-empty asserts the auth.generic_oauth block is emitted without
any of the isolation keys.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-15 15:31:45 +03:00
IvanHunters 1422495b88 fix(monitoring): do not render users-Job when mode=None (upgrade safety)
Caught by dev3 end-to-end test on the PR working tree
(cozystack-pr-test skill): the users-Job was rendered in every mode
including `None` on the rationale that a `mode: System → None`
switch should garbage-collect OIDC-provisioned local accounts. That
rationale ignored the upgrade path from the pre-PR chart: an
existing tenant that never had `spec.oidc` on their Monitoring CR
would receive `mode: None` on first render under the PR, the Job
would fire on every Flux reconcile of the outer HR, and its prune
pass would DELETE every non-admin Main-Org member — including users
the tenant added by hand through the Grafana UI on the pre-PR chart.
Silent breaking change with data loss.

Wrap the whole Job resource in `{{- if ne $mode "None" }} … {{- end }}`
so `mode: None` = "chart owns nothing in Grafana orgs, we don't
touch what's there". Operators who need to prune OIDC-provisioned
accounts after switching System | CustomConfig → None do it
themselves through the Grafana UI or admin API — same posture as
"we don't own the Keycloak directory objects" from the earlier
architectural change.

The `assertUsersPrologue` include stays outside the `if` so an
operator listing users in `mode: None` still gets the schema and
email-shape validation at render time — the Job body just doesn't
emit.

Test coverage:
- Flip the previous "mode=None still renders the users-Job so a
  mode switch can prune stale members" case to
  `mode=None does NOT render the users-Job (upgrade safety guard)`
  as a regression lock.
- Add `mode=None with non-empty users still fails render` covering
  the prologue-only-fires-on-schema-issue path.

Docs: rewrite the "Users and RBAC" section step 3 to name the
upgrade-safety motivation and the manual-cleanup handoff.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-15 15:31:45 +03:00
IvanHunters c31a9f8ec3 fix(monitoring): promote oidc.users[].role to required + regenerate schema
The `OIDCUser` typedef in `packages/extra/monitoring/values.yaml`
declared only `@field {string} email`; `cozyvalues-gen` regenerated
`values.schema.json`, `README.md`, and the mirrored `openAPISchema`
on the `Monitoring` ApplicationDefinition WITHOUT a `role` field.
Consequences:

- The `@enum OIDCRole` declaration was orphan — referenced from
  nowhere, so the schema's items object had `required: ["email"]`
  and one property.
- A dashboard-driven CR would offer a form without any role picker;
  a schema-conformant `{email}` submission would render into the
  users-Job with `role: null`, `jq --raw-output` yields the string
  `"null"`, the `POST /api/orgs/1/users` body carries
  `"role":"null"`, Grafana returns HTTP 400, and the post-install
  hook exhausts `backoffLimit: 6` — HelmRelease never becomes Ready.
- Unit + e2e tests all hardcoded `role: Admin` / `role: Viewer`, so
  the contract mismatch never surfaced in CI.

Fix:

- Add `@field {OIDCRole} role - ...` to the `OIDCUser` typedef in
  `packages/extra/monitoring/values.yaml` so cozyvalues-gen materialises
  the field with enum + required.
- Regenerate `values.schema.json`, `README.md`, and mirrored
  `openAPISchema` via `make generate`.
- Add `monitoring.oidc.assertUsersRoleShape` template-level guard
  (same shape as `assertUsersEmailShape`) so a stale-schema CR or a
  direct-helm invocation fails at render time with a legible error
  instead of at hook execution with a Grafana 400.
- Consolidate all `spec.oidc.users` guards into a single
  `monitoring.oidc.assertUsersPrologue` include and call it once from
  `oidc-users-job.yaml`, ordering the incompat check
  (`assertCustomSecretRefUsersEmpty`) before the per-entry role /
  email regexes so operators see the more actionable message first.
  Drop the duplicate include from `grafana.yaml` (Job template renders
  in every mode anyway).
- Two new helm-unittest cases lock the regression:
  `oidc.users entry missing role fails the render` and
  `oidc.users entry with unknown role fails the render`.
- Point the previous `secretRef + users` incompat unit test at the
  users-Job template since that's where the prologue now fires.
- `docs/oidc-grafana.md`: correct the failed-hook post-mortem window
  claim — with `ttlSecondsAfterFinished: 3600` the Kubernetes TTL
  controller reaps the Job + Pod 1h after the terminal condition,
  not "until the next helm-upgrade".

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-15 15:31:45 +03:00
IvanHunters ad67dee95b test(monitoring): lock the CustomConfig chart-force contract in helm-unittest
Two blockers on the previous review cycle were "code says X but no
test enforces X" — so future edits to `grafana.yaml` could silently
break the contract again. Add three assertions that fail loudly if
the merge behavior regresses:

- `mode=CustomConfig inline preserves the operator's map + chart-forced
  settings` — asserts `skip_org_role_sync=true` and
  `oauth_allow_insecure_email_lookup=true` are both present alongside
  the operator's `client_id` / `client_secret` / `auth_url`.
- `mode=CustomConfig inline — operator cannot override chart-forced
  OIDC settings` — sets both settings to "false" in the operator's
  map; asserts they render as "true" (proves `merge` argument order
  is correct).
- `mode=CustomConfig secretRef + non-empty users fails the render
  (contract incompat)` — asserts the new
  `monitoring.oidc.assertCustomSecretRefUsersEmpty` helper's error
  message fires so operators cannot silently strand their users-Job
  assignments by combining incompatible fields.
- `mode=CustomConfig secretRef + empty users renders successfully`
  — the compatible path.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-15 15:31:45 +03:00
IvanHunters c493f2cb96 test(monitoring): rewrite OIDC unit tests around the users-map contract
Rewrites the helm-unittest coverage for the Phase-1 OIDC selector
against the new chart contract:

- Drop assertions for the three KeycloakRealmGroups and for
  `role_attribute_path` / `allow_assign_grafana_admin` / the
  `oidc.grafanaAdmin` values lever — none of those render anymore.
- Assert the System-mode Grafana config carries
  `skip_org_role_sync = "true"` and
  `oauth_allow_insecure_email_lookup = "true"` and NO
  `role_attribute_path`.
- Cover the new users-Job template: helm-hook annotations, absence
  of a ServiceAccount, `activeDeadlineSeconds`, admin-credentials
  wiring from the chart-managed `grafana-admin-password` Secret,
  and correct serialisation of `spec.oidc.users` into the
  `DESIRED_USERS_JSON` env.
- Assert the users-Job also renders under `mode: None` so a
  System -> None flip still gets a prune pass.
- Cover the defensive `.Values.oidc | default dict` paths — omitted
  block and explicit `oidc: ~` — across the three OIDC templates.
- Tighten the client-secret regex to the actual base64-encoded
  length (`^[A-Za-z0-9+/]{43}=$`) so shortened random secrets are
  caught.

32 tests, all green under helm-unittest v1.0.3.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-15 15:31:45 +03:00