Files
scooby87 cffe509dcf feat(monitoring): per-tenant OTLP collector (tracing phase 2) (#3763)
## What this PR does

Phase 2 of distributed tracing (design cozystack/community#38, epic
#3761), built on the phase-1 backend (#3762, merged): a per-tenant
**OpenTelemetry Collector** — the app-facing OTLP ingest gateway. It
lives in the tenant namespace (like a per-tenant vmagent), so
app→collector traffic is intra-namespace and never crosses the tenant
boundary. No OTel operator exists in the platform, so this is a plain
Deployment + ConfigMap + Service (the fluent-bit pattern).

## Changes

- **`templates/vtraces/collector.yaml`** — OTLP receivers (gRPC `:4317`,
HTTP `:4318`) → `memory_limiter` + head sampling
(`probabilistic_sampler`, default 10%) + optional `attributes/redact`
(PII/secret control point) + `batch` → `otlphttp` exporter to the
`tracingStorages` backend (`vtinsert:10481` cluster / `vtsingle:10428`
single, endpoints per upstream docs). PSS-restricted securityContext,
digest-pinned image via `cozy-lib.image`, config-checksum roll.
- **app-facing Service `otel-traces`** (4317/4318).
- **`tracingCollector` values** — `enabled` / `replicas` /
`samplingPercentage` (schema-bounded 0-100) / `redactAttributes` /
`resources`. A **fail-guard** rejects an enabled collector fronting more
than one `tracingStorages` backend (it exports to a single backend by
design).
- **`otel-traces` WorkloadMonitor** and dashboard resource-map entry,
rendered under the collector's own gate. The same resource-map Role also
gets the pre-existing `alermanager` typo corrected to `alertmanager`, so
the alertmanager WorkloadMonitor is readable from the dashboard.
- helm-unittest coverage for the collector and its WorkloadMonitor;
regenerated `values.schema.json` / `README.md` / `monitoring-rd`
ApplicationDefinition.

## Verification

- `helm unittest` — `system/monitoring` 110, `extra/monitoring` 15, all
pass.
- `helm template` renders cleanly; `make generate` idempotent.
- Live check on a test cluster (single-mode backend): the collector
starts with `redactAttributes` containing a numeric, a colon-bearing and
an empty key; an OTLP/HTTP span sent to `otel-traces:4318` is accepted
and is queryable from the backend through the Jaeger API, with the
redacted attributes removed and the others kept.
- Local `/branch-review` — **LGTM** (one blocker + four recommendations
fixed, each with a test: multi-backend guard, memory_limiter cgroup
limit, configurable replicas, sampling bounds, trust-boundary wording).

## Scope / deferred (epic #3761)

`AccountID`/`ProjectID` header injection for the shared-central topology
(phase 4) and engine integrations (phase 3) are follow-up PRs.

### Screenshots

N/A, no UI change.

### Downstream repositories

The diff touches only `packages/extra/monitoring`,
`packages/system/monitoring` and `packages/system/monitoring-rd`. The
website's monitoring reference page is regenerated from the package
README by the release bot, and the Terraform provider models only the
tenant's `monitoring` flag, not these values, so no follow-up is needed.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): deploy a per-tenant OpenTelemetry (OTLP) collector for traces once a VictoriaTraces backend is configured. Applications send OTLP to the in-namespace `otel-traces` Service (gRPC `:4317`, HTTP `:4318`); the collector applies head sampling (`samplingPercentage`, default 10%) and optional attribute redaction (`redactAttributes`) before exporting to the backend. Off by default (no backend configured).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added configurable trace storage in cluster or single-instance modes,
with retention settings and Grafana datasources.
* Added an enabled-by-default OpenTelemetry collector with configurable
replicas, sampling, attribute redaction, and resource limits.
* Added workload monitoring and automatic resource recommendations for
configured trace components.
  * Added readiness health checks for both tracing backend modes.

* **Bug Fixes**
* Corrected dashboard permissions for alert monitoring and trace
collector resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 16:27:45 +03:00
..

Monitoring Hub

Parameters

Common parameters

Name Description Type Value
host The hostname used to access Grafana externally (defaults to 'grafana' subdomain for the tenant host). string ""

Metrics storage configuration

Name Description Type Value
metricsStorages Configuration of metrics storage instances. []object [...]
metricsStorages[i].name Name of the storage instance. string ""
metricsStorages[i].retentionPeriod Retention period for metrics. string ""
metricsStorages[i].deduplicationInterval Deduplication interval for metrics. string ""
metricsStorages[i].storage Persistent volume size. string 10Gi
metricsStorages[i].storageClassName StorageClass used for the data. string ""
metricsStorages[i].vminsert Configuration for vminsert. object {}
metricsStorages[i].vminsert.minAllowed Minimum guaranteed resources. object {}
metricsStorages[i].vminsert.minAllowed.cpu CPU request. quantity ""
metricsStorages[i].vminsert.minAllowed.memory Memory request. quantity ""
metricsStorages[i].vminsert.maxAllowed Maximum allowed resources. object {}
metricsStorages[i].vminsert.maxAllowed.cpu CPU limit. quantity ""
metricsStorages[i].vminsert.maxAllowed.memory Memory limit. quantity ""
metricsStorages[i].vmselect Configuration for vmselect. object {}
metricsStorages[i].vmselect.minAllowed Minimum guaranteed resources. object {}
metricsStorages[i].vmselect.minAllowed.cpu CPU request. quantity ""
metricsStorages[i].vmselect.minAllowed.memory Memory request. quantity ""
metricsStorages[i].vmselect.maxAllowed Maximum allowed resources. object {}
metricsStorages[i].vmselect.maxAllowed.cpu CPU limit. quantity ""
metricsStorages[i].vmselect.maxAllowed.memory Memory limit. quantity ""
metricsStorages[i].vmstorage Configuration for vmstorage. object {}
metricsStorages[i].vmstorage.minAllowed Minimum guaranteed resources. object {}
metricsStorages[i].vmstorage.minAllowed.cpu CPU request. quantity ""
metricsStorages[i].vmstorage.minAllowed.memory Memory request. quantity ""
metricsStorages[i].vmstorage.maxAllowed Maximum allowed resources. object {}
metricsStorages[i].vmstorage.maxAllowed.cpu CPU limit. quantity ""
metricsStorages[i].vmstorage.maxAllowed.memory Memory limit. quantity ""

Logs storage configuration

Name Description Type Value
logsStorages Configuration of logs storage instances. []object [...]
logsStorages[i].name Name of the storage instance. string ""
logsStorages[i].retentionPeriod Retention period for logs. string 1
logsStorages[i].storage Persistent volume size. string 10Gi
logsStorages[i].storageClassName StorageClass used to store the data. string replicated

Traces storage configuration

Name Description Type Value
tracingStorages Configuration of traces storage instances. Empty by default (tracing off): no backend and no trace collector are rendered. Adding an entry provisions a VictoriaTraces backend and, while tracingCollector.enabled is true, deploys the OTLP collector in front of it. []object []
tracingStorages[i].name Name of the storage instance: a lowercase RFC 1123 label of at most 42 characters. Treat as immutable once a backend exists: the VTCluster/VTSingle, its Services, PVCs and Grafana datasource are all named after it, so renaming an entry deploys a new, empty backend and removes the old one, the same way changing mode does (single mode loses the old PVC, cluster mode leaves the old vtstorage PVCs Bound until the whole Monitoring release is deleted). string ""
tracingStorages[i].mode Deployment mode: "cluster" renders a VTCluster; "single" renders a single-binary VTSingle for edge/dev/small clusters. Treat as immutable once a backend exists: mode selects which Kind is rendered under the same name, so changing it on a running backend re-identifies the object rather than reconfiguring it. Switching from "single" to "cluster" prunes the VTSingle and the operator garbage-collects its PVC (stored traces are lost); switching from "cluster" to "single" (or removing the entry) leaves the vtstorage PVCs Bound and billed until the whole Monitoring release is deleted. The API cannot guard this (transition rules are not available on an array-item field), so this note is the only guard: migrate deliberately, accepting the loss. string cluster
tracingStorages[i].retentionPeriod Retention period for traces. string 14d
tracingStorages[i].storage Persistent volume size. quantity 10Gi
tracingStorages[i].storageClassName StorageClass used to store the data. string replicated
tracingStorages[i].retentionDiskUsageBytes Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. "8GB" or "8GiB"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so "8Gi" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value. string ""

Traces collector configuration

Name Description Type Value
tracingCollector OpenTelemetry Collector for trace ingestion. object {}
tracingCollector.enabled Deploy the OTLP collector. When disabled, applications must send OTLP straight to the traces backend (no central sampling/redaction). The collector fronts a single backend: while it is enabled, tracingStorages must hold at most one entry, and adding a second one fails the whole Monitoring render until the extra entry is removed or the collector is disabled. bool true
tracingCollector.replicas Number of collector replicas. Head sampling is trace-id-deterministic, so replicas>1 are safe and remove the single-ingest-path SPOF. int 1
tracingCollector.samplingPercentage Head-sampling percentage applied at the collector before export. int 10
tracingCollector.redactAttributes Span attribute keys to drop before export (the PII/secret control point, e.g. db.statement). []string []
tracingCollector.resources Resource configuration for the collector. object {}
tracingCollector.resources.requests Resource requests. object {}
tracingCollector.resources.requests.cpu CPU request. quantity 100m
tracingCollector.resources.requests.memory Memory request. quantity 256Mi
tracingCollector.resources.limits Resource limits. object {}
tracingCollector.resources.limits.cpu CPU limit. quantity 1
tracingCollector.resources.limits.memory Memory limit. quantity 1Gi

Alerta configuration

Name Description Type Value
alerta Configuration for the Alerta service. object {}
alerta.storage Persistent volume size for the database. string 10Gi
alerta.storageClassName StorageClass used for the database. string ""
alerta.resources Resource configuration. object {}
alerta.resources.requests Resource requests. object {}
alerta.resources.requests.cpu CPU request. quantity 100m
alerta.resources.requests.memory Memory request. quantity 256Mi
alerta.resources.limits Resource limits. object {}
alerta.resources.limits.cpu CPU limit. quantity 1
alerta.resources.limits.memory Memory limit. quantity 1Gi
alerta.alerts Alert routing configuration. object {}
alerta.alerts.telegram Configuration for Telegram alerts. object {}
alerta.alerts.telegram.token Telegram bot token. string ""
alerta.alerts.telegram.chatID Telegram chat ID(s), separated by commas. string ""
alerta.alerts.telegram.disabledSeverity List of severities without alerts (e.g. ["informational","warning"]). []string []
alerta.alerts.slack Configuration for Slack alerts. object {}
alerta.alerts.slack.url Configuration uri for Slack alerts. string ""
alerta.alerts.slack.disabledSeverity List of severities without alerts (e.g. ["informational","warning"]). []string []

Grafana configuration

Name Description Type Value
grafana Configuration for Grafana. object {}
grafana.db Database configuration. object {}
grafana.db.size Persistent volume size for the database. string 10Gi
grafana.resources Resource configuration. object {}
grafana.resources.requests Resource requests. object {}
grafana.resources.requests.cpu CPU request. quantity 100m
grafana.resources.requests.memory Memory request. quantity 256Mi
grafana.resources.limits Resource limits. object {}
grafana.resources.limits.cpu CPU limit. quantity 1
grafana.resources.limits.memory Memory limit. quantity 1Gi

Vmagent configuration

Name Description Type Value
vmagent Configuration for VictoriaMetrics Agent. object {}
vmagent.externalLabels External labels applied to all metrics. object {}
vmagent.remoteWrite Remote write configuration. object {}

OIDC authentication

Name Description Type Value
oidc OIDC authentication for the Grafana instance. See docs/oidc-grafana.md for the operator guide. object {}
oidc.mode Identity mode. None: no OIDC, only the grafana-admin-password Secret works. System: trust the platform cozy realm via a per-instance confidential client and audience binding; zero-config default. CustomConfig: trust a tenant-supplied issuer directly (BYO); cozy is not in the path. string None
oidc.customConfig Tenant-supplied [auth.generic_oauth] payload; consumed only when mode: CustomConfig. object {}
oidc.customConfig.config Inline map of [auth.generic_oauth] keys. Written verbatim into the Grafana CR spec.config. Mutually exclusive with secretRef.name. object {}
oidc.customConfig.secretRef Reference to a Secret holding an auth.ini fragment; mounted under /etc/grafana/oidc and loaded via GF_PATHS_CUSTOM_INI. object {}
oidc.customConfig.secretRef.name Name of an existing Secret in the release namespace whose auth.ini key holds a ready-made [auth.generic_oauth] block. Mutually exclusive with customConfig.config. string ""
oidc.users Users granted access to this Grafana instance's org. Each entry is reconciled into Grafana org membership + role by a chart-owned post-install/post-upgrade Job. Works for both System and CustomConfig modes; ignored when mode: None. Users not listed here get nothing (no default org role from group mapping). []object []
oidc.users[i].email Email address matched against the email claim from the issuer. The chart pre-provisions a Grafana account with this email so the users-Job can set the org role before the operator's first login (oauth_allow_insecure_email_lookup = true glues the two together). Must match a conservative pattern: local part is [A-Za-z0-9._%+-]+, domain is [A-Za-z0-9.-]+\.[A-Za-z]{2,}. Whitespace, quoted literals, and unusual URL-reserved characters are rejected on schema level so the reconcile-Job's shell handling never has to defend against malformed input. string ""
oidc.users[i].role Grafana org role to bind: Admin, Editor, or Viewer. Applied to the release's Grafana org (Main Org.); server-level GrafanaAdmin promotion is out of scope for Phase 1. Required — the users-Job POSTs {loginOrEmail, role} to /api/orgs/1/users and PATCHes {role} to converge across re-runs; omitting the field would render role: "null" into the body and fail with HTTP 400. string {}