mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 22:13:18 +09:00
## What this PR does Phase 2 of distributed tracing (design cozystack/community#38, epic #3761), built on the phase-1 backend (#3762, merged): a per-tenant **OpenTelemetry Collector** — the app-facing OTLP ingest gateway. It lives in the tenant namespace (like a per-tenant vmagent), so app→collector traffic is intra-namespace and never crosses the tenant boundary. No OTel operator exists in the platform, so this is a plain Deployment + ConfigMap + Service (the fluent-bit pattern). ## Changes - **`templates/vtraces/collector.yaml`** — OTLP receivers (gRPC `:4317`, HTTP `:4318`) → `memory_limiter` + head sampling (`probabilistic_sampler`, default 10%) + optional `attributes/redact` (PII/secret control point) + `batch` → `otlphttp` exporter to the `tracingStorages` backend (`vtinsert:10481` cluster / `vtsingle:10428` single, endpoints per upstream docs). PSS-restricted securityContext, digest-pinned image via `cozy-lib.image`, config-checksum roll. - **app-facing Service `otel-traces`** (4317/4318). - **`tracingCollector` values** — `enabled` / `replicas` / `samplingPercentage` (schema-bounded 0-100) / `redactAttributes` / `resources`. A **fail-guard** rejects an enabled collector fronting more than one `tracingStorages` backend (it exports to a single backend by design). - **`otel-traces` WorkloadMonitor** and dashboard resource-map entry, rendered under the collector's own gate. The same resource-map Role also gets the pre-existing `alermanager` typo corrected to `alertmanager`, so the alertmanager WorkloadMonitor is readable from the dashboard. - helm-unittest coverage for the collector and its WorkloadMonitor; regenerated `values.schema.json` / `README.md` / `monitoring-rd` ApplicationDefinition. ## Verification - `helm unittest` — `system/monitoring` 110, `extra/monitoring` 15, all pass. - `helm template` renders cleanly; `make generate` idempotent. - Live check on a test cluster (single-mode backend): the collector starts with `redactAttributes` containing a numeric, a colon-bearing and an empty key; an OTLP/HTTP span sent to `otel-traces:4318` is accepted and is queryable from the backend through the Jaeger API, with the redacted attributes removed and the others kept. - Local `/branch-review` — **LGTM** (one blocker + four recommendations fixed, each with a test: multi-backend guard, memory_limiter cgroup limit, configurable replicas, sampling bounds, trust-boundary wording). ## Scope / deferred (epic #3761) `AccountID`/`ProjectID` header injection for the shared-central topology (phase 4) and engine integrations (phase 3) are follow-up PRs. ### Screenshots N/A, no UI change. ### Downstream repositories The diff touches only `packages/extra/monitoring`, `packages/system/monitoring` and `packages/system/monitoring-rd`. The website's monitoring reference page is regenerated from the package README by the release bot, and the Terraform provider models only the tenant's `monitoring` flag, not these values, so no follow-up is needed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(monitoring): deploy a per-tenant OpenTelemetry (OTLP) collector for traces once a VictoriaTraces backend is configured. Applications send OTLP to the in-namespace `otel-traces` Service (gRPC `:4317`, HTTP `:4318`); the collector applies head sampling (`samplingPercentage`, default 10%) and optional attribute redaction (`redactAttributes`) before exporting to the backend. Off by default (no backend configured). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable trace storage in cluster or single-instance modes, with retention settings and Grafana datasources. * Added an enabled-by-default OpenTelemetry collector with configurable replicas, sampling, attribute redaction, and resource limits. * Added workload monitoring and automatic resource recommendations for configured trace components. * Added readiness health checks for both tracing backend modes. * **Bug Fixes** * Corrected dashboard permissions for alert monitoring and trace collector resources. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Monitoring Hub
Parameters
Common parameters
| Name | Description | Type | Value |
|---|---|---|---|
host |
The hostname used to access Grafana externally (defaults to 'grafana' subdomain for the tenant host). | string |
"" |
Metrics storage configuration
| Name | Description | Type | Value |
|---|---|---|---|
metricsStorages |
Configuration of metrics storage instances. | []object |
[...] |
metricsStorages[i].name |
Name of the storage instance. | string |
"" |
metricsStorages[i].retentionPeriod |
Retention period for metrics. | string |
"" |
metricsStorages[i].deduplicationInterval |
Deduplication interval for metrics. | string |
"" |
metricsStorages[i].storage |
Persistent volume size. | string |
10Gi |
metricsStorages[i].storageClassName |
StorageClass used for the data. | string |
"" |
metricsStorages[i].vminsert |
Configuration for vminsert. | object |
{} |
metricsStorages[i].vminsert.minAllowed |
Minimum guaranteed resources. | object |
{} |
metricsStorages[i].vminsert.minAllowed.cpu |
CPU request. | quantity |
"" |
metricsStorages[i].vminsert.minAllowed.memory |
Memory request. | quantity |
"" |
metricsStorages[i].vminsert.maxAllowed |
Maximum allowed resources. | object |
{} |
metricsStorages[i].vminsert.maxAllowed.cpu |
CPU limit. | quantity |
"" |
metricsStorages[i].vminsert.maxAllowed.memory |
Memory limit. | quantity |
"" |
metricsStorages[i].vmselect |
Configuration for vmselect. | object |
{} |
metricsStorages[i].vmselect.minAllowed |
Minimum guaranteed resources. | object |
{} |
metricsStorages[i].vmselect.minAllowed.cpu |
CPU request. | quantity |
"" |
metricsStorages[i].vmselect.minAllowed.memory |
Memory request. | quantity |
"" |
metricsStorages[i].vmselect.maxAllowed |
Maximum allowed resources. | object |
{} |
metricsStorages[i].vmselect.maxAllowed.cpu |
CPU limit. | quantity |
"" |
metricsStorages[i].vmselect.maxAllowed.memory |
Memory limit. | quantity |
"" |
metricsStorages[i].vmstorage |
Configuration for vmstorage. | object |
{} |
metricsStorages[i].vmstorage.minAllowed |
Minimum guaranteed resources. | object |
{} |
metricsStorages[i].vmstorage.minAllowed.cpu |
CPU request. | quantity |
"" |
metricsStorages[i].vmstorage.minAllowed.memory |
Memory request. | quantity |
"" |
metricsStorages[i].vmstorage.maxAllowed |
Maximum allowed resources. | object |
{} |
metricsStorages[i].vmstorage.maxAllowed.cpu |
CPU limit. | quantity |
"" |
metricsStorages[i].vmstorage.maxAllowed.memory |
Memory limit. | quantity |
"" |
Logs storage configuration
| Name | Description | Type | Value |
|---|---|---|---|
logsStorages |
Configuration of logs storage instances. | []object |
[...] |
logsStorages[i].name |
Name of the storage instance. | string |
"" |
logsStorages[i].retentionPeriod |
Retention period for logs. | string |
1 |
logsStorages[i].storage |
Persistent volume size. | string |
10Gi |
logsStorages[i].storageClassName |
StorageClass used to store the data. | string |
replicated |
Traces storage configuration
| Name | Description | Type | Value |
|---|---|---|---|
tracingStorages |
Configuration of traces storage instances. Empty by default (tracing off): no backend and no trace collector are rendered. Adding an entry provisions a VictoriaTraces backend and, while tracingCollector.enabled is true, deploys the OTLP collector in front of it. | []object |
[] |
tracingStorages[i].name |
Name of the storage instance: a lowercase RFC 1123 label of at most 42 characters. Treat as immutable once a backend exists: the VTCluster/VTSingle, its Services, PVCs and Grafana datasource are all named after it, so renaming an entry deploys a new, empty backend and removes the old one, the same way changing mode does (single mode loses the old PVC, cluster mode leaves the old vtstorage PVCs Bound until the whole Monitoring release is deleted). | string |
"" |
tracingStorages[i].mode |
Deployment mode: "cluster" renders a VTCluster; "single" renders a single-binary VTSingle for edge/dev/small clusters. Treat as immutable once a backend exists: mode selects which Kind is rendered under the same name, so changing it on a running backend re-identifies the object rather than reconfiguring it. Switching from "single" to "cluster" prunes the VTSingle and the operator garbage-collects its PVC (stored traces are lost); switching from "cluster" to "single" (or removing the entry) leaves the vtstorage PVCs Bound and billed until the whole Monitoring release is deleted. The API cannot guard this (transition rules are not available on an array-item field), so this note is the only guard: migrate deliberately, accepting the loss. | string |
cluster |
tracingStorages[i].retentionPeriod |
Retention period for traces. | string |
14d |
tracingStorages[i].storage |
Persistent volume size. | quantity |
10Gi |
tracingStorages[i].storageClassName |
StorageClass used to store the data. | string |
replicated |
tracingStorages[i].retentionDiskUsageBytes |
Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. "8GB" or "8GiB"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so "8Gi" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value. | string |
"" |
Traces collector configuration
| Name | Description | Type | Value |
|---|---|---|---|
tracingCollector |
OpenTelemetry Collector for trace ingestion. | object |
{} |
tracingCollector.enabled |
Deploy the OTLP collector. When disabled, applications must send OTLP straight to the traces backend (no central sampling/redaction). The collector fronts a single backend: while it is enabled, tracingStorages must hold at most one entry, and adding a second one fails the whole Monitoring render until the extra entry is removed or the collector is disabled. | bool |
true |
tracingCollector.replicas |
Number of collector replicas. Head sampling is trace-id-deterministic, so replicas>1 are safe and remove the single-ingest-path SPOF. | int |
1 |
tracingCollector.samplingPercentage |
Head-sampling percentage applied at the collector before export. | int |
10 |
tracingCollector.redactAttributes |
Span attribute keys to drop before export (the PII/secret control point, e.g. db.statement). | []string |
[] |
tracingCollector.resources |
Resource configuration for the collector. | object |
{} |
tracingCollector.resources.requests |
Resource requests. | object |
{} |
tracingCollector.resources.requests.cpu |
CPU request. | quantity |
100m |
tracingCollector.resources.requests.memory |
Memory request. | quantity |
256Mi |
tracingCollector.resources.limits |
Resource limits. | object |
{} |
tracingCollector.resources.limits.cpu |
CPU limit. | quantity |
1 |
tracingCollector.resources.limits.memory |
Memory limit. | quantity |
1Gi |
Alerta configuration
| Name | Description | Type | Value |
|---|---|---|---|
alerta |
Configuration for the Alerta service. | object |
{} |
alerta.storage |
Persistent volume size for the database. | string |
10Gi |
alerta.storageClassName |
StorageClass used for the database. | string |
"" |
alerta.resources |
Resource configuration. | object |
{} |
alerta.resources.requests |
Resource requests. | object |
{} |
alerta.resources.requests.cpu |
CPU request. | quantity |
100m |
alerta.resources.requests.memory |
Memory request. | quantity |
256Mi |
alerta.resources.limits |
Resource limits. | object |
{} |
alerta.resources.limits.cpu |
CPU limit. | quantity |
1 |
alerta.resources.limits.memory |
Memory limit. | quantity |
1Gi |
alerta.alerts |
Alert routing configuration. | object |
{} |
alerta.alerts.telegram |
Configuration for Telegram alerts. | object |
{} |
alerta.alerts.telegram.token |
Telegram bot token. | string |
"" |
alerta.alerts.telegram.chatID |
Telegram chat ID(s), separated by commas. | string |
"" |
alerta.alerts.telegram.disabledSeverity |
List of severities without alerts (e.g. ["informational","warning"]). | []string |
[] |
alerta.alerts.slack |
Configuration for Slack alerts. | object |
{} |
alerta.alerts.slack.url |
Configuration uri for Slack alerts. | string |
"" |
alerta.alerts.slack.disabledSeverity |
List of severities without alerts (e.g. ["informational","warning"]). | []string |
[] |
Grafana configuration
| Name | Description | Type | Value |
|---|---|---|---|
grafana |
Configuration for Grafana. | object |
{} |
grafana.db |
Database configuration. | object |
{} |
grafana.db.size |
Persistent volume size for the database. | string |
10Gi |
grafana.resources |
Resource configuration. | object |
{} |
grafana.resources.requests |
Resource requests. | object |
{} |
grafana.resources.requests.cpu |
CPU request. | quantity |
100m |
grafana.resources.requests.memory |
Memory request. | quantity |
256Mi |
grafana.resources.limits |
Resource limits. | object |
{} |
grafana.resources.limits.cpu |
CPU limit. | quantity |
1 |
grafana.resources.limits.memory |
Memory limit. | quantity |
1Gi |
Vmagent configuration
| Name | Description | Type | Value |
|---|---|---|---|
vmagent |
Configuration for VictoriaMetrics Agent. | object |
{} |
vmagent.externalLabels |
External labels applied to all metrics. | object |
{} |
vmagent.remoteWrite |
Remote write configuration. | object |
{} |
OIDC authentication
| Name | Description | Type | Value |
|---|---|---|---|
oidc |
OIDC authentication for the Grafana instance. See docs/oidc-grafana.md for the operator guide. | object |
{} |
oidc.mode |
Identity mode. None: no OIDC, only the grafana-admin-password Secret works. System: trust the platform cozy realm via a per-instance confidential client and audience binding; zero-config default. CustomConfig: trust a tenant-supplied issuer directly (BYO); cozy is not in the path. |
string |
None |
oidc.customConfig |
Tenant-supplied [auth.generic_oauth] payload; consumed only when mode: CustomConfig. |
object |
{} |
oidc.customConfig.config |
Inline map of [auth.generic_oauth] keys. Written verbatim into the Grafana CR spec.config. Mutually exclusive with secretRef.name. |
object |
{} |
oidc.customConfig.secretRef |
Reference to a Secret holding an auth.ini fragment; mounted under /etc/grafana/oidc and loaded via GF_PATHS_CUSTOM_INI. |
object |
{} |
oidc.customConfig.secretRef.name |
Name of an existing Secret in the release namespace whose auth.ini key holds a ready-made [auth.generic_oauth] block. Mutually exclusive with customConfig.config. |
string |
"" |
oidc.users |
Users granted access to this Grafana instance's org. Each entry is reconciled into Grafana org membership + role by a chart-owned post-install/post-upgrade Job. Works for both System and CustomConfig modes; ignored when mode: None. Users not listed here get nothing (no default org role from group mapping). |
[]object |
[] |
oidc.users[i].email |
Email address matched against the email claim from the issuer. The chart pre-provisions a Grafana account with this email so the users-Job can set the org role before the operator's first login (oauth_allow_insecure_email_lookup = true glues the two together). Must match a conservative pattern: local part is [A-Za-z0-9._%+-]+, domain is [A-Za-z0-9.-]+\.[A-Za-z]{2,}. Whitespace, quoted literals, and unusual URL-reserved characters are rejected on schema level so the reconcile-Job's shell handling never has to defend against malformed input. |
string |
"" |
oidc.users[i].role |
Grafana org role to bind: Admin, Editor, or Viewer. Applied to the release's Grafana org (Main Org.); server-level GrafanaAdmin promotion is out of scope for Phase 1. Required — the users-Job POSTs {loginOrEmail, role} to /api/orgs/1/users and PATCHes {role} to converge across re-runs; omitting the field would render role: "null" into the body and fail with HTTP 400. |
string |
{} |