mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-13 08:05:19 +09:00
f48a94de6d53d23801f4a27b016090e03ff24632
78
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 myasnikovdaniil
|
f69958b15d |
docs: move the operator guides out of docs/ to where readers find them (#3780)
## What this PR does
Five documents under `docs/` were operator facing guides written
alongside the feature PRs that added the code, and left in the source
tree where no user looks. They are on the website now, so this deletes
them here and repoints every reference.
```
docs/oidc-tenant.md -> kubernetes/oidc-authentication
docs/oidc-grafana.md -> operations/services/monitoring/oidc-authentication
docs/gpu-vgpu.md -> virtualization/vgpu
+ operations/troubleshooting/gpu-operator-host-driver
docs/vm-external-vlan.md -> networking/vm-external-vlan
docs/hubble-observability.md -> networking/hubble
```
`docs/vm-external-vlan.md` needed splitting, not moving. Its "Staging
the reference CNI plugins" section is a contract,
`hack/cni-plugins-staging-contract.bats` greps six literal sentences out
of it and asserts them against the multus Dockerfile, the daemonset
manifest and the platform values. That half is package documentation so
it moves verbatim into a new `packages/system/multus/README.md` and the
suite reads it there, the operator guide goes to the website. The bats
messages name the README now, so a future failure points at the file
that has to change.
References repointed in the `@param` blocks, template comments and test
comments that named the guides. The generated files come from `make
generate` with cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`. Where a reference points at text
that exists only on `next/` of the website (the aggregated API servers
section of the tenant OIDC guide), the comment names the page and the
section instead of a `v1.6` URL, which would not contain it.
Two changelog links in `docs/changelogs/v1.6.0.md` repointed as well.
The wording is untouched, only the location of the documents it names
has changed.
One bit of process framing dropped on the way past, per #3777:
`bundles/system.yaml` said GrafanaAdmin promotion was "out of scope for
Phase 1" when it is simply not implemented.
Tests: 10 cni-plugins contract, 47 install-wait, 346 kubernetes chart,
22 multus chart, 301 monitoring chart, rd-presets-check.
### Downstream repositories
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#648 (the four guides),
cozystack/website#754 (edits made to the originals in `docs/` after the
import), cozystack/website#647 (grafana guide made reachable in v1.6),
cozystack/website#596 (tenant kubectl guide, rebased)
Walked the rest of the trigger map against the diff. No provider
follow-up: the kubernetes `values.schema.json` diff is two `description`
strings and the monitoring one is one, with no field added, removed or
renamed, no enum, no default, and no `kind`, `plural` or
`release.prefix` change. `packages/core/platform/values.yaml` changed in
a comment only and no key, so the hand written platform-package table on
the site is unaffected.
Please merge after cozystack/website#754 and cozystack/website#647. The
`@param` URLs only resolve once those pages deploy, and
cozystack/website#754 carries the edits made to the guides since the
import, which deleting them here would lose.
### Release note
```release-note
docs: operator guides for tenant Kubernetes OIDC, Grafana OIDC, NVIDIA vGPU, VM external VLAN and Hubble now live on cozystack.io instead of the docs/ directory of this repository, and in-repo references point there. The CNI plugin staging contract that shared the VLAN guide moved to packages/system/multus/README.md.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Updated configuration and test references to link to the published
Cozystack documentation.
* Added Multus guidance covering staged CNI plugins, configuration,
troubleshooting, and rollout behavior.
* Removed outdated local guides for GPU/vGPU, Hubble observability, OIDC
authentication, and external VM VLANs.
* Clarified that server-level Grafana administrator promotion is not
currently supported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Myasnikov Daniil
|
f3486f4e3a |
docs: move the operator guides out of docs/ to where readers find them
Five guides in docs/ were written next to the feature code that needed them and left in the source tree, where no operator looks. They are on the website, so a second copy here only drifts: edits kept landing in docs/ after the website copy was made. docs/vm-external-vlan.md is split rather than moved. Its section on staging the reference CNI plugins is a contract that hack/cni-plugins-staging-contract.bats asserts against the multus Dockerfile, the daemonset manifest and the platform values, so that half is package documentation and goes to packages/system/multus/README.md, which the suite now reads. The operator half goes to the website. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Yan Bondarenko
|
9d4221b890 |
fix(monitoring): send alerts to every Alertmanager replica
vmalert had a single notifier, the headless vmalertmanager-alertmanager Service. It resolves to all three pods, but each alert reached one replica at a time and moved to another when vmalert reconnected. On clusters running 1.6.4 all active alerts sat on one replica while the other two had none, and on one of them the alert count of each replica changed about 170 times a day. The Alertmanager cluster deduplicates notifications only when every replica receives every alert. The replica an alert left let it expire and sent a resolved notification, and the replica it reached sent a new firing one. vmalert now lists each Alertmanager pod as its own notifier, using the <name>-<i>.<service>.<namespace>.svc address the operator itself gives StatefulSet pods. The replica count comes from one helper that also sets spec.replicaCount of the VMAlertmanager, so the two cannot drift. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
youdie006
|
b7708fb285 |
docs(monitoring): correct the release name the OIDC comment, guide and test fixture encode
The naming block in _helpers.tpl, the guide's opening paragraph and the oidc test fixture all describe the per-tenant Grafana as the release `monitoring`. Both instances are released as `monitoring-system` by the monitoring application and differ only by namespace: the platform one in `tenant-root`, a per-tenant one in each tenant namespace. Assisted-by: LLM Signed-off-by: youdie006 <youdie006@users.noreply.github.com> |
||
|
Aleksei Sviridkin
|
520af57905 |
chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out internal mirror block, an internal product name, and a reviewer's name in a comment do not belong in a public repository; drop them and replace what needs a stand-in with neutral placeholders. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Alexey Artamonov
|
e63b9e439c |
fix(monitoring): count a central request as the receiver admitted it
The limiter refuses for good any request it counts above burst, and burst was the receiver's 4 MiB cap. A request grows on its way to the limiter: the sampler adds a tracestate to every span it keeps, and the limiter counts a request as the collector re-encodes it, which writes the empty value of every attribute the wire may omit, so a valid request of spans with empty-valued attributes is counted 1.4x. A request the receiver admitted near its cap was therefore refused on every retry. The limiter now runs right after memory_limiter, so it counts the request before sampling. The receiver caps a request at 3 MiB and burst stays at 5 MiB, which holds 1.4x the cap; a larger burst would also be what each pod start lets through above the rate. The rate is counted before sampling, so the store keeps only the sampled share of it. The texts say that a request above the cap gets a 400, which senders do not retry, and that each collector rollout a tenant causes lets that burst through again on every pod. The replica bound that keeps that allowance small now applies only to a central tenant's collector, whose render fails above ten replicas: a collector writing to the tenant's own store has no rate, and the replica counts it shipped with stay valid. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
65f0caae2b |
fix(monitoring): keep a central tenant's write login out of its Grafana
A tenant's Grafana pod may reach the shared traces store's vmauth, and
the Secret a tenant names in oidc.customConfig.secretRef was mounted
into it whole. The apps API does not prune keys the Monitoring schema
leaves out, so a tenant could also set grafana.image: an image of its
choosing, with the collector's Secret mounted, writes past the rate
limit the collector enforces.
Grafana's image is the platform's, like the collector's, and for a
central tenant only the auth.ini key Grafana reads is mounted from the
named Secret. Other tenants have no collector login to keep out and
keep the whole Secret, which an auth.ini reading a sibling key with
$__file{} relies on.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
Alexey Artamonov
|
cc51cac63d |
fix(monitoring): give a central tenant's Grafana a login that cannot write
The collector holds a central tenant to its write rate, but the one login it writes with was also the login of the tenant's Grafana datasource, and it could both read and write. vmauth cleans the request path before it picks a route, so any user the tenant admits to its Grafana, a Viewer included, could POST through the datasource proxy to ../insert and write into the shared store past the limiter. Each central tenant now has two VMUsers on its one account: the collector's, which may only write, and Grafana's, which may only read, each with its own chart-owned Secret and password. A namespace name holds no dot, so the reader's username never matches another tenant's. On upgrade the datasource moves to the new login in the same release that creates it, so reads may fail until vmauth loads the new user. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
3497855e15 |
fix(platform): read the tracing rate only as plain decimal digits
Sprig's int64 reads a string with a leading zero as octal and turns a parse error into 0, so "0100" would reach the collectors as 64, and "08" or digits beyond int64 would leave the default in place without a word. A float beyond int64 converts differently on each CPU architecture, and the collector adds its replica count to the rate before splitting it. A string rate must now be plain decimal digits, and any rate above 1 TiB/s fails the render; the collector also falls back to the default on digits it cannot hold rather than start with a rate of zero, which the processor refuses. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
fc0db49187 |
fix(monitoring): bound what a central tenant writes above its rate
Each collector pod starts with a full token bucket, and burst was at least 64 MiB, the largest request vtinsert accepts. A tenant controls when its collector pods start, through any Monitoring change or by deleting them, and how many there are, so it could write tens of mebibytes above its rate per pod start and gigabytes per rollout of many replicas, which the two-day bound does not survive. The limiter only sees requests the receiver admits, so in central mode the receiver now caps them at 4 MiB, decompressed for HTTP, and burst drops to that floor; a pod start lets at most 4 MiB through above the rate. The replica count is capped at 10. The texts now say what the default rate guarantees: one tenant writes at most a default store's disk cap over two days counted uncompressed, rather than that it cannot fill it, and they name the per-pod-start allowance and that a sender pinned to one replica gets that replica's share. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
177b207a27 |
feat(monitoring): hold each shared-central tenant to a write rate
The design has the per-tenant collector rate-limit its tenant, so that a noisy or hostile tenant cannot exhaust a shared backend. The disk cap alone does not do that: VictoriaTraces always keeps the last two days, so a tenant writing faster than the cap allows over two days fills the disk for every tenant on the store. The platform sets the rate with monitoring.tracingCentralTenantBytesPerSecond, and it reaches every tenant in _cluster through the cozystack-values Secret. A change there is a change in the values each monitoring HelmRelease reads, so Flux upgrades them all; a rate read with lookup at render time would never reach a tenant whose release is already in sync. The platform writes the key only when it is set, since every release in every tenant reads that Secret. Without it the collector holds the tenant to the rate of a default 10Gi store. Each central tenant's collector splits the rate across its replicas and refuses data above it before the batch processor, so the sender sees the refusal. An OTLP/HTTP sender gets a 429 and retries; the limiter answers gRPC with RESOURCE_EXHAUSTED and no RetryInfo, which the OTLP specification makes final, so central tenants should export over HTTP. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
d5fc723208 |
feat(monitoring): bound every traces store by disk usage
The distributed tracing design requires each traces backend to be disk-bounded, not only age-bounded, so a full volume cannot block ingest. retentionDiskUsageBytes was optional, so a store without it filled its volume and stopped taking spans; with a shared-central store that stops ingest for every tenant at once. A store without a byte cap now drops its oldest day once its filesystem is 80% full. VictoriaTraces refuses to start with both limits set, so a byte cap, when given, replaces the percentage one. It always keeps the last two days, which the values text now says. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
27b7376bb9 |
feat(monitoring): add a shared-central tracing topology
A per-tenant traces backend costs every tenant its own VictoriaTraces stack. A tenant can now send new spans to one shared store in tenant-root instead, switched on by _namespace.tracingCentral, which the tenant chart publishes and the apps API does not let an application set. tenant-root decides whether it hosts that store, with tracingCentralHost next to a cluster-mode store named generic. VictoriaTraces takes the tenant account from request headers and does no authorization of its own, so other tenants reach the store only through a vmauth in tenant-root; tenant-root itself, with whoever it admits to its Grafana, stays inside the trust boundary. Each central tenant gets a VMUser that pins an account derived from its namespace name and UID for both writes and reads and caps its requests in flight, which the collector's two export senders stay under; its collector exports and its Grafana reads with credentials the chart generates and keeps, and vmauth overrides any account a client sends. The chart owns those credentials because the operator publishes a VMUser's own Secret only once a VMAuth selects it; the collector rolls when they change, since it reads them at start. The vmauth internal routes, whose /metrics names every tenant's user, are served on a port tenants cannot reach, and the vmauth is sized by a VPA, spread across nodes under a disruption budget, has a WorkloadMonitor like the other monitoring workloads, and may reach the apiserver, where its config-reloader reads its config Secret. Renders refuse the states in which spans would be dropped with every object Ready: a central tenant while tenant-root hosts nothing, tenant-root dropping the hosting while central tenants remain, and the host flag without a store to host. Each refusal reads the other side at render time, so a concurrent opt-in and stop-hosting can pass both; the values text names that race and its recovery. At run time tenant-root's release is not Ready while the vmauth has failed. Only the collector's export moves: local stores the tenant still lists keep rendering with their datasources, because the switch belongs to the parent tenant and the stores to this one, and the one-store limit of the collector holds in both modes, since turning central off makes it export locally again. tenant-root never counts as central. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Yan Bondarenko
|
17aaf3e038 |
feat(monitoring): add optional email alert delivery
Platform alerts need an email delivery path alongside Alerta without exposing arbitrary namespace Secrets through tenant-controlled values. SMTP credentials therefore use a fixed Secret name. Alertmanager parses the addresses and connects to the SMTP host only when it sends a notification, so a malformed value there drops alerts instead of failing the release. The chart validates these fields when it renders. Addresses must be literal dot-atom mailboxes, and a domain literal must hold a dotted-quad IPv4 address, because IPv6 literal syntax differs across the supported Go mail parsers. A bracketed smarthost must be a valid IPv4 or IPv6 address, and its port must be between 1 and 65535 without leading zeros. The repeat interval follows the Alertmanager duration grammar and range, so a value that renders is one Alertmanager loads. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Assisted-by: LLM |
||
|
IvanHunters
|
37f59369d3 |
fix(charts): fail fast on empty tenant apex in host composition
Charts that publish a service compose their hostname as <sub>.<_namespace.host>. When the injected tenant apex is empty and no per-app host override is set, they used to emit a malformed trailing-dot hostname (e.g. "harbor.") into Ingresses, HTTPRoutes and Certificate dnsNames silently. Source the apex through cozy-lib.ns-host-required at each hostname-emitting site so the render fails with a clear message instead, matching the gateway path. Covered: harbor, postgres (external), qdrant (external), kubernetes (control-plane endpoint), vpn, seaweedfs, bootbox, monitoring (grafana, alerta, oidc client), and system/bucket (adds the cozy-lib subchart symlink it lacked). The guard is placed at the point each resource is actually emitted, so standalone renders, no-exposure tenants and the kubernetes no-etcd beacon path are unaffected. The post-delete cleanup hooks are left unguarded so an empty apex cannot block uninstall. Behavior change: a tenant with an empty injected apex and no host override that enables one of these components now fails its render (HelmRelease NotReady) instead of deploying a broken hostname. In practice the apex is always injected (root has root-host, children inherit it), so this only trips a genuinely misconfigured tenant. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Alexey Artamonov
|
60b1b05434 |
feat(monitoring): add per-tenant OTLP collector for distributed tracing
Applications need one in-namespace OTLP endpoint that also enforces
sampling and redaction before spans reach the VictoriaTraces backend.
No OpenTelemetry operator ships with the platform, so the collector is
a plain Deployment, ConfigMap and Service ('otel-traces', gRPC 4317,
HTTP 4318) in the tenant namespace; app-to-collector traffic therefore
never leaves the tenant, and isolation comes from the existing tenant
network policies.
The collector renders only when it is enabled and a tracingStorages
backend is configured. Tracing is off by default, and an empty backend
list is a clean off-state rather than a render failure; an app exporter
that finds no Service simply retries. It writes to the first backend
only and rejects a list with several, instead of silently dropping
spans for the others.
redactAttributes is the PII and secret control point, so its keys are
rendered quoted and empty keys are dropped: an unquoted key with YAML
structural characters, or an empty one, makes the collector fail at
startup, and a key YAML re-types (numbers, booleans, null) silently
escapes redaction. replicas has a lower bound of 1 so a bad value fails
schema validation instead of the whole release at apply time; a zeroed
collector is expressed with enabled: false.
The collector gets a WorkloadMonitor and a dashboard-resources Role
entry under the same render gate, so it is part of tenant workload
accounting like every other workload in the chart.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
Aleksei Sviridkin
|
2d2186d1f0 |
feat(monitoring): VictoriaTraces backend + Grafana datasource (tracing phase 1) (#3762)
## What this PR does Phase 1 of distributed tracing (design: cozystack/community#38; epic: #3761): a **VictoriaTraces backend and Grafana traces datasource** in the monitoring stack, mirroring the logs (`VLCluster`) implementation. No new operator: `VTCluster`/`VTSingle` already ship in victoria-metrics-operator. - **`tracingStorages`** values in `extra/monitoring` (annotated, schema-generating) and `system/monitoring` (defaults), per-entry `mode: cluster|single`. **Empty by default, so tracing is off out of the box**: nothing produces spans until the collector ships, so no backend is provisioned until an operator adds an entry. - **`templates/vtraces/vtraces.yaml`**: `VTCluster` (default) or `VTSingle` (edge/dev/small); `retentionPeriod` plus optional `retentionMaxDiskSpaceUsageBytes`; a release-scoped **PVC cleanup label**, reused by the existing post-delete hook, so no hook change is needed. - **Render-time validation** (`_helpers.tpl`): the installed VictoriaTraces CRDs are the trimmed ones, so a bad value would otherwise reach the operator and fail after admission. Each `tracingStorages` entry is checked before anything renders: `name` must be an RFC 1123 label of at most 42 characters (the `vtstorage-<name>-<hash>` pod revision label is capped at 63) and unique across the list, `mode` must be `cluster` or `single`, `storage` must be a positive quantity (a zero size makes the operator mount an EmptyDir in single mode, and a negative one fails when the PVC is created), and `retentionDiskUsageBytes` must be a VictoriaMetrics `BytesString` (the operator rejects the Kubernetes-quantity `8Gi` grammar that `storage` uses). The `storage` field is also typed as a quantity in the schema, and the rendered name is quoted so a numeric name stays a string. - **`templates/vtraces/grafana-datasource.yaml`**: a Jaeger-typed traces datasource per entry (built-in Grafana core datasource, no plugin). - **VPA**: a `VerticalPodAutoscaler` for each trace component (`vtinsert`/`vtselect` Deployments, `vtstorage` StatefulSet, `vtsingle` Deployment), matching the metrics/logs sizing, with `updateMode: Initial`. - **Readiness gate**: `VTCluster`/`VTSingle` `healthCheckExprs` on `status.updateStatus`, alongside the existing `VLCluster` gate. - **VictoriaLogs datasource fix** (`templates/vlogs/grafana-datasource.yaml`): the range over `logsStorages` emitted no `---` between documents, so two or more entries collapsed into one document and only the last datasource survived. Each entry is now its own document. - **Tests and schema**: helm-unittest suites for off-by-default, cluster vs single, datasource type and both query URLs, retention, PVC label, VPA targets, every validation guard with its boundaries, and per-entry vlogs datasource documents, plus regenerated `values.schema.json`, `README.md` and the `monitoring-rd` ApplicationDefinition. #### Upgrade impact With default values the only rendered object that changes is the `monitoring-system` HelmRelease: it gains the `VTCluster`/`VTSingle` `healthCheckExprs` and `tracingStorages: []`. Flux evaluates those expressions only against objects in the release inventory, so they are inert while tracing is off. No VictoriaTraces object is created on upgrade. Installs with two or more `logsStorages` entries additionally gain the `GrafanaDatasource` objects that were previously dropped (see the VictoriaLogs fix above); nothing else under VictoriaLogs or VictoriaMetrics changes. #### Verification - `helm unittest`: `system/monitoring` (89 tests) and `extra/monitoring` (11 tests) pass. Each validation guard and each branch of the zero-storage pattern was mutation-checked: reverting it turns its own test red. - `helm template` renders `VTCluster`/`VTSingle`/datasource/VPA cleanly. `tracingStorages: []` (the default) renders nothing. An invalid, duplicate or over-long `name`, a zero or negative `storage`, or a bad `retentionDiskUsageBytes` (`8Gi`) is refused at render with an actionable message; a numeric `name` renders as a string. `make generate` is idempotent. - **Datasource URLs verified against upstream docs** ([VictoriaTraces and Grafana](https://docs.victoriametrics.com/victoriatraces/querying/grafana/)): cluster `vtselect:10471/select/jaeger`, single `vtsingle:10428/select/jaeger`. `jaeger` is the stable, built-in Grafana datasource; the Tempo Query API (`/select/tempo`, `type: tempo`, VictoriaTraces v0.9.4+) has a richer TraceQL UX but is upstream-experimental, so Jaeger is the default. Both URLs are pinned by unit tests. - The `retentionDiskUsageBytes` pattern, the zero-storage check and the name limit were checked against victoria-metrics-operator v0.68.4 (`BytesString.UnmarshalJSON`, `isStorageEmpty`, the `vtstorage-<name>` StatefulSet name) and the Kubernetes `ControllerRevisionName` naming. #### Scope / deferred (tracked in #3761) Per-tenant OTLP Collector, engine integrations, shared-central topology, and docs are follow-up PRs. This PR intentionally does **not** enable the OTLP/gRPC listener: apps can use `vtinsert`'s native OTLP/HTTP, and the gRPC listener lands with the collector. Builds on the now-merged design proposal cozystack/community#38. ### Screenshots Not a UI change. ### Downstream repositories Walked against the diff, no box ticked yet: - cozystack/website: `monitoring` is in the website `Makefile` `SERVICES` list, so its reference page is regenerated from `packages/extra/monitoring/README.md` automatically on the next stable tag. A user guide for tracing belongs to the docs follow-up in #3761; no website issue or PR exists for it yet. - cozystack/terraform-provider-cozystack: the provider has no `Monitoring` resource (only the tenant's `monitoring` flag), so the new schema field does not reach it. - The remaining repositories in the trigger map are not reached by this diff: no platform values, variants, installer values, `hack/` tooling, node prerequisites, ApplicationDefinition semantics or telemetry names change. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(monitoring): add a VictoriaTraces backend and a Grafana traces datasource to the monitoring stack (distributed tracing, phase 1). Tracing is off by default; enable it per tenant by adding a `tracingStorages` entry (`cluster` or `single` mode). Also fix the VictoriaLogs Grafana datasources: with two or more `logsStorages` entries, every entry except the last was silently dropped, so existing installs with several log stores gain the missing datasources on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable trace storage with cluster or single-node deployment modes. * Supports retention periods, persistent storage sizing, storage classes, and optional disk-usage limits. * Automatically configures Grafana tracing data sources for each enabled trace backend. * Monitoring readiness reflects the health of configured trace storage services. * Invalid trace storage settings are rejected during configuration. * **Bug Fixes** * Every VictoriaLogs storage now gets its own Grafana datasource; with several log storages, all but the last were previously dropped. * **Tests** * Added coverage for storage rendering, retention settings, validation guards, Grafana endpoints, health checks, and cleanup behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Alexey Artamonov
|
76445ba9b6 |
fix(monitoring): reject a negative tracingStorages storage
The storage guard only matched zero-shaped values, so -1Gi passed it and the schema's quantity pattern alike, rendered into the VTCluster/VTSingle spec, and left the release failing under retries: -1 once the API server refused the PVC. Refuse a leading minus in the same guard, so the error names the field before anything is applied and matches what the message promises. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
Alexey Artamonov
|
37789e1768 |
fix(monitoring): reject zero storage and over-long names in tracingStorages
Two tracingStorages values render cleanly and only fail once the operator
acts on them.
- A zero storage is silently destructive in single mode: the operator's
isStorageEmpty gate checks IsZero() and mounts an EmptyDir instead of a
PVC, the VTSingle reports operational and the release goes Ready, and every
stored span is lost on the next reschedule. Type the storage field as a
quantity so the schema refuses non-quantities at admission, and refuse
every zero spelling ("", 0, 0Gi, 0.0, .0, 0e3, 0m, +0, -0Mi) in the render
guard, since the quantity pattern accepts those.
- A name over 42 characters wedges cluster mode: the operator names the
StatefulSet vtstorage-<name>, and each pod gets the label
controller-revision-hash: vtstorage-<name>-<hash>, whose value is capped at
63 with up to 10 characters of hash. Past that no pod is created, the
VTCluster goes failed and the release stays not-Ready under retries: -1.
Cap the length in the same guard.
- Quote the rendered name, so a numeric name such as 2024 stays a string
instead of failing the apply on metadata.name.
- Rewrite the helper's rationale so it matches the trimmed CRDs and the
pinned operator, and state in the mode field description that the API
cannot enforce immutability on an array-item field, so the note is the only
guard.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
|
||
|
Alexey Artamonov
|
05eebbdf4f |
refactor(monitoring): keep tracing rationale in template comments
The `#` prose in the traces datasource and in the HelmRelease health-check
block is not stripped at render the way `{{/* */}}` is, so it shipped into
every tenant's rendered GrafanaDatasource and HelmRelease. Move it into
template comments: the reasoning stays in the chart source but no longer lands
in the emitted objects.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
|
||
|
Alexey Artamonov
|
30fcfe6877 |
fix(monitoring): emit each vlogs GrafanaDatasource as its own document
The range over logsStorages emitted its documents with no `---` separator, so two or more entries concatenated into a single document with duplicate keys, each later entry overwrote the earlier one, and every log store but the last silently got no datasource. Add the separator inside the range, matching the traces datasource template. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
Alexey Artamonov
|
f20bd7b334 |
fix(monitoring): validate tracingStorages entries and flag mode/retention hazards
The installed VictoriaTraces CRDs are the trimmed ones (crds.plain: false), so their spec is x-kubernetes-preserve-unknown-fields and a bad value reaches the operator instead of being refused by the API server. Catch the ones a render guard can, and document the one it cannot. - name is used verbatim as the VTCluster/VTSingle and GrafanaDatasource metadata.name, so an invalid name is a late admission error and a duplicate silently overwrites the first entry (last write wins in the range). Guard it against an RFC 1123 label and a uniqueness set. - retentionDiskUsageBytes maps to the operator's BytesString field, whose UnmarshalJSON rejects anything outside ^[0-9]+(kb|mb|gb|tb|KB|MB|GB|TB|KiB|MiB|GiB|TiB)?$, notably not the Gi grammar of the sibling storage field. The operator then marks the CR failed and the release fails well after the value was written. Refuse it at render with the upstream pattern instead. - changing mode re-identifies the object instead of reconfiguring it, so switching single to cluster prunes the VTSingle and the operator garbage-collects its PVC, and cluster to single (or dropping the entry) leaks the vtstorage PVCs until the release is deleted. A stateless template cannot see the prior mode, so record the loss and the one-way migration in the field description. The name, uniqueness, mode and retention checks share one pass in a helper; the inline mode guard in vtraces.yaml moves there. The retention description also notes that the cap is per vtstorage replica in cluster mode. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
Alexey Artamonov
|
41aa2aad1d |
feat(monitoring): add VPA for VictoriaTraces backend components
vpa.yaml sized only metrics and logs components; the trace components got no VerticalPodAutoscaler, so they sat at the operator's static defaults with no way to size them (tracingStorages has no resource knobs and the template hardcodes resources/replicaCount). Add VPAs per tracingStorages entry, branching on mode: cluster mode gets vtinsert and vtselect (Deployments) + vtstorage (StatefulSet), single mode gets vtsingle (Deployment). Kinds verified against victoria-metrics-operator v0.68.4. updateMode: Initial like the rest of the file (no eviction, because of the Cilium IP-reuse race), which is also required for VTSingle's plain RWO PVC (an evicting VPA would deadlock on multi-attach). Bounds are hardcoded, without the dead min/maxAllowed conditionals the logs entries carry (tracingStorages declares no such properties). Empty tracingStorages renders no trace VPA. Tests cover the cluster three-VPA split, the single-mode single VPA with its updateMode, and off-by-default. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
Alexey Artamonov
|
00ff19ec5c |
fix(monitoring): default tracing off and defer fail-on-empty guard
tracingStorages had two defaults (extra + system), so 'tracingStorages: null' silently backfilled the inner default and a VTCluster rendered anyway, while 'tracingStorages: []' aborted the whole monitoring release. There was no clean 'off', and the default shipped a 6-pod VTCluster per tenant for a pipeline that nothing writes spans into yet. - Default tracingStorages to [] in both packages/system/monitoring/values.yaml and packages/extra/monitoring/values.yaml, so absence has nothing to fall through to and the generated schema advertises the shipped (off) state. - Drop the fail-on-empty guard in templates/vtraces/vtraces.yaml. Nothing produces spans without a collector, so an absent backend drops no data. The guard belongs with the collector, where it becomes correct: an enabled producer with no backend really would drop spans, the logsStorages #3181 rationale. - healthCheckExprs stay unconditional (Flux evaluates them only against CRs present in the inventory, so they are inert when nothing renders). - Tests: replace the empty-fails case with off-by-default (renders nothing); set tracingStorages explicitly where cases relied on the old default. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
Alexey Artamonov
|
3774e09159 |
fix(monitoring): reject unsupported tracingStorages mode
tracingStorages[].mode accepted any string, so a typo (e.g. 'singel') silently rendered a VTCluster instead of failing. Add a render guard that fails on anything but cluster|single, restrict the schema property with a TracingMode enum, and add a helm-unittest case asserting the rejected mode. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
Alexey Artamonov
|
32f170949e |
test(monitoring): pin verified VictoriaTraces query URLs; jaeger stable default
Datasource service/port/path verified against upstream docs (https://docs.victoriametrics.com/victoriatraces/querying/grafana/): cluster vtselect:10471/select/jaeger, single vtsingle:10428/select/jaeger. Jaeger is the stable, built-in Grafana core datasource; the Tempo API (/select/tempo, type: tempo, VictoriaTraces v0.9.4+) is upstream-experimental, kept as a documented alternative. Add helm-unittest assertions pinning both the cluster and single-mode query URLs. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
Alexey Artamonov
|
c2fce3d41d |
feat(monitoring): add VictoriaTraces backend + Grafana datasource for tracing
Phase 1 of the distributed-tracing design proposal (cozystack/community#38): a per-tenant traces backend in the monitoring stack, mirroring the logs (VLCluster) implementation. No new operator: VTCluster/VTSingle already ship in victoria-metrics-operator. - tracingStorages values (extra + system), per-entry mode cluster|single - templates/vtraces/vtraces.yaml: VTCluster (default) / VTSingle (edge/dev), #3181-style empty-list render guard, retentionPeriod + optional disk cap (retentionMaxDiskSpaceUsageBytes), release-scoped PVC cleanup label - templates/vtraces/grafana-datasource.yaml: Jaeger-typed traces datasource - extra/monitoring readiness gate: VTCluster/VTSingle healthCheckExprs on status.updateStatus, alongside VLCluster - helm-unittest suites (required/empty-guard, cluster-vs-single, datasource, PVC label) + regenerated values.schema.json / README / ApplicationDefinition The per-tenant OTLP Collector, engine integrations, shared-central topology and docs are out of scope and tracked in the tracing epic. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
Alexey Artamonov
|
949096694b |
fix(platform): reject non-string realm names and shell-quote the hook realm
A realmName that YAML resolves to an int or a bool (a bare 123, or a reserved word like on/off) was coerced with toString, silently renaming the realm to something the operator never typed (on -> "true"). Fail the render with a quoting hint at the source instead of surfacing it later as a broken issuer URL. The update-email hook Job interpolated the realm into its shell script unquoted, so a name carrying a space or a shell metacharacter could break or inject into the script; single-quote it. The Grafana and managed-cluster issuer helpers build the URL with printf %s straight off the channel value, so guard them with toString for the same non-string reason. Name the configured realm in the tenant kube-apiserver oidc.mode description and the Grafana identity-mode comment, which still read cozy as if it were fixed. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
44a9a8e735 |
feat(platform): make the Keycloak realm name configurable
The realm name "cozy" was hardcoded in keycloak-configure (which owns the realm, its clients and the cozystack-cluster-admin roles), in the dashboard and linstor-gui gatekeepers, in the info kubeconfig, and in the Grafana and managed-Kubernetes apiserver OIDC issuers. Operators could not rename the realm without patching vendored templates, which `make update` reverts. The apiserver OIDC issuer is already configurable via talm oidcIssuerUrl, so pointing it at a custom realm left every SSO component on "cozy" and broke login with an issuer mismatch. authentication.oidc.realmName (default "cozy") now rides the _cluster channel so every consumer resolves the same realm, and the default keycloakInternalUrl is derived from it so the realm is named in one place. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Andrei Kvapil
|
705e39d3d7 |
feat(monitoring): install Grafana plugins from the in-cluster mirror
Point GF_INSTALL_PLUGINS at the grafana-dashboards static file server instead of the public Grafana catalog, so plugin installation needs no internet egress at pod startup and keeps working on air-gapped installs. Route the upstream Grafana image reference through cozy-lib.image so mirrored-registry installs keep working, and add regression tests for the image reference and the plugin install path. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
44ff0fb286 |
feat(monitoring): run Grafana on the upstream image with catalog plugins
Point the Grafana deployment at the upstream grafana/grafana image and install the VictoriaLogs datasource and panel plugins at startup via GF_INSTALL_PLUGINS, pulled from the official Grafana plugin catalog (https://grafana.com/grafana/plugins) instead of being baked into a custom rebuilt image. The VictoriaLogs datasource is now published and signed in the catalog (victoriametrics-logs-datasource), so allow_loading_unsigned_plugins and the custom /var/lib/grafana-plugins path are no longer needed; plugins land in the writable default GF_PATHS_PLUGINS. Plugin versions are pinned to releases compatible with the shipped Grafana version. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Ivan Okhotnikov
|
0164a6afa4 |
fix(monitoring): set groups_attribute_path=groups so allowed_groups actually works on Grafana v11.x
Verified end-to-end on dev3 with Grafana v11.6.15: OAuth flow reaches Grafana, id_token and userinfo response both carry the correct `groups` claim (`[cozystack-cluster-admin, tenant-root-super-admin]` for kvaps), `allowed_groups` in grafana.ini lists `tenant-root-super-admin`, and login still fails with `user not a member of one of the required groups`. Grafana debug log shows `Attributes: map[]` — the extracted user's Groups slice is empty. On v11.x+ Grafana does not auto-map the `groups` claim onto user.Groups unless `groups_attribute_path` is set explicitly (the field is documented as optional but the gate is a silent no-op without it — a claim-name-string is not enough). Fix: hard-wire `groups_attribute_path=groups` in the chart-forced System-mode block alongside `allowed_groups`. "groups" points at the top-level `groups` array from userinfo, matching the shape Keycloak's oidc-group-membership-mapper emits (verified on the same dev3 run). Two helm-unittest cases updated to assert `groups_attribute_path` lives next to `allowed_groups` in both the users-map-active and empty-users branches (rejecting the silent no-op regression). Docs also updated to name the requirement. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
b002c967ac |
fix(monitoring): close cross-tenant Grafana OIDC hole via tenant-group gate
Round-3 review from lllamnyp: any user in the shared `cozy` Keycloak
realm could sign into any tenant's Grafana. The audience mapper stops
replay of a minted token across releases, but does not gate who may
mint one — tenant-alice could complete the OAuth flow against
tenant-bob's Grafana. With `spec.oidc.users` non-empty the earlier
`allow_sign_up=false` fix mostly closed the door (though tenant-alice
still hit `error 401` after landing in Grafana, revealing the release
exists); with empty users under mode: System — a config the schema
permits and the docs describe as hands-off — the login succeeded and
minted a Viewer account for every cozy-realm identity.
The fix uses the KeycloakRealmGroups the tenant chart already
provisions (`<namespace>-{view,use,admin,super-admin}` in
packages/apps/tenant/templates/keycloakgroups.yaml) — no new
directory objects owned by the Monitoring release. Two chart edits:
1. oidc-keycloak.yaml: add the platform-managed realm-default `groups`
scope to `defaultClientScopes` explicitly. EDP's KeycloakClient
does NOT merge realm-level defaults into `defaultClientScopes`, so
the scope must be listed here or the `groups` claim would be
missing from the token.
2. grafana.yaml: in System mode, unconditionally emit
`allowed_groups: <ns>-view <ns>-use <ns>-admin <ns>-super-admin`
in `auth.generic_oauth` and add `groups` to `scopes`. Grafana
rejects any login whose token does not carry at least one of the
groups.
The gate is chart-owned and does not depend on the users-map — with
`users: []` tenant members still get a self-service Viewer through
`auto_assign_org_role`, cross-tenant users get 401 in Keycloak's OAuth
callback. Docs updated to reflect the two-layered contract
(tenant-membership gate + optional users-map).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
f64ba91872 |
fix(monitoring): do not render users-Job when spec.oidc.users is empty
Address PR#3176 review round-2 gap-2: the users-Job's prune pass deletes every non-admin Main-Org member whose email is not in the desired list. When the desired list is empty every helm-upgrade wiped the whole org. Three empty-users states used to fall into this trap: * customConfig.secretRef: users[] is forbidden by assertCustomSecretRefUsersEmpty, so the desired list was structurally always empty and the chart wiped an org whose auth.ini fragment the operator advertised as authoritative. * System with users[] unset: the operator opted into OIDC but not into the chart-managed users-map, yet the Job still pruned every Main-Org member on their behalf. * CustomConfig-inline with users[] unset: same shape as System. The mode=None guard (from an earlier round) already had the same shape; extend the render gate to cover the other empty-list doors — now the Job renders only when `mode != None AND users non-empty`. The trade-off: taking `users:` back to `[]` no longer prunes the last entry — matching the already-documented `System → None` posture — called out explicitly in the docs. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
ff4ef2c189 |
fix(monitoring): force allow_sign_up=false and gate isolation dict on users-map
Address PR#3176 review round-2 gap-1: the docs promise "users not in spec.oidc.users get nothing", but Grafana's default allow_sign_up=true combined with the chart-forced skip_org_role_sync=true was minting a Viewer account for every cozy-realm identity that completed the OAuth flow, then reaping it on the next helm-upgrade prune pass. The chart-forced isolation dict (now three keys: skip_org_role_sync, oauth_allow_insecure_email_lookup, allow_sign_up=false) is applied only when spec.oidc.users is non-empty. A BYO-IdP operator who does not use the users-map gets none of the three forced on top of their inline config, matching the review's minor: forcing insecure_email_lookup against an issuer that may not verify emails is a needless account- linking surface, and allow_sign_up=false with no pre-provisioned accounts would lock everyone out. Tests split the two branches: users-active asserts all three keys are present and the operator cannot override them via CustomConfig-inline; users-empty asserts the auth.generic_oauth block is emitted without any of the isolation keys. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
b2fd32385e |
fix(monitoring): rewrite users-Job shell in POSIX (alpine BusyBox compat)
CI E2E caught the Job container crash-looping within 20s of the first Start event. Root cause: the `alpine/k8s:1.33.4` image is Alpine-based (BusyBox coreutils + BusyBox grep), and the Job shell used four GNU-specific constructs that BusyBox does not implement: - `head --bytes=N` (BusyBox `head` only accepts `-c N`). - `tr --delete` (BusyBox `tr` only accepts `-d`). - `jq --raw-output0` (NUL-terminated output — jq 1.7+ feature; also paired with `read -r -d ''` which is a bash-ism, not POSIX `sh` and not honoured by BusyBox `ash`). - `grep --null-data --fixed-strings --line-regexp` (BusyBox `grep` supports `-F` and `-x` short forms but NOT `-z`/`--null-data`). Rewrite: - `head -c 24 /dev/urandom | base64 | tr -d '=+/' | head -c 32` — same random-32-char password, portable to BusyBox. - Emit desired emails and org membership one record per line (jq `--raw-output`) instead of NUL-terminated. Safe because the template-level `monitoring.oidc.assertUsersEmailShape` guard already rejects any email containing whitespace / control characters, so line-oriented iteration cannot get confused. - Read TSV membership records with `IFS="$(printf '\t')"` (POSIX- portable TAB) + plain `read -r uid ulogin uemail` — no need for `-d ''` once records are newline-terminated. - Match desired list with `grep -Fxq -e "$uemail" -e "$ulogin"` — short-form flags supported by both GNU and BusyBox `grep`. Syntax validated with both `sh -n` and `bash -n` on the extracted inline script. helm-unittest 34/34 still green (render shape is unchanged, only the shell literal is different). Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
1422495b88 |
fix(monitoring): do not render users-Job when mode=None (upgrade safety)
Caught by dev3 end-to-end test on the PR working tree
(cozystack-pr-test skill): the users-Job was rendered in every mode
including `None` on the rationale that a `mode: System → None`
switch should garbage-collect OIDC-provisioned local accounts. That
rationale ignored the upgrade path from the pre-PR chart: an
existing tenant that never had `spec.oidc` on their Monitoring CR
would receive `mode: None` on first render under the PR, the Job
would fire on every Flux reconcile of the outer HR, and its prune
pass would DELETE every non-admin Main-Org member — including users
the tenant added by hand through the Grafana UI on the pre-PR chart.
Silent breaking change with data loss.
Wrap the whole Job resource in `{{- if ne $mode "None" }} … {{- end }}`
so `mode: None` = "chart owns nothing in Grafana orgs, we don't
touch what's there". Operators who need to prune OIDC-provisioned
accounts after switching System | CustomConfig → None do it
themselves through the Grafana UI or admin API — same posture as
"we don't own the Keycloak directory objects" from the earlier
architectural change.
The `assertUsersPrologue` include stays outside the `if` so an
operator listing users in `mode: None` still gets the schema and
email-shape validation at render time — the Job body just doesn't
emit.
Test coverage:
- Flip the previous "mode=None still renders the users-Job so a
mode switch can prune stale members" case to
`mode=None does NOT render the users-Job (upgrade safety guard)`
as a regression lock.
- Add `mode=None with non-empty users still fails render` covering
the prologue-only-fires-on-schema-issue path.
Docs: rewrite the "Users and RBAC" section step 3 to name the
upgrade-safety motivation and the manual-cleanup handoff.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
c31a9f8ec3 |
fix(monitoring): promote oidc.users[].role to required + regenerate schema
The `OIDCUser` typedef in `packages/extra/monitoring/values.yaml`
declared only `@field {string} email`; `cozyvalues-gen` regenerated
`values.schema.json`, `README.md`, and the mirrored `openAPISchema`
on the `Monitoring` ApplicationDefinition WITHOUT a `role` field.
Consequences:
- The `@enum OIDCRole` declaration was orphan — referenced from
nowhere, so the schema's items object had `required: ["email"]`
and one property.
- A dashboard-driven CR would offer a form without any role picker;
a schema-conformant `{email}` submission would render into the
users-Job with `role: null`, `jq --raw-output` yields the string
`"null"`, the `POST /api/orgs/1/users` body carries
`"role":"null"`, Grafana returns HTTP 400, and the post-install
hook exhausts `backoffLimit: 6` — HelmRelease never becomes Ready.
- Unit + e2e tests all hardcoded `role: Admin` / `role: Viewer`, so
the contract mismatch never surfaced in CI.
Fix:
- Add `@field {OIDCRole} role - ...` to the `OIDCUser` typedef in
`packages/extra/monitoring/values.yaml` so cozyvalues-gen materialises
the field with enum + required.
- Regenerate `values.schema.json`, `README.md`, and mirrored
`openAPISchema` via `make generate`.
- Add `monitoring.oidc.assertUsersRoleShape` template-level guard
(same shape as `assertUsersEmailShape`) so a stale-schema CR or a
direct-helm invocation fails at render time with a legible error
instead of at hook execution with a Grafana 400.
- Consolidate all `spec.oidc.users` guards into a single
`monitoring.oidc.assertUsersPrologue` include and call it once from
`oidc-users-job.yaml`, ordering the incompat check
(`assertCustomSecretRefUsersEmpty`) before the per-entry role /
email regexes so operators see the more actionable message first.
Drop the duplicate include from `grafana.yaml` (Job template renders
in every mode anyway).
- Two new helm-unittest cases lock the regression:
`oidc.users entry missing role fails the render` and
`oidc.users entry with unknown role fails the render`.
- Point the previous `secretRef + users` incompat unit test at the
users-Job template since that's where the prologue now fires.
- `docs/oidc-grafana.md`: correct the failed-hook post-mortem window
claim — with `ttlSecondsAfterFinished: 3600` the Kubernetes TTL
controller reaps the Job + Pod 1h after the terminal condition,
not "until the next helm-upgrade".
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
26e5f06d27 |
fix(monitoring): chart-force skip_org_role_sync + email lookup in every OIDC mode
The chart previously injected the two settings the users-Job's
contract depends on — `skip_org_role_sync = true` and
`oauth_allow_insecure_email_lookup = true` — only inside the
`mode: System` branch of `grafana.yaml`. `values.yaml` and
`docs/oidc-grafana.md` both promised the settings would be forced
regardless of mode, but under `mode: CustomConfig` the operator's
inline `auth.generic_oauth` map was written verbatim, both
settings were absent, and the users-Job's role assignments were
silently stranded (no lookup by email → new Grafana account minted
at login) or transiently overwritten (no skip_org_role_sync → any
`role_attribute_path` or Grafana's `auto_assign_org_role: Viewer`
overwrote the Job's role on each login).
Fix: merge a `$chartForcedOidc` dict on top of both the generated
System block and the operator's inline map (`merge` wins for
overlapping keys, so the operator can NOT override the two
settings). CustomConfig secretRef mode cannot receive the merge
because the ini fragment is authoritative — the chart now hard-fails
render when both `customConfig.secretRef.name` and non-empty
`spec.oidc.users` are set, forcing the operator to either switch to
inline config or manage authorization themselves inside the ini.
Also:
- Align the inner HR `timeout` on `packages/extra/monitoring/templates/helmrelease.yaml`
from 10m to 15m so it matches the outer cozyrds
`release.cozystack.io/helm-install-timeout: "15m"` and does not
fail while the users-Job is still inside its
`activeDeadlineSeconds: 900`.
- Route the users-Job image through `cozy-lib.image` so
mirrored / air-gapped installations resolve it (matches the
pattern in `packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml`).
- Add `spec.ttlSecondsAfterFinished: 3600` on the users-Job so a
succeeded Job + Pod are garbage-collected on the next reconcile
interval rather than lingering until the next hook activation
(which never fires on release uninstall).
- Harden the users-Job shell:
* emails read NUL-terminated (`jq --raw-output0`) so an address
containing whitespace cannot corrupt the `for email in ...`
loop's field splitting;
* `/api/users/lookup?loginOrEmail=<email>` URL-encoded via
`jq @uri` so `&`/`=`/`?` in the local part parse correctly on
the Grafana side;
* prune loop switched from pipe-delimited fields to TSV inside
NUL-terminated records, with `cut -f` for extraction.
- Add a template-level assertion (`monitoring.oidc.assertUsersEmailShape`)
rejecting emails that do not match a conservative regex —
cheaper failure mode than a 400 from Grafana.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
01d68904ea |
feat(monitoring): drop chart-owned Keycloak groups; app-side users-map + reconcile Job
Replace the group-based authorization model on the Phase-1 Grafana OIDC selector with an app-side users map, mirroring the tenant kube-apiserver sibling in cozystack/cozystack#3044. What is removed - The three chart-owned KeycloakRealmGroup objects in the cozy realm (`<clientId>-{admin,editor,viewer}`). A Helm release must not own directory objects — toggling `spec.oidc.mode: System -> None` or deleting the Monitoring release would Helm-delete the groups, and with them any membership curated out-of-band. Per-instance groups in the platform realm is exactly the design question left open at the end of cozystack/community#24; establishing it as precedent from a monitoring PR is out of scope. - Dash-separated group naming (`<namespace>-<release>-<role>`) was also collision-prone against tenant/subtenant paths, since both the namespace and release names can themselves contain dashes. - `role_attribute_path` in the Grafana `generic_oauth` block and the default-Viewer fallback, which handed org-Viewer to every cozy realm identity that logged in — cross-tenant metrics read across every opted-in Grafana. Also removed the `groups` client scope from the KeycloakClient defaults. - `monitoring.oidc.allowAssignGrafanaAdmin` helper and the internal `oidc.grafanaAdmin` values key: the feature was already deferred and the helper's release-name-only guard was tenant-spoofable. What replaces it - `spec.oidc.users: [{email, role: Admin|Editor|Viewer}]` on the Monitoring chart values. - A chart-owned post-install/post-upgrade Job (`templates/grafana/oidc-users-job.yaml`) that reconciles that list into Grafana's Main Org. via the admin API: pre-provisions a local Grafana account per listed email so the reconcile can set the org role before the operator's first OIDC login, adds the account to the org, PATCHes the role to converge across re-runs, and prunes every non-admin Main-Org member whose email is not in the list (handles both entry removal and `mode: None`). - `skip_org_role_sync = true` + `oauth_allow_insecure_email_lookup = true` in the generic_oauth block. The former stops a login from overwriting the Job's assignments; the latter binds the OIDC identity to the pre-provisioned local account. - Job talks to Grafana over the in-cluster Service and reads admin creds from the chart-managed `grafana-admin-password` Secret via `envFrom`: no ServiceAccount / Role / RoleBinding. `activeDeadlineSeconds: 900` caps a stuck Job so a broken Grafana cannot hold the release in `pending-upgrade`. Defensive rendering - `.Values.oidc | default dict` in the four templates that dereference the `oidc` block so `oidc: ~` or an omitted block no longer nil-derefs. - `oidc-client-secret.yaml` falls back to a fresh random when the existing Secret lookup succeeds but `data.client-secret` is empty. Rationale: cozystack/cozystack#3176 review from @lllamnyp (architectural CHANGES_REQUESTED points 1, 2, 3 + the default-Viewer leak). The three points stack: dropping the groups incidentally fixes the naming-collision and the default-Viewer leak, and the users-Job gives operators the same UX story as the tenant kube-apiserver PR across both sides of the platform. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
b46543e5f2 |
fix(monitoring): defer platform GrafanaAdmin lever to a future phase
The MAJOR finding from the cozy-review agent: the platform-side `oidc.grafanaAdmin: true` documented in the PR body and in docs/oidc-grafana.md never actually reached the platform Grafana's HelmRelease. `Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin` was set in packages/core/platform/templates/bundles/system.yaml, but `monitoring-system` (and `monitoring`) have no `install:` block in packages/core/platform/sources/monitoring-application.yaml — only `monitoring-rd` does. PackageReconciler skips components without an `install:` block before ever reaching the assignment that copies `pkg.Spec.Components[].Values` onto `hr.Spec.Values` (internal/operator/package_reconciler.go around lines 197-201 and 292-293). The values-override dictionary was silently dropped, no HR carried `oidc.grafanaAdmin: true`, and the platform Grafana rendered with the chart default (`false`). The `bundles_monitoring_grafana_admin_test.yaml` unit test only asserted against the emitted Package CR spec — exactly the layer where the value still existed but was about to be discarded — so the broken propagation slipped past. Trim the broken promise from Phase 1. The chart-side helper (`monitoring.oidc.allowAssignGrafanaAdmin`) stays as-is so a future phase that routes the override through the tenant-root chart's monitoring HR values (or adds an `install:` block on the `monitoring-system` component) lands on a green chart-side contract. The helm-unittest that verifies chart-side emission stays too, but with a docstring calling out that end-to-end wiring is deferred. Everything else that promised the lever works — bundle line, docs, platform bundle test — is either removed or rewritten to match reality. Also picks up two MINOR findings from the same review: * Drop the misleading `secret-hash` annotation on the OIDC client-secret Secret. The comment claimed it "rotates the KeycloakClient CR and therefore the Grafana Deployment env when the client-secret changes", but nothing downstream watches an annotation on the Secret itself — no rotation is triggered. The `lookup`-based fallback already keeps the value stable across reconciles, so no rotation is even needed. Removing the dead annotation avoids planting a broken assumption for future readers. * Add `release.cozystack.io/helm-install-timeout: "15m"` on the monitoring cozyrds ApplicationDefinition. Under `mode: System` the chart now emits, on top of the existing Grafana + Alerta + VMCluster + CNPG stack, four additional Keycloak CRs whose reconciliation is asynchronous against an operator in `cozy-keycloak` (not the tenant namespace). Flux's default 5m install window was already tight for the pre-OIDC stack; 15m matches the precedent set by tenant-rd/kubernetes-rd cozyrds. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
586e77c361 |
fix(monitoring): hard-fail System OIDC render on missing Keycloak CRD
`mode: System` previously wrapped every Keycloak resource in the `Capabilities.APIVersions.Has "v1.edp.epam.com/v1"` guard, so a cluster where the Keycloak operator CRDs were not yet registered silently dropped the whole Keycloak side while Grafana still came up with an `auth.generic_oauth` block pointing at a client that never got provisioned. Login broke with an opaque OIDC error and no chart-side signal that the platform was misconfigured. Turn the missing CRD into an actionable render error. A new `monitoring.oidc.assertSystemKeycloakCRD` helper fails with a message that points at `cozystack.keycloak-operator`, called from both `templates/grafana/oidc-keycloak.yaml` (replacing the silent capability guard) and `templates/grafana/grafana.yaml` (symmetric defense-in-depth so a future refactor of oidc-keycloak.yaml cannot regress the invariant). For the platform `monitoring-system` release the oidc variant's `dependsOn: cozystack.keycloak-operator` already prevents the race; for tenant releases the assertion converts the misconfiguration into an operator-facing error instead of a broken login flow. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
ffeb8e5e4f |
fix(monitoring): close self-review blockers for the OIDC PR
Address the two blocking-class findings from the cozy-review pass on
this PR:
(1) Missing PackageSource dependency on cozystack.keycloak-operator
(MAJOR cold-install race).
Without this edge, the monitoring chart can reconcile before the
v1.edp.epam.com CRDs are registered on a fresh cluster — the
`.Capabilities.APIVersions.Has` guard in
templates/grafana/oidc-keycloak.yaml then silently drops all
Keycloak-side objects for mode: System, and Grafana's
auth.generic_oauth block points at a client Keycloak never
provisions. Add the ordering edge, plus a
sources_monitoring_application_dependson_test.yaml regression
guard so a future edit cannot drop it silently.
(2) allow_assign_grafana_admin driven by `.Release.Name ==
monitoring-system` (MINOR fragile coupling).
Replace the release-name sniff with an explicit
`oidc.grafanaAdmin: false` values field, and flip it on for the
platform Grafana release via
Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin
in packages/core/platform/templates/bundles/system.yaml —
mirroring the pattern monitoring-agents already uses for
per-component overrides. Tenant Grafana releases inherit the
chart default false and stay at org-level Admin.
Two new helm-unittest cases in packages/system/monitoring/tests/
lock the values-driven behaviour on both sides
(grafanaAdmin=true → Admin uplift, default → tenant stays at org
Admin), plus bundles_monitoring_grafana_admin_test.yaml in
packages/core/platform/tests/ regression-guards the Package's
component override so the platform release keeps its uplift on
future edits.
Also document the mode-toggle group-membership loss in
docs/oidc-grafana.md — mode: System → CustomConfig/None deletes the
three chart-owned KeycloakRealmGroups; membership is not restored on
toggle-back. Not a bug, but the operator UX cost of the toggle
deserves a bullet in Failure modes.
Regenerated artifacts (cozyvalues-gen + hack/update-crd.sh):
packages/extra/monitoring/values.schema.json
packages/extra/monitoring/README.md
packages/system/monitoring-rd/cozyrds/monitoring.yaml
Test suites still green: packages/system/monitoring 23/23,
packages/core/platform 81/81.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
|
||
|
Ivan Okhotnikov
|
8eff03b323 |
feat(monitoring): wire auth.generic_oauth into the Grafana CR
Patch the Grafana CR so that the oidc.mode selector drives the actual Grafana runtime config: - mode: System — emit a full auth.generic_oauth section under spec.config with client_id / client_secret / auth_url / token_url / api_url / scopes / role_attribute_path / allow_assign_grafana_admin built from the helpers, and inject GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET as an env from the persistent Secret introduced in the previous commit. role_attribute_path maps membership in the per-instance -admin / -editor / -viewer groups to Grafana's Admin / Editor / Viewer roles, defaulting to Viewer on authenticated identities with none of them. allow_assign_grafana_admin is set to true only for the platform release (Release.Name == 'monitoring-system'); tenant Grafana stays at org-level Admin. - mode: CustomConfig (inline) — pass the operator-supplied config map verbatim to spec.config.auth.generic_oauth. `cozy` is not in the path. - mode: CustomConfig (secretRef) — mount the operator-managed Secret under /etc/grafana/oidc and load it via GF_PATHS_CUSTOM_INI. The ini fragment is authoritative; the chart does not emit auth.generic_oauth under spec.config in this branch. - mode: None (default) — no change to the rendered manifests. CustomConfig branch fails the render at the XOR helper if both or neither of `config` / `secretRef.name` are set. `disable_login_form` stays false so the admin_user/admin_password Secret remains a documented break-glass path in every mode. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
473b112f66 |
feat(monitoring): persist confidential OIDC client secret for System mode
New oidc-client-secret.yaml renders a Kubernetes Secret with the confidential client_secret used by both the EDP-Keycloak operator (via \$<secret>:client-secret from oidc-keycloak.yaml) and the Grafana pod (via GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET, wired by the next commit). Value is generated at first install (randAlphaNum 32) and preserved across upgrades through the same \`lookup\` pattern used by packages/system/dashboard. secret-hash annotation surfaces the value change to whichever controller is watching, ensuring KeycloakClient + Grafana Deployment re-reconcile if it ever rotates. Rendered only for mode: System — CustomConfig operators bring their own credentials (inline plaintext or a Secret they manage out-of-band). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
9dd9f1d4dd |
feat(monitoring): render Keycloak client + scope + groups for System mode
New oidc-keycloak.yaml template mirrors the tenant kube-apiserver Phase-1 shape (cozystack/cozystack#3044) with three Grafana-specific adjustments: - Client is confidential (public: false, directAccess: false); Grafana is server-side so a client_secret is required. The value is wired via the EDP-Keycloak `$<secret>:<key>` reference; the backing Secret is created by the next commit. - redirectUri is https://grafana.<host>/login/generic_oauth — the Grafana generic_oauth callback path. - Three KeycloakRealmGroups (`<clientId>-{admin,editor,viewer}`) are chart-owned. Membership is set by an operator through the Keycloak UI or a KeycloakRealmUser CR; grafana.yaml will map group membership to Admin / Editor / Viewer via role_attribute_path. Guarded by mode == System AND _cluster.oidc-enabled == true (via the assertSystemEnabled helper). Renders nothing for None or CustomConfig. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
e12bb1a3ba |
feat(monitoring): scaffold OIDC selector on the Grafana instance
Add the plumbing needed for a Phase-1 OIDC integration on Grafana, matching the shape of cozystack/cozystack#3044 for the tenant kube-apiserver: - values.yaml grows a top-level oidc block with a three-value mode enum (None/System/CustomConfig) and a customConfig payload with the XOR-inline/secretRef surface. - monitoring-rd advertises the new selector via its openAPISchema and keysOrder so operators see it on the Monitoring CR. - templates/_helpers.tpl introduces the naming helpers reused by every downstream OIDC template: clientId, audienceScopeName, clientSecretName, grafanaHost, redirectUri, systemIssuerURL, roleAttributePath, allowAssignGrafanaAdmin (platform-only), assertSystemEnabled, assertCustomConfigXor. No behaviour change on default values — mode: None short-circuits and no OIDC objects are rendered. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
scooby87
|
f91841fce6 |
fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3215)
## What this PR does Addresses #3181, where platform log ingestion could fail **silently**: fluent-bit shipped to `vlinsert-generic` while the monitoring `HelmRelease` reported `Ready`, and a `vlinsert-generic` `ExternalName` in `cozy-monitoring` dead-ended at a `tenant-root` VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary layers make the failure loud instead of silent: 1. **Readiness gate** — `packages/extra/monitoring/templates/helmrelease.yaml` now sets a `poller` `waitStrategy` and a `healthCheckExprs` entry on the `VLCluster` `status.updateStatus`. A missing/unhealthy log store makes the `monitoring-system` release **not-ready** instead of flipping `Ready` the moment the CR is applied. Mirrors the existing seaweedfs CNPG pattern. 2. **No empty-log-store hole** — `packages/system/monitoring/templates/vlogs/vlogs.yaml` fails the render on an empty `logsStorages`. With no entry no `VLCluster` exists, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is now a loud misconfiguration. 3. **No silent fallback** — the `cozy-monitoring → tenant-root` `vlinsert`/`vminsert` `ExternalName` redirects in `cozystack-basics` are created only when the root tenant hosts monitoring. The template reads the live `Tenant/root` `spec.monitoring` via `lookup` (source of truth, so it cannot drift from the tenant's real state) and falls back to a new platform value `monitoring.rootEnabled` (via `_cluster.monitoring-enabled`) when `lookup` is empty (dry-run / first render). The default preserves the historical always-create behaviour. Covered by helm-unittest across all four affected charts (141 tests): readiness-gate fields, empty-vs-populated `logsStorages` contract, `ExternalName` present/absent fallback contract, and the platform wiring into the `cozystack-values` Secret. Also adds the missing `test:` target to `packages/extra/monitoring/Makefile` so its suite runs in CI. **Known limitation / follow-up:** the legacy platform monitoring release installed directly into `cozy-monitoring` via the `cozystack.monitoring` PackageSource cannot carry the readiness gate, because the engine's `ComponentInstall`/HelmRelease builder and the vendored `helm-controller` API v1.4.3 lack `waitStrategy`/`healthCheckExprs`. That source is not referenced by any active bundle; full coverage there needs a helm-controller bump plus engine support. ### Release note ```release-note fix(monitoring): surface missing platform VictoriaLogs as not-ready and stop the dangling vlinsert-generic ExternalName from silently dropping logs (#3181) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a platform setting (`monitoring.rootEnabled`) to control whether root-level monitoring is enabled. * Monitoring redirect ExternalName services are now rendered only when root monitoring is enabled (defaulting to enabled). * **Bug Fixes** * Ensures an explicitly set `monitoring.rootEnabled=false` is preserved when generating platform configuration. * Prevents incomplete monitoring deployments by failing rendering when `logsStorages` is empty. * Improves VictoriaMetrics readiness by delaying “ready” until the operator reports an operational status. * **Tests** * Added/expanded Helm-unittest suites covering monitoring wiring, redirects, readiness gating, and `logsStorages` requirements. * Added a `make test` target to run chart unittests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Алексей Артамонов
|
f700c9043c |
fix(monitoring): backfill release label onto pre-existing VM/VL storage PVCs
Re-review on #3094 (IvanHunters, MAJOR): the volumeClaimTemplate label lands on PVCs only at creation — the StatefulSet controller never re-labels existing PVCs and vm-operator's PVC reconcile is resize-only. So on an already-deployed cluster (exactly the #3091 population) the pre-upgrade VM/VL storage PVCs stay unlabeled and the cleanup hook's selector misses them. Add migration 49 (mirrors the ClickHouse-keeper backfill in migration 48 for the sibling #3072). It relabels the pre-existing vmstorage/vmselect/vlstorage PVCs of every cozystack-managed VMCluster/VLCluster with apps.cozystack.io/application.name read from the CR's own spec.managedMetadata, so only monitoring's clusters are touched (a user's own VMCluster carries no such label) and the value always matches what the cleanup hook selects. Best-effort like #48: a skipped relabel is no worse than the pre-existing leak, so transient apiserver errors never abort the platform upgrade. Bump platform targetVersion 49 -> 50. Also correct the misleading comments (IvanHunters MINOR): spec.managedMetadata is never applied to PVCs by vm-operator at any current version, so the claim-template label is the only mechanism — do not imply a v0.71.0 crossover. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
e01b725a48 |
fix(monitoring): stamp release-scoping label on VM/VL storage PVCs so cleanup matches
Re-review on #3094 (IvanHunters, lexfrei): the release-scoped selector added in ba5797f4 was a silent no-op on the shipped vm-operator v0.68.4. The apps.cozystack.io/application.name label lived only under the VMCluster/VLCluster spec.managedMetadata.labels, and v0.68.4 does not propagate managedMetadata onto PVCs (that lands only in v0.71.0). So the delete matched nothing and the #3091 storage leak stayed open. Stamp the label where it actually reaches the PVCs: on spec.{vmselect,vmstorage,vlstorage}.storage.volumeClaimTemplate.metadata.labels of the VMCluster/VLCluster CRs (packages/system/monitoring). vm-operator copies claim-template labels onto the PVCs (IntoSTSVolume) even on v0.68.4, so the existing cleanup selector (managed-by=vm-operator,apps.cozystack.io/application.name= <release>-system) now matches and stays scoped to this release. Verified by rendering the CRs (label resolves to monitoring-system, matching the hook). Also add the explicit subjects[].namespace on the cleanup RoleBinding (lexfrei non-blocking) and a helm-unittest suite asserting the label lands on all three storage claim templates. Corrected the misleading test comment about how the label reaches the PVCs. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |