mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 18:47:04 +09:00
docs/system-memory-limits
88
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
d73c102945 |
test(system): anchor index-addressed negative asserts to what they name
notContains passes whenever its path does not resolve, and also
whenever it resolves to something that simply lacks the forbidden
content. A different container at the same index qualifies, so these
guards go green the moment a sidecar is placed ahead of the workload
they were written for, while the flag or env var they forbid comes
back on the real container.
Assert the identity at that index first, in each affected case: the
container name for the Deployment-shaped ones, and for the Cilium
policy its selector, since a deny rule has no name. The selector
identifies that rule only while there is one rule to identify, so it
is paired with a length assertion, and with a second one on the
rule's toPorts, because that path carries two indexes and anchoring
the outer one alone leaves the inner free.
The anchor goes only where nothing else holds the index. A positive
contains on the same path fails on its own once a sidecar arrives
ahead of the workload, and an equal on the whole argument list fails
the same way, so neither carries a separate anchor. A negative does,
including the per-element form below: a wildcard resolving to the
sidecar's own arguments matches nothing and passes.
Several of the guards could not fail for the reason they name, and
the anchor alone would have left that hidden behind a comment saying
they were covered. contains and notContains compare elements for
equality, so a mapping is compared whole: `{name: GF_...}` never
matched an entry carrying a value beside the name, and `{port: 7473}`
would not match that port re-added with a protocol beside it, which
is how the neighbouring entry in the same list is written. Those get
`any: true`.
Equality against a rendered argument is the same failure worn
differently, and it arrives in three disguises. A prefix that never
equals an element: `--pinned-tenants=` never equalled
`--pinned-tenants=tenant-a=shard0`, and `--target-host=` never
equalled a rendered `--target-host=<value>`. One value out of many
under a name that claims the flag: `--compatibility=cilium` left the
kilo default render free to carry `--compatibility=flannel` off the
same values key, and the two `--proxy-fqdn` negatives forbade only
the FQDN the default cluster-domain produces. And a form that holds
by luck: `--gateway-api` still fails today only because that template
renders it bare, so an upstream switch to `--gateway-api=true` would
retire the guard in silence.
All of those become notMatchRegex over args[*], which matches the
flag name per element and fails on any value and on the bare form
alike. A length assertion would not have covered the prefix cases,
since an upstream bump that adds the flag while consolidating another
argument keeps the count. A wildcard that resolves to nothing still
passes, so the index anchors stay.
The whole argument list is a different instrument, answering what the
default invocation is rather than whether one flag is absent. It
fails on an argument added, removed or swapped instead of only on the
ones someone thought to enumerate, which is worth having where the
template is first-party and a failure is a deliberate change asking
to be read. flux-shard-operator gets it in `renders the default
Deployment shape`, replacing eleven individual contains that listed
exactly the same eleven elements; `--shard-cpu-limit=` was dead for
the prefix reason and is dropped rather than repaired, because that
list now covers it. The vendored ouroboros templates get no such pin:
upstream churn arrives there on its own schedule, and a whole-list
assertion would fail a test named for one flag for a reason that has
nothing to do with that flag.
Anchor every index on a path, not the outermost one. The Cilium
assertion reads spec.ingressDeny[0].toPorts[0].ports, and pinning the
rule list alone would still let the port return on a second toPorts
entry of the same rule.
Each guard was checked by rendering what it forbids onto the element
it names, and each anchor by rendering the shift it exists to catch.
The form is already used in this tree, in opensearch-operator and
dashboard. Assertions of the same shape remain in files this commit
does not touch, including one in a package it does, and stay out of
scope here.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
|
||
|
Andrei Kvapil
|
705e39d3d7 |
feat(monitoring): install Grafana plugins from the in-cluster mirror
Point GF_INSTALL_PLUGINS at the grafana-dashboards static file server instead of the public Grafana catalog, so plugin installation needs no internet egress at pod startup and keeps working on air-gapped installs. Route the upstream Grafana image reference through cozy-lib.image so mirrored-registry installs keep working, and add regression tests for the image reference and the plugin install path. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
9c63df7545 |
build(monitoring): stop building and publishing the Grafana rebuild
Remove the custom Grafana image build now that the deployment runs on the upstream grafana/grafana image with catalog plugins. Drops the Dockerfile, the pinned image tag, the image targets in both monitoring Makefiles, and the build step from the root Makefile, so ghcr.io/cozystack/cozystack/grafana is no longer produced or published. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
44ff0fb286 |
feat(monitoring): run Grafana on the upstream image with catalog plugins
Point the Grafana deployment at the upstream grafana/grafana image and install the VictoriaLogs datasource and panel plugins at startup via GF_INSTALL_PLUGINS, pulled from the official Grafana plugin catalog (https://grafana.com/grafana/plugins) instead of being baked into a custom rebuilt image. The VictoriaLogs datasource is now published and signed in the catalog (victoriametrics-logs-datasource), so allow_loading_unsigned_plugins and the custom /var/lib/grafana-plugins path are no longer needed; plugins land in the writable default GF_PATHS_PLUGINS. Plugin versions are pinned to releases compatible with the shipped Grafana version. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
cozystack-ci[bot]
|
235d38dae1 |
Prepare release v1.6.0 (promoted from v1.6.0-rc.4)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
cozystack-ci[bot]
|
d95685c427 |
Prepare release v1.6.0-rc.4
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Ivan Okhotnikov
|
0164a6afa4 |
fix(monitoring): set groups_attribute_path=groups so allowed_groups actually works on Grafana v11.x
Verified end-to-end on dev3 with Grafana v11.6.15: OAuth flow reaches Grafana, id_token and userinfo response both carry the correct `groups` claim (`[cozystack-cluster-admin, tenant-root-super-admin]` for kvaps), `allowed_groups` in grafana.ini lists `tenant-root-super-admin`, and login still fails with `user not a member of one of the required groups`. Grafana debug log shows `Attributes: map[]` — the extracted user's Groups slice is empty. On v11.x+ Grafana does not auto-map the `groups` claim onto user.Groups unless `groups_attribute_path` is set explicitly (the field is documented as optional but the gate is a silent no-op without it — a claim-name-string is not enough). Fix: hard-wire `groups_attribute_path=groups` in the chart-forced System-mode block alongside `allowed_groups`. "groups" points at the top-level `groups` array from userinfo, matching the shape Keycloak's oidc-group-membership-mapper emits (verified on the same dev3 run). Two helm-unittest cases updated to assert `groups_attribute_path` lives next to `allowed_groups` in both the users-map-active and empty-users branches (rejecting the silent no-op regression). Docs also updated to name the requirement. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
b002c967ac |
fix(monitoring): close cross-tenant Grafana OIDC hole via tenant-group gate
Round-3 review from lllamnyp: any user in the shared `cozy` Keycloak
realm could sign into any tenant's Grafana. The audience mapper stops
replay of a minted token across releases, but does not gate who may
mint one — tenant-alice could complete the OAuth flow against
tenant-bob's Grafana. With `spec.oidc.users` non-empty the earlier
`allow_sign_up=false` fix mostly closed the door (though tenant-alice
still hit `error 401` after landing in Grafana, revealing the release
exists); with empty users under mode: System — a config the schema
permits and the docs describe as hands-off — the login succeeded and
minted a Viewer account for every cozy-realm identity.
The fix uses the KeycloakRealmGroups the tenant chart already
provisions (`<namespace>-{view,use,admin,super-admin}` in
packages/apps/tenant/templates/keycloakgroups.yaml) — no new
directory objects owned by the Monitoring release. Two chart edits:
1. oidc-keycloak.yaml: add the platform-managed realm-default `groups`
scope to `defaultClientScopes` explicitly. EDP's KeycloakClient
does NOT merge realm-level defaults into `defaultClientScopes`, so
the scope must be listed here or the `groups` claim would be
missing from the token.
2. grafana.yaml: in System mode, unconditionally emit
`allowed_groups: <ns>-view <ns>-use <ns>-admin <ns>-super-admin`
in `auth.generic_oauth` and add `groups` to `scopes`. Grafana
rejects any login whose token does not carry at least one of the
groups.
The gate is chart-owned and does not depend on the users-map — with
`users: []` tenant members still get a self-service Viewer through
`auto_assign_org_role`, cross-tenant users get 401 in Keycloak's OAuth
callback. Docs updated to reflect the two-layered contract
(tenant-membership gate + optional users-map).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
f64ba91872 |
fix(monitoring): do not render users-Job when spec.oidc.users is empty
Address PR#3176 review round-2 gap-2: the users-Job's prune pass deletes every non-admin Main-Org member whose email is not in the desired list. When the desired list is empty every helm-upgrade wiped the whole org. Three empty-users states used to fall into this trap: * customConfig.secretRef: users[] is forbidden by assertCustomSecretRefUsersEmpty, so the desired list was structurally always empty and the chart wiped an org whose auth.ini fragment the operator advertised as authoritative. * System with users[] unset: the operator opted into OIDC but not into the chart-managed users-map, yet the Job still pruned every Main-Org member on their behalf. * CustomConfig-inline with users[] unset: same shape as System. The mode=None guard (from an earlier round) already had the same shape; extend the render gate to cover the other empty-list doors — now the Job renders only when `mode != None AND users non-empty`. The trade-off: taking `users:` back to `[]` no longer prunes the last entry — matching the already-documented `System → None` posture — called out explicitly in the docs. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
ff4ef2c189 |
fix(monitoring): force allow_sign_up=false and gate isolation dict on users-map
Address PR#3176 review round-2 gap-1: the docs promise "users not in spec.oidc.users get nothing", but Grafana's default allow_sign_up=true combined with the chart-forced skip_org_role_sync=true was minting a Viewer account for every cozy-realm identity that completed the OAuth flow, then reaping it on the next helm-upgrade prune pass. The chart-forced isolation dict (now three keys: skip_org_role_sync, oauth_allow_insecure_email_lookup, allow_sign_up=false) is applied only when spec.oidc.users is non-empty. A BYO-IdP operator who does not use the users-map gets none of the three forced on top of their inline config, matching the review's minor: forcing insecure_email_lookup against an issuer that may not verify emails is a needless account- linking surface, and allow_sign_up=false with no pre-provisioned accounts would lock everyone out. Tests split the two branches: users-active asserts all three keys are present and the operator cannot override them via CustomConfig-inline; users-empty asserts the auth.generic_oauth block is emitted without any of the isolation keys. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
b2fd32385e |
fix(monitoring): rewrite users-Job shell in POSIX (alpine BusyBox compat)
CI E2E caught the Job container crash-looping within 20s of the first Start event. Root cause: the `alpine/k8s:1.33.4` image is Alpine-based (BusyBox coreutils + BusyBox grep), and the Job shell used four GNU-specific constructs that BusyBox does not implement: - `head --bytes=N` (BusyBox `head` only accepts `-c N`). - `tr --delete` (BusyBox `tr` only accepts `-d`). - `jq --raw-output0` (NUL-terminated output — jq 1.7+ feature; also paired with `read -r -d ''` which is a bash-ism, not POSIX `sh` and not honoured by BusyBox `ash`). - `grep --null-data --fixed-strings --line-regexp` (BusyBox `grep` supports `-F` and `-x` short forms but NOT `-z`/`--null-data`). Rewrite: - `head -c 24 /dev/urandom | base64 | tr -d '=+/' | head -c 32` — same random-32-char password, portable to BusyBox. - Emit desired emails and org membership one record per line (jq `--raw-output`) instead of NUL-terminated. Safe because the template-level `monitoring.oidc.assertUsersEmailShape` guard already rejects any email containing whitespace / control characters, so line-oriented iteration cannot get confused. - Read TSV membership records with `IFS="$(printf '\t')"` (POSIX- portable TAB) + plain `read -r uid ulogin uemail` — no need for `-d ''` once records are newline-terminated. - Match desired list with `grep -Fxq -e "$uemail" -e "$ulogin"` — short-form flags supported by both GNU and BusyBox `grep`. Syntax validated with both `sh -n` and `bash -n` on the extracted inline script. helm-unittest 34/34 still green (render shape is unchanged, only the shell literal is different). Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
1422495b88 |
fix(monitoring): do not render users-Job when mode=None (upgrade safety)
Caught by dev3 end-to-end test on the PR working tree
(cozystack-pr-test skill): the users-Job was rendered in every mode
including `None` on the rationale that a `mode: System → None`
switch should garbage-collect OIDC-provisioned local accounts. That
rationale ignored the upgrade path from the pre-PR chart: an
existing tenant that never had `spec.oidc` on their Monitoring CR
would receive `mode: None` on first render under the PR, the Job
would fire on every Flux reconcile of the outer HR, and its prune
pass would DELETE every non-admin Main-Org member — including users
the tenant added by hand through the Grafana UI on the pre-PR chart.
Silent breaking change with data loss.
Wrap the whole Job resource in `{{- if ne $mode "None" }} … {{- end }}`
so `mode: None` = "chart owns nothing in Grafana orgs, we don't
touch what's there". Operators who need to prune OIDC-provisioned
accounts after switching System | CustomConfig → None do it
themselves through the Grafana UI or admin API — same posture as
"we don't own the Keycloak directory objects" from the earlier
architectural change.
The `assertUsersPrologue` include stays outside the `if` so an
operator listing users in `mode: None` still gets the schema and
email-shape validation at render time — the Job body just doesn't
emit.
Test coverage:
- Flip the previous "mode=None still renders the users-Job so a
mode switch can prune stale members" case to
`mode=None does NOT render the users-Job (upgrade safety guard)`
as a regression lock.
- Add `mode=None with non-empty users still fails render` covering
the prologue-only-fires-on-schema-issue path.
Docs: rewrite the "Users and RBAC" section step 3 to name the
upgrade-safety motivation and the manual-cleanup handoff.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
c31a9f8ec3 |
fix(monitoring): promote oidc.users[].role to required + regenerate schema
The `OIDCUser` typedef in `packages/extra/monitoring/values.yaml`
declared only `@field {string} email`; `cozyvalues-gen` regenerated
`values.schema.json`, `README.md`, and the mirrored `openAPISchema`
on the `Monitoring` ApplicationDefinition WITHOUT a `role` field.
Consequences:
- The `@enum OIDCRole` declaration was orphan — referenced from
nowhere, so the schema's items object had `required: ["email"]`
and one property.
- A dashboard-driven CR would offer a form without any role picker;
a schema-conformant `{email}` submission would render into the
users-Job with `role: null`, `jq --raw-output` yields the string
`"null"`, the `POST /api/orgs/1/users` body carries
`"role":"null"`, Grafana returns HTTP 400, and the post-install
hook exhausts `backoffLimit: 6` — HelmRelease never becomes Ready.
- Unit + e2e tests all hardcoded `role: Admin` / `role: Viewer`, so
the contract mismatch never surfaced in CI.
Fix:
- Add `@field {OIDCRole} role - ...` to the `OIDCUser` typedef in
`packages/extra/monitoring/values.yaml` so cozyvalues-gen materialises
the field with enum + required.
- Regenerate `values.schema.json`, `README.md`, and mirrored
`openAPISchema` via `make generate`.
- Add `monitoring.oidc.assertUsersRoleShape` template-level guard
(same shape as `assertUsersEmailShape`) so a stale-schema CR or a
direct-helm invocation fails at render time with a legible error
instead of at hook execution with a Grafana 400.
- Consolidate all `spec.oidc.users` guards into a single
`monitoring.oidc.assertUsersPrologue` include and call it once from
`oidc-users-job.yaml`, ordering the incompat check
(`assertCustomSecretRefUsersEmpty`) before the per-entry role /
email regexes so operators see the more actionable message first.
Drop the duplicate include from `grafana.yaml` (Job template renders
in every mode anyway).
- Two new helm-unittest cases lock the regression:
`oidc.users entry missing role fails the render` and
`oidc.users entry with unknown role fails the render`.
- Point the previous `secretRef + users` incompat unit test at the
users-Job template since that's where the prologue now fires.
- `docs/oidc-grafana.md`: correct the failed-hook post-mortem window
claim — with `ttlSecondsAfterFinished: 3600` the Kubernetes TTL
controller reaps the Job + Pod 1h after the terminal condition,
not "until the next helm-upgrade".
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
ad67dee95b |
test(monitoring): lock the CustomConfig chart-force contract in helm-unittest
Two blockers on the previous review cycle were "code says X but no test enforces X" — so future edits to `grafana.yaml` could silently break the contract again. Add three assertions that fail loudly if the merge behavior regresses: - `mode=CustomConfig inline preserves the operator's map + chart-forced settings` — asserts `skip_org_role_sync=true` and `oauth_allow_insecure_email_lookup=true` are both present alongside the operator's `client_id` / `client_secret` / `auth_url`. - `mode=CustomConfig inline — operator cannot override chart-forced OIDC settings` — sets both settings to "false" in the operator's map; asserts they render as "true" (proves `merge` argument order is correct). - `mode=CustomConfig secretRef + non-empty users fails the render (contract incompat)` — asserts the new `monitoring.oidc.assertCustomSecretRefUsersEmpty` helper's error message fires so operators cannot silently strand their users-Job assignments by combining incompatible fields. - `mode=CustomConfig secretRef + empty users renders successfully` — the compatible path. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
26e5f06d27 |
fix(monitoring): chart-force skip_org_role_sync + email lookup in every OIDC mode
The chart previously injected the two settings the users-Job's
contract depends on — `skip_org_role_sync = true` and
`oauth_allow_insecure_email_lookup = true` — only inside the
`mode: System` branch of `grafana.yaml`. `values.yaml` and
`docs/oidc-grafana.md` both promised the settings would be forced
regardless of mode, but under `mode: CustomConfig` the operator's
inline `auth.generic_oauth` map was written verbatim, both
settings were absent, and the users-Job's role assignments were
silently stranded (no lookup by email → new Grafana account minted
at login) or transiently overwritten (no skip_org_role_sync → any
`role_attribute_path` or Grafana's `auto_assign_org_role: Viewer`
overwrote the Job's role on each login).
Fix: merge a `$chartForcedOidc` dict on top of both the generated
System block and the operator's inline map (`merge` wins for
overlapping keys, so the operator can NOT override the two
settings). CustomConfig secretRef mode cannot receive the merge
because the ini fragment is authoritative — the chart now hard-fails
render when both `customConfig.secretRef.name` and non-empty
`spec.oidc.users` are set, forcing the operator to either switch to
inline config or manage authorization themselves inside the ini.
Also:
- Align the inner HR `timeout` on `packages/extra/monitoring/templates/helmrelease.yaml`
from 10m to 15m so it matches the outer cozyrds
`release.cozystack.io/helm-install-timeout: "15m"` and does not
fail while the users-Job is still inside its
`activeDeadlineSeconds: 900`.
- Route the users-Job image through `cozy-lib.image` so
mirrored / air-gapped installations resolve it (matches the
pattern in `packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml`).
- Add `spec.ttlSecondsAfterFinished: 3600` on the users-Job so a
succeeded Job + Pod are garbage-collected on the next reconcile
interval rather than lingering until the next hook activation
(which never fires on release uninstall).
- Harden the users-Job shell:
* emails read NUL-terminated (`jq --raw-output0`) so an address
containing whitespace cannot corrupt the `for email in ...`
loop's field splitting;
* `/api/users/lookup?loginOrEmail=<email>` URL-encoded via
`jq @uri` so `&`/`=`/`?` in the local part parse correctly on
the Grafana side;
* prune loop switched from pipe-delimited fields to TSV inside
NUL-terminated records, with `cut -f` for extraction.
- Add a template-level assertion (`monitoring.oidc.assertUsersEmailShape`)
rejecting emails that do not match a conservative regex —
cheaper failure mode than a 400 from Grafana.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
c493f2cb96 |
test(monitoring): rewrite OIDC unit tests around the users-map contract
Rewrites the helm-unittest coverage for the Phase-1 OIDC selector
against the new chart contract:
- Drop assertions for the three KeycloakRealmGroups and for
`role_attribute_path` / `allow_assign_grafana_admin` / the
`oidc.grafanaAdmin` values lever — none of those render anymore.
- Assert the System-mode Grafana config carries
`skip_org_role_sync = "true"` and
`oauth_allow_insecure_email_lookup = "true"` and NO
`role_attribute_path`.
- Cover the new users-Job template: helm-hook annotations, absence
of a ServiceAccount, `activeDeadlineSeconds`, admin-credentials
wiring from the chart-managed `grafana-admin-password` Secret,
and correct serialisation of `spec.oidc.users` into the
`DESIRED_USERS_JSON` env.
- Assert the users-Job also renders under `mode: None` so a
System -> None flip still gets a prune pass.
- Cover the defensive `.Values.oidc | default dict` paths — omitted
block and explicit `oidc: ~` — across the three OIDC templates.
- Tighten the client-secret regex to the actual base64-encoded
length (`^[A-Za-z0-9+/]{43}=$`) so shortened random secrets are
caught.
32 tests, all green under helm-unittest v1.0.3.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
01d68904ea |
feat(monitoring): drop chart-owned Keycloak groups; app-side users-map + reconcile Job
Replace the group-based authorization model on the Phase-1 Grafana OIDC selector with an app-side users map, mirroring the tenant kube-apiserver sibling in cozystack/cozystack#3044. What is removed - The three chart-owned KeycloakRealmGroup objects in the cozy realm (`<clientId>-{admin,editor,viewer}`). A Helm release must not own directory objects — toggling `spec.oidc.mode: System -> None` or deleting the Monitoring release would Helm-delete the groups, and with them any membership curated out-of-band. Per-instance groups in the platform realm is exactly the design question left open at the end of cozystack/community#24; establishing it as precedent from a monitoring PR is out of scope. - Dash-separated group naming (`<namespace>-<release>-<role>`) was also collision-prone against tenant/subtenant paths, since both the namespace and release names can themselves contain dashes. - `role_attribute_path` in the Grafana `generic_oauth` block and the default-Viewer fallback, which handed org-Viewer to every cozy realm identity that logged in — cross-tenant metrics read across every opted-in Grafana. Also removed the `groups` client scope from the KeycloakClient defaults. - `monitoring.oidc.allowAssignGrafanaAdmin` helper and the internal `oidc.grafanaAdmin` values key: the feature was already deferred and the helper's release-name-only guard was tenant-spoofable. What replaces it - `spec.oidc.users: [{email, role: Admin|Editor|Viewer}]` on the Monitoring chart values. - A chart-owned post-install/post-upgrade Job (`templates/grafana/oidc-users-job.yaml`) that reconciles that list into Grafana's Main Org. via the admin API: pre-provisions a local Grafana account per listed email so the reconcile can set the org role before the operator's first OIDC login, adds the account to the org, PATCHes the role to converge across re-runs, and prunes every non-admin Main-Org member whose email is not in the list (handles both entry removal and `mode: None`). - `skip_org_role_sync = true` + `oauth_allow_insecure_email_lookup = true` in the generic_oauth block. The former stops a login from overwriting the Job's assignments; the latter binds the OIDC identity to the pre-provisioned local account. - Job talks to Grafana over the in-cluster Service and reads admin creds from the chart-managed `grafana-admin-password` Secret via `envFrom`: no ServiceAccount / Role / RoleBinding. `activeDeadlineSeconds: 900` caps a stuck Job so a broken Grafana cannot hold the release in `pending-upgrade`. Defensive rendering - `.Values.oidc | default dict` in the four templates that dereference the `oidc` block so `oidc: ~` or an omitted block no longer nil-derefs. - `oidc-client-secret.yaml` falls back to a fresh random when the existing Secret lookup succeeds but `data.client-secret` is empty. Rationale: cozystack/cozystack#3176 review from @lllamnyp (architectural CHANGES_REQUESTED points 1, 2, 3 + the default-Viewer leak). The three points stack: dropping the groups incidentally fixes the naming-collision and the default-Viewer leak, and the users-Job gives operators the same UX story as the tenant kube-apiserver PR across both sides of the platform. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
aa63786acd |
docs(monitoring): sync oidc.grafanaAdmin values docstring with deferred-lever posture
Second-iteration cozy-review flagged this as a leftover from the platform-lever removal: `packages/system/monitoring/values.yaml` still described `grafanaAdmin` as "Set true for the platform Grafana release only ... the platform bundle passes this flag through Package.spec.components", which contradicts the deferred-lever reality already documented in the same commit's updates to `_helpers.tpl` and `docs/oidc-grafana.md`. Rewrite the docstring to match: chart-internal lever, defaults `false`, Phase 1 has no end-to-end wiring; kept as a values key so a future phase can add the wiring without touching the chart contract. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
b46543e5f2 |
fix(monitoring): defer platform GrafanaAdmin lever to a future phase
The MAJOR finding from the cozy-review agent: the platform-side `oidc.grafanaAdmin: true` documented in the PR body and in docs/oidc-grafana.md never actually reached the platform Grafana's HelmRelease. `Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin` was set in packages/core/platform/templates/bundles/system.yaml, but `monitoring-system` (and `monitoring`) have no `install:` block in packages/core/platform/sources/monitoring-application.yaml — only `monitoring-rd` does. PackageReconciler skips components without an `install:` block before ever reaching the assignment that copies `pkg.Spec.Components[].Values` onto `hr.Spec.Values` (internal/operator/package_reconciler.go around lines 197-201 and 292-293). The values-override dictionary was silently dropped, no HR carried `oidc.grafanaAdmin: true`, and the platform Grafana rendered with the chart default (`false`). The `bundles_monitoring_grafana_admin_test.yaml` unit test only asserted against the emitted Package CR spec — exactly the layer where the value still existed but was about to be discarded — so the broken propagation slipped past. Trim the broken promise from Phase 1. The chart-side helper (`monitoring.oidc.allowAssignGrafanaAdmin`) stays as-is so a future phase that routes the override through the tenant-root chart's monitoring HR values (or adds an `install:` block on the `monitoring-system` component) lands on a green chart-side contract. The helm-unittest that verifies chart-side emission stays too, but with a docstring calling out that end-to-end wiring is deferred. Everything else that promised the lever works — bundle line, docs, platform bundle test — is either removed or rewritten to match reality. Also picks up two MINOR findings from the same review: * Drop the misleading `secret-hash` annotation on the OIDC client-secret Secret. The comment claimed it "rotates the KeycloakClient CR and therefore the Grafana Deployment env when the client-secret changes", but nothing downstream watches an annotation on the Secret itself — no rotation is triggered. The `lookup`-based fallback already keeps the value stable across reconciles, so no rotation is even needed. Removing the dead annotation avoids planting a broken assumption for future readers. * Add `release.cozystack.io/helm-install-timeout: "15m"` on the monitoring cozyrds ApplicationDefinition. Under `mode: System` the chart now emits, on top of the existing Grafana + Alerta + VMCluster + CNPG stack, four additional Keycloak CRs whose reconciliation is asynchronous against an operator in `cozy-keycloak` (not the tenant namespace). Flux's default 5m install window was already tight for the pre-OIDC stack; 15m matches the precedent set by tenant-rd/kubernetes-rd cozyrds. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
586e77c361 |
fix(monitoring): hard-fail System OIDC render on missing Keycloak CRD
`mode: System` previously wrapped every Keycloak resource in the `Capabilities.APIVersions.Has "v1.edp.epam.com/v1"` guard, so a cluster where the Keycloak operator CRDs were not yet registered silently dropped the whole Keycloak side while Grafana still came up with an `auth.generic_oauth` block pointing at a client that never got provisioned. Login broke with an opaque OIDC error and no chart-side signal that the platform was misconfigured. Turn the missing CRD into an actionable render error. A new `monitoring.oidc.assertSystemKeycloakCRD` helper fails with a message that points at `cozystack.keycloak-operator`, called from both `templates/grafana/oidc-keycloak.yaml` (replacing the silent capability guard) and `templates/grafana/grafana.yaml` (symmetric defense-in-depth so a future refactor of oidc-keycloak.yaml cannot regress the invariant). For the platform `monitoring-system` release the oidc variant's `dependsOn: cozystack.keycloak-operator` already prevents the race; for tenant releases the assertion converts the misconfiguration into an operator-facing error instead of a broken login flow. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
ffeb8e5e4f |
fix(monitoring): close self-review blockers for the OIDC PR
Address the two blocking-class findings from the cozy-review pass on
this PR:
(1) Missing PackageSource dependency on cozystack.keycloak-operator
(MAJOR cold-install race).
Without this edge, the monitoring chart can reconcile before the
v1.edp.epam.com CRDs are registered on a fresh cluster — the
`.Capabilities.APIVersions.Has` guard in
templates/grafana/oidc-keycloak.yaml then silently drops all
Keycloak-side objects for mode: System, and Grafana's
auth.generic_oauth block points at a client Keycloak never
provisions. Add the ordering edge, plus a
sources_monitoring_application_dependson_test.yaml regression
guard so a future edit cannot drop it silently.
(2) allow_assign_grafana_admin driven by `.Release.Name ==
monitoring-system` (MINOR fragile coupling).
Replace the release-name sniff with an explicit
`oidc.grafanaAdmin: false` values field, and flip it on for the
platform Grafana release via
Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin
in packages/core/platform/templates/bundles/system.yaml —
mirroring the pattern monitoring-agents already uses for
per-component overrides. Tenant Grafana releases inherit the
chart default false and stay at org-level Admin.
Two new helm-unittest cases in packages/system/monitoring/tests/
lock the values-driven behaviour on both sides
(grafanaAdmin=true → Admin uplift, default → tenant stays at org
Admin), plus bundles_monitoring_grafana_admin_test.yaml in
packages/core/platform/tests/ regression-guards the Package's
component override so the platform release keeps its uplift on
future edits.
Also document the mode-toggle group-membership loss in
docs/oidc-grafana.md — mode: System → CustomConfig/None deletes the
three chart-owned KeycloakRealmGroups; membership is not restored on
toggle-back. Not a bug, but the operator UX cost of the toggle
deserves a bullet in Failure modes.
Regenerated artifacts (cozyvalues-gen + hack/update-crd.sh):
packages/extra/monitoring/values.schema.json
packages/extra/monitoring/README.md
packages/system/monitoring-rd/cozyrds/monitoring.yaml
Test suites still green: packages/system/monitoring 23/23,
packages/core/platform 81/81.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
|
||
|
Ivan Okhotnikov
|
1f44bc06cf |
test(monitoring): helm-unittest coverage for the OIDC selector
15 cases across the three modes plus the two fail-close guards: - mode: None emits nothing OIDC-related — no cozy-realm objects, no client-secret Secret, no auth.generic_oauth key on the Grafana CR, no GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET / GF_PATHS_CUSTOM_INI env. - mode: System renders KeycloakClient + KeycloakClientScope + three KeycloakRealmGroups + persistent client-secret Secret, and pins the Grafana spec.config to the cozy realm with a role_attribute_path that maps the -admin/-editor/-viewer groups to Grafana roles. - mode: System fails the render when the platform-level OIDC feature is off, via the assertSystemEnabled helper. - mode: CustomConfig inline writes the operator-supplied map verbatim into spec.config.auth.generic_oauth without touching cozy. - mode: CustomConfig secretRef mounts the operator Secret under /etc/grafana/oidc + wires GF_PATHS_CUSTOM_INI, and does NOT emit auth.generic_oauth under spec.config (the ini fragment is authoritative). - CustomConfig branch fails-close on both/neither of the two payloads via the assertCustomConfigXor helper. - Platform release (Release.Name == monitoring-system) upgrades the Admin role to server-level GrafanaAdmin through the allow_assign_grafana_admin flag; tenant Grafana stays at org-level Admin. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
8eff03b323 |
feat(monitoring): wire auth.generic_oauth into the Grafana CR
Patch the Grafana CR so that the oidc.mode selector drives the actual Grafana runtime config: - mode: System — emit a full auth.generic_oauth section under spec.config with client_id / client_secret / auth_url / token_url / api_url / scopes / role_attribute_path / allow_assign_grafana_admin built from the helpers, and inject GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET as an env from the persistent Secret introduced in the previous commit. role_attribute_path maps membership in the per-instance -admin / -editor / -viewer groups to Grafana's Admin / Editor / Viewer roles, defaulting to Viewer on authenticated identities with none of them. allow_assign_grafana_admin is set to true only for the platform release (Release.Name == 'monitoring-system'); tenant Grafana stays at org-level Admin. - mode: CustomConfig (inline) — pass the operator-supplied config map verbatim to spec.config.auth.generic_oauth. `cozy` is not in the path. - mode: CustomConfig (secretRef) — mount the operator-managed Secret under /etc/grafana/oidc and load it via GF_PATHS_CUSTOM_INI. The ini fragment is authoritative; the chart does not emit auth.generic_oauth under spec.config in this branch. - mode: None (default) — no change to the rendered manifests. CustomConfig branch fails the render at the XOR helper if both or neither of `config` / `secretRef.name` are set. `disable_login_form` stays false so the admin_user/admin_password Secret remains a documented break-glass path in every mode. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
473b112f66 |
feat(monitoring): persist confidential OIDC client secret for System mode
New oidc-client-secret.yaml renders a Kubernetes Secret with the confidential client_secret used by both the EDP-Keycloak operator (via \$<secret>:client-secret from oidc-keycloak.yaml) and the Grafana pod (via GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET, wired by the next commit). Value is generated at first install (randAlphaNum 32) and preserved across upgrades through the same \`lookup\` pattern used by packages/system/dashboard. secret-hash annotation surfaces the value change to whichever controller is watching, ensuring KeycloakClient + Grafana Deployment re-reconcile if it ever rotates. Rendered only for mode: System — CustomConfig operators bring their own credentials (inline plaintext or a Secret they manage out-of-band). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
9dd9f1d4dd |
feat(monitoring): render Keycloak client + scope + groups for System mode
New oidc-keycloak.yaml template mirrors the tenant kube-apiserver Phase-1 shape (cozystack/cozystack#3044) with three Grafana-specific adjustments: - Client is confidential (public: false, directAccess: false); Grafana is server-side so a client_secret is required. The value is wired via the EDP-Keycloak `$<secret>:<key>` reference; the backing Secret is created by the next commit. - redirectUri is https://grafana.<host>/login/generic_oauth — the Grafana generic_oauth callback path. - Three KeycloakRealmGroups (`<clientId>-{admin,editor,viewer}`) are chart-owned. Membership is set by an operator through the Keycloak UI or a KeycloakRealmUser CR; grafana.yaml will map group membership to Admin / Editor / Viewer via role_attribute_path. Guarded by mode == System AND _cluster.oidc-enabled == true (via the assertSystemEnabled helper). Renders nothing for None or CustomConfig. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
e12bb1a3ba |
feat(monitoring): scaffold OIDC selector on the Grafana instance
Add the plumbing needed for a Phase-1 OIDC integration on Grafana, matching the shape of cozystack/cozystack#3044 for the tenant kube-apiserver: - values.yaml grows a top-level oidc block with a three-value mode enum (None/System/CustomConfig) and a customConfig payload with the XOR-inline/secretRef surface. - monitoring-rd advertises the new selector via its openAPISchema and keysOrder so operators see it on the Monitoring CR. - templates/_helpers.tpl introduces the naming helpers reused by every downstream OIDC template: clientId, audienceScopeName, clientSecretName, grafanaHost, redirectUri, systemIssuerURL, roleAttributePath, allowAssignGrafanaAdmin (platform-only), assertSystemEnabled, assertCustomConfigXor. No behaviour change on default values — mode: None short-circuits and no OIDC objects are rendered. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
renovate[bot]
|
5b9b420ffe | chore(deps): update container image & action digests | ||
|
scooby87
|
f91841fce6 |
fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3215)
## What this PR does Addresses #3181, where platform log ingestion could fail **silently**: fluent-bit shipped to `vlinsert-generic` while the monitoring `HelmRelease` reported `Ready`, and a `vlinsert-generic` `ExternalName` in `cozy-monitoring` dead-ended at a `tenant-root` VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary layers make the failure loud instead of silent: 1. **Readiness gate** — `packages/extra/monitoring/templates/helmrelease.yaml` now sets a `poller` `waitStrategy` and a `healthCheckExprs` entry on the `VLCluster` `status.updateStatus`. A missing/unhealthy log store makes the `monitoring-system` release **not-ready** instead of flipping `Ready` the moment the CR is applied. Mirrors the existing seaweedfs CNPG pattern. 2. **No empty-log-store hole** — `packages/system/monitoring/templates/vlogs/vlogs.yaml` fails the render on an empty `logsStorages`. With no entry no `VLCluster` exists, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is now a loud misconfiguration. 3. **No silent fallback** — the `cozy-monitoring → tenant-root` `vlinsert`/`vminsert` `ExternalName` redirects in `cozystack-basics` are created only when the root tenant hosts monitoring. The template reads the live `Tenant/root` `spec.monitoring` via `lookup` (source of truth, so it cannot drift from the tenant's real state) and falls back to a new platform value `monitoring.rootEnabled` (via `_cluster.monitoring-enabled`) when `lookup` is empty (dry-run / first render). The default preserves the historical always-create behaviour. Covered by helm-unittest across all four affected charts (141 tests): readiness-gate fields, empty-vs-populated `logsStorages` contract, `ExternalName` present/absent fallback contract, and the platform wiring into the `cozystack-values` Secret. Also adds the missing `test:` target to `packages/extra/monitoring/Makefile` so its suite runs in CI. **Known limitation / follow-up:** the legacy platform monitoring release installed directly into `cozy-monitoring` via the `cozystack.monitoring` PackageSource cannot carry the readiness gate, because the engine's `ComponentInstall`/HelmRelease builder and the vendored `helm-controller` API v1.4.3 lack `waitStrategy`/`healthCheckExprs`. That source is not referenced by any active bundle; full coverage there needs a helm-controller bump plus engine support. ### Release note ```release-note fix(monitoring): surface missing platform VictoriaLogs as not-ready and stop the dangling vlinsert-generic ExternalName from silently dropping logs (#3181) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a platform setting (`monitoring.rootEnabled`) to control whether root-level monitoring is enabled. * Monitoring redirect ExternalName services are now rendered only when root monitoring is enabled (defaulting to enabled). * **Bug Fixes** * Ensures an explicitly set `monitoring.rootEnabled=false` is preserved when generating platform configuration. * Prevents incomplete monitoring deployments by failing rendering when `logsStorages` is empty. * Improves VictoriaMetrics readiness by delaying “ready” until the operator reports an operational status. * **Tests** * Added/expanded Helm-unittest suites covering monitoring wiring, redirects, readiness gating, and `logsStorages` requirements. * Added a `make test` target to run chart unittests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Алексей Артамонов
|
f700c9043c |
fix(monitoring): backfill release label onto pre-existing VM/VL storage PVCs
Re-review on #3094 (IvanHunters, MAJOR): the volumeClaimTemplate label lands on PVCs only at creation — the StatefulSet controller never re-labels existing PVCs and vm-operator's PVC reconcile is resize-only. So on an already-deployed cluster (exactly the #3091 population) the pre-upgrade VM/VL storage PVCs stay unlabeled and the cleanup hook's selector misses them. Add migration 49 (mirrors the ClickHouse-keeper backfill in migration 48 for the sibling #3072). It relabels the pre-existing vmstorage/vmselect/vlstorage PVCs of every cozystack-managed VMCluster/VLCluster with apps.cozystack.io/application.name read from the CR's own spec.managedMetadata, so only monitoring's clusters are touched (a user's own VMCluster carries no such label) and the value always matches what the cleanup hook selects. Best-effort like #48: a skipped relabel is no worse than the pre-existing leak, so transient apiserver errors never abort the platform upgrade. Bump platform targetVersion 49 -> 50. Also correct the misleading comments (IvanHunters MINOR): spec.managedMetadata is never applied to PVCs by vm-operator at any current version, so the claim-template label is the only mechanism — do not imply a v0.71.0 crossover. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
e01b725a48 |
fix(monitoring): stamp release-scoping label on VM/VL storage PVCs so cleanup matches
Re-review on #3094 (IvanHunters, lexfrei): the release-scoped selector added in ba5797f4 was a silent no-op on the shipped vm-operator v0.68.4. The apps.cozystack.io/application.name label lived only under the VMCluster/VLCluster spec.managedMetadata.labels, and v0.68.4 does not propagate managedMetadata onto PVCs (that lands only in v0.71.0). So the delete matched nothing and the #3091 storage leak stayed open. Stamp the label where it actually reaches the PVCs: on spec.{vmselect,vmstorage,vlstorage}.storage.volumeClaimTemplate.metadata.labels of the VMCluster/VLCluster CRs (packages/system/monitoring). vm-operator copies claim-template labels onto the PVCs (IntoSTSVolume) even on v0.68.4, so the existing cleanup selector (managed-by=vm-operator,apps.cozystack.io/application.name= <release>-system) now matches and stays scoped to this release. Verified by rendering the CRs (label resolves to monitoring-system, matching the hook). Also add the explicit subjects[].namespace on the cleanup RoleBinding (lexfrei non-blocking) and a helm-unittest suite asserting the label lands on all three storage claim templates. Corrected the misleading test comment about how the label reaches the PVCs. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Alexey Artamonov
|
f9afc73096 |
fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3181)
Platform log ingestion could fail silently: fluent-bit shipped to vlinsert-generic while the monitoring HelmRelease reported Ready, and a vlinsert-generic ExternalName in cozy-monitoring dead-ended at a tenant-root VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary changes: - Readiness (do not stay silently Ready): the monitoring-system HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on its VLCluster (status.updateStatus), so a missing/unhealthy log store makes the release not-ready instead of flipping Ready as soon as the CR is applied. Mirrors the seaweedfs CNPG pattern. - No empty-log-store hole: system/monitoring rejects an empty logsStorages at render time (helm fail). With no entry no VLCluster is created, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is made a loud misconfiguration rather than a documented footgun. - No silent fallback: the cozy-monitoring -> tenant-root vlinsert/vminsert ExternalName redirects in cozystack-basics are now created only when the root tenant hosts monitoring, gated on _cluster.monitoring-enabled — fed from a new platform value monitoring.rootEnabled (default true, preserving the historical always-create behaviour). monitoring.rootEnabled is the single render-time source of truth: changing it re-renders cozystack-basics (helm-controller watches the cozystack-values Secret it flows into). A live `lookup` of Tenant/root spec.monitoring is deliberately NOT used: cozystack-basics is an operator-generated HelmRelease with drift detection off, so it re-renders only on install/upgrade/values-change — never on a reconcile tick or when a lookup result changes — and a lookup would go stale in both directions. Operators must keep monitoring.rootEnabled in sync with Tenant/root spec.monitoring. Covered by helm-unittest across all four charts: readiness gate fields, empty-vs-populated logsStorages contract, ExternalName present/absent and null-guard contract, and the platform wiring into the cozystack-values Secret. The CEL health predicates are exercised live, not in helm-unittest (which cannot evaluate CEL). Adds the missing `test:` target to packages/extra/monitoring/Makefile so its suite runs in CI. Known limitation (follow-up): the legacy cozystack.monitoring PackageSource that installs system/monitoring directly into cozy-monitoring cannot carry the readiness gate, because the engine's ComponentInstall/HelmRelease builder and the vendored helm-controller API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not referenced by any active bundle; full coverage needs a helm-controller bump plus engine support. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
37dbc586dc |
chore(monitoring): bump Grafana to 11.6.15, victorialogs datasource and alerta
Refreshes the monitoring image anchors onto current upstream to clear the published advisories carried by the stale pins. - grafana image: base grafana/grafana 11.4.0 -> 11.6.15 (latest 11.x). The rebuild clears the Go-dependency advisories in the grafana binary (x/crypto, moby/spdystream, jackc/pgx/v5, russellhaering/goxmldsig, buger/jsonparser, golang-jwt/jwt/v4, getkin/kin-openapi). It stays on the 11.x line rather than 12.x to keep the bundled panel plugins compatible (grafana-worldmap-panel is deprecated upstream) and avoid a dashboard-schema migration. - victorialogs-datasource plugin: v0.14.1 -> v0.28.0 (grafanaDependency >=10.4.0, so it loads on 11.6.15). - alerta-web: 9.0.4 -> 9.1.0, clearing the python cryptography advisory. - images/grafana.tag: re-pinned to the rebuilt multi-arch digest. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
cozystack-ci[bot]
|
e1aaa90ed3 |
Prepare release v1.5.0
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
myasnikovdaniil
|
aeaedca4a4 |
ci: consolidated platform & e2e stabilization batch (#2948)
## What Consolidated landing branch for a set of interdependent platform/CI stabilization fixes. It began as the source PRs in the table below (in dependency order) and has since grown with review-driven fixes and a few production-behavior changes — the full current inventory is under **Scope** below. Landing them as one unit lets CI run against the real combined state instead of a fragile bottom-up merge train. ## Source PRs (dependency order) | # | Commit | Source PR | Addresses | |---|--------|-----------|-----------| | 1 | fix(capi): startupProbe on capi-controller-manager | #2946 | capi crashloop during cert provisioning | | 2 | test(e2e): bucket.bats port-forward + S3 client reliability | #2944 | flaky bucket test | | 3 | fix(seaweedfs): restore -lock BucketClass, s3 svc name, drop volumeSizeLimitMB | #2943 | missing bucket/access classes | | 4 | test(e2e): in-cluster Cilium endpoint-leak healer (install + apps) | #2874 | cilium "IP already in use" leak | | 5 | refactor(build): mode=max registry cache + main-only warmer | #2938 | build cache | | 6 | ci(build): isolate each PR build on its own ephemeral runner VM | #2939 | shared-buildkit contention | | 7 | fix(e2e): LINSTOR post-install waits on a single 15m deadline | #2928 | LINSTOR provisioning timeouts | | 8 | test(metallb): assert digest-pinned image form, not version literal | #2873 | brittle metallb assertion | ## Why consolidated These form a dependency DAG (verified from CI logs): e.g. 2943 needs 2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested bottom-up, each lower PR runs with none of the fixes above it and can't go green alone. This branch carries all of them, so CI runs against the real combined state. ## Verification status CI is green on the latest head (`5653c30`): full-suite E2E passes end-to-end and is reproducible (2 of 3 runs on this SHA green). The one red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the sandbox, unrelated to the diff — a different environmental subsystem failed each run (details in the comments below). The earlier 3-hour crust-gather snapshot hang is fixed in `5653c30`. ## Scope beyond the original 8 The branch has grown past the 8 source PRs above with review-driven fixes and a few production-behavior changes. For reviewer transparency, the full set: **Review fixes (@lexfrei review):** - **B1 (blocker)** — `8f41910` converts the SeaweedFS `-lock`/`-readonly` BucketClasses and the s3 service-name override into `patches/`, wired into `make update`, so a re-vendor no longer drops them. - **FU1** — `cc99230` grants the cilium leak-healer `delete` on pods (both delete remedies were RBAC-forbidden). - **FU3** — `014de3f` adds a unit test for the HelmRelease update conflict-retry path. - **FU4** — `6deddc6` makes the leak-healer refuse a disconnect for any non-terminal owner phase, not just `Running`. - **FU7** — `f3914d2` sets `persist-credentials: false` on the build-cache checkout (SHA-pinning is handled repo-wide in #2849). **Production-behavior changes (not in the table above):** - VPA `updateMode: Auto → Initial` for etcd (`6e9ff90`) and monitoring (`0997105`), plus `vmselect`/`vmstorage` `minAllowed` floors — stops install-time eviction churn. - cozystack-api: `retry.RetryOnConflict` on the Application→HelmRelease update path (`728f676`). **E2E robustness follow-ups:** per-test crust-gather snapshots (`50e5f94`), EtcdBackupSchedule wait (`763b85e`), tenant-node single-deadline wait (`9e43d45`), harbor BucketClaim 10m budget (`d7f244f`), tenant API via LoadBalancer (`9beda4e`), crust-gather pin (`95f42aa`). **Not addressed (by decision):** FU2 — the s3 Service keeps `seaweedfs.name`-based naming via patch (rendered output unchanged; cozystack renders `fullname == seaweedfs`, so it matches the `componentName`/`fullname` siblings in practice). FU6 — fork PR build push is handled by the existing mirror-to-same-repo-branch workflow. ## Relates to #2946 #2944 #2943 #2874 #2938 #2939 #2928 #2873 — if this lands, those can be closed; otherwise they remain the granular per-PR review path. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added CI build cache warming for main branch builds. * Added a best-effort in-cluster Cilium endpoint self-heal watchdog for e2e installs. * Extended SeaweedFS COSI with object-lock support plus separate readonly access. * **Improvements** * Reduced e2e install flakiness with a shared readiness deadline, safer waits, and improved cleanup/diagnostics capture. * Prevented install-time churn by switching VPA update mode to **Initial**. * Added container startup probes and updated SeaweedFS volume sizing. * **Tests** * Improved e2e robustness (etcd backup schedule waiting, digest-pinned image checks, and corrected S3 port-forwarding). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
0997105b05 |
fix(monitoring): set VPA updateMode to Initial to stop install-time eviction churn
In Auto mode the VPA Updater evicts running pods to apply new recommendations. During install that eviction churn recreates VictoriaMetrics pods faster than Cilium releases their endpoint IPs, tripping the cluster-pool IPAM "IP already in use" (putEndpointIdInvalid) leak that wedges a pod and stalls the install. Initial sets requests at pod creation without ever evicting, removing the trigger (the cilium-leak healer remains as defense for non-VPA churn paths). Also raise vmstorage/vmselect minAllowed floors to 200m/512Mi: under Initial a pod is sized once at creation and never re-sized at runtime, so the floor is the safety net against the heavy query/storage tiers being left under-provisioned. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
d3958c5bcb |
refactor(build): mode=max registry cache with a main-only warmer (#2938)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
9e3ae7447b |
fix(ingress): keep tls section in operator wildcard mode
Dropping the whole tls block in wildcard mode disabled the forced HTTP-to-HTTPS redirect: ingress-nginx only issues the 308 redirect when TLS is enabled for the Ingress, while a host without a tls section is still served on 443 with the default certificate but stays reachable over plain HTTP. Keep the tls section with the host list and omit only the per-host secretName, so ingress-nginx falls back to the operator wildcard configured as the default SSL certificate and the redirect stays enforced, matching the non-wildcard behaviour. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
b1f5d9579c |
feat(ingress): serve operator wildcard via default SSL certificate
When publishing.certificates.wildcardSecretName is set, the root tenant's ingress-nginx controller now serves the operator-supplied wildcard Secret as its --default-ssl-certificate, and the platform service Ingresses stop requesting their own per-host ACME certificates. The flag is set only on the publishing controller (Release.Namespace == expose-ingress) because ingress-nginx loads the Secret from its own namespace; child-tenant controllers cannot reach a Secret in another namespace, so they keep the existing ACME path. Each service Ingress (dashboard, keycloak, grafana, alerta, bucket, linstor-gui, harbor, seaweedfs S3, bootbox) drops its cert-manager cluster-issuer annotation, the http01 challenge-class annotation, and its tls block in wildcard mode; nginx then falls back to the default certificate for those hosts. Other annotations and routing are unchanged. Each file keeps its existing render guard intact, and the annotations block collapses cleanly where the cert annotations were its only content. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
5982018629 |
feat(monitoring): add tenant overview dashboard (#2809)
## What Adds a cross-tenant **Tenant Overview** dashboard to the root/infra Grafana. ## Why Platform admins currently have no single view of which tenants consume what. This shows resource utilization, app counts, and health per tenant at a glance. ## Panels - **Fleet summary** — tenants, apps, running pods, CPU/memory/storage totals - **Tenant leaderboard** — per-tenant table: CPU & memory (usage vs requests), storage, pods, PVCs, app count, not-ready apps - **Top consumers** — Top-N bar gauges by CPU, memory, storage, app count - **Trends** — per-tenant CPU & memory usage over time - **Health** — pod restart rate and not-ready HelmReleases per tenant ## Details - Registered in `dashboards-infra.list` only, so it deploys to the root/`cozy-monitoring` Grafana and never exposes cross-tenant data on per-tenant instances (same pattern as `gpu/gpu-tenants`). - Apps are counted from `gotk_resource_info` HelmReleases, excluding the paired `*-system`/`cozy-*` charts. - Uses only standard cadvisor / kube-state-metrics / Flux metrics. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## New Features * Added a Tenant Overview dashboard for comprehensive tenant infrastructure monitoring, including: * Fleet summary with tenant counts, app deployments, and aggregate resource usage * Tenant performance leaderboard with CPU, memory, and storage metrics * Top consumers visualization by resource type and app count * Historical usage trends and pod health metrics <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
9868c43e53 |
fix(build): read --cache-from from ghcr :latest so PR builds warm-start
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold. Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
b759446fa6 |
feat(monitoring): add tenant overview dashboard
Add a cross-tenant overview dashboard for the root/infra Grafana that surfaces which tenants consume what resources, how many apps (Flux HelmReleases) each runs, and per-tenant health. Panels: - Fleet summary stats (tenants, apps, pods, CPU/mem/storage) - Tenant leaderboard table (CPU/mem usage vs requests, storage, pods, PVCs, app count, not-ready apps) - Top-N bar gauges by CPU, memory, storage and app count - CPU/memory usage trends per tenant - Pod restart rate and not-ready HelmReleases per tenant Registered in dashboards-infra.list so it deploys only to the root/cozy-monitoring Grafana, keeping cross-tenant data off per-tenant instances. Uses standard cadvisor, kube-state-metrics and gotk_resource_info metrics. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
3756f5c459 |
feat(networking): Gateway API support via Cilium (supersedes #2213) (#2470)
## What this PR does
Replaces ingress-nginx as the default ingress layer for cozystack-native
services with **Cilium Gateway API**, materialised per-tenant via a new
`TenantGateway` CRD reconciled by `cozystack-controller`. Supersedes
#2213 / #2208. Stacked on top of #2468.
## Architecture
The chart renders one `gateway.cozystack.io/v1alpha1 TenantGateway` CR
per tenant that owns its own Gateway. The cozystack-controller
reconciles the actual `Gateway`, per-tenant `Issuer`, `Certificate`
(wildcard or per-listener), and HTTP→HTTPS redirect `HTTPRoute` from
there. Helm does not render `Gateway` or `Certificate` directly — that
prevents the Helm-vs-controller race on `Gateway.spec.listeners` that
route-driven listener additions would otherwise cause.
```
┌── TenantGateway CR (rendered by extra/gateway chart for owning tenants)
│
└──▶ cozystack-controller reconciles:
├─ Gateway (with dynamic listeners, label-based allowedRoutes)
├─ Issuer (per-tenant ACME account)
├─ Certificate(s) — wildcard + per-child SANs in DNS-01, per-listener in HTTP-01
├─ HTTPRoute (controller-owned http→https redirect)
├─ Namespace label patching (cozy-* system namespaces from spec.AttachedNamespaces)
└─ Watches HTTPRoute / TLSRoute attachments to materialise listeners
```
## Inheritance: tenants attach by default, opt into their own Gateway
A tenant gets a dedicated Gateway (own LB Service, own LB IP, own
Certificate) only when it explicitly asks via
`tenant.spec.gateway=true`. Every other tenant in the tree publishes
through the Gateway of the nearest ancestor that owns one — same shape
as the existing `_namespace.ingress` inheritance.
The `apps/tenant` chart writes a `namespace.cozystack.io/gateway` label
on each tenant namespace, carrying either the tenant's own name (when
owning a Gateway) or the inherited ancestor name (when inheriting). The
Gateway's listener `allowedRoutes.namespaces.selector` is keyed on that
label, so the same selector admits routes from every namespace pointing
at the owner — descendants and cozy-* system namespaces alike.
cozystack-controller patches the same label onto every namespace in
`tgw.Spec.AttachedNamespaces` and garbage-collects labels it wrote when
an entry is removed from the attach list (annotation-gated GC so labels
written by the apps/tenant chart are never stripped).
In DNS-01 mode, the controller extends the parent's wildcard Certificate
with `<child-apex>` + `*.<child-apex>` SANs per inheriting child, and
adds a `*.<child-apex>` HTTPS listener per child apex referencing the
same cert. Without this expansion the parent's single-level wildcard
cannot match a child route's hostname (`harbor.alice.example.org` is two
labels past the parent's `*.example.org`).
Opt-out into own Gateway is the right choice when: a tenant needs its
own LB IP (DNS pinned, firewall rule), the apex is not derived from the
parent (custom `host` value the ancestor's cert can't cover), or the
tenant wants its own ACME account / cert authority. Otherwise leave
`gateway` unset and inherit.
## End-to-end flow
1. Platform: `gateway.enabled=true` toggles cert-manager solver,
switches per-system-app templates from Ingress to HTTPRoute / TLSRoute.
2. Per tenant: explicit `tenant.spec.gateway=true` renders a
`TenantGateway` CR and triggers the controller to materialise the owner
Gateway. Tenants with the field unset inherit through
`_namespace.gateway` propagation in the tenant chart.
3. Adding a published app under any tenant (owner or inheriting) is just
deploying its HTTPRoute. The controller picks up the hostname, adds a
per-listener Certificate (HTTP-01) or extends the wildcard cert's SANs
(DNS-01), and updates the owner Gateway's listeners.
Defaults stay legacy: `gateway.enabled=false`,
`tenant.spec.gateway=false`. Existing clusters upgrade unchanged.
## External IP allocation — mechanism-agnostic
The per-tenant Gateway's auto-created `LoadBalancer` Service draws its
IP from whatever LB allocator the cluster admin has configured at the
platform layer — same shape as ingress-nginx today. The tenant API stays
mechanism-agnostic — no `gatewayIP` field, no allocator-specific
manifest in the tenant chart. Cozystack itself ships MetalLB installed
but does not render any `IPAddressPool` / `L2Advertisement` /
`BGPAdvertisement` from this chart; admins set up the allocator that
suits their environment (MetalLB pool with L2 / BGP, Cilium LB-IPAM with
announcer, robotlb against a cloud provider, or
`Service.spec.externalIPs` pinning).
If a tenant needs a specific address (DNS already pinned, firewall rule,
etc.), the operator pre-allocates it on the admin side: either
pre-create the Service with `loadBalancerIP` set, or hand the tenant a
reference to a named admin-managed pool. Per-Service IP uniqueness is
the allocator's responsibility — same as for any other LoadBalancer
Service.
## Cert mode: HTTP-01 (default) vs DNS-01 (opt-in)
`publishing.certificates.solver` selects how the controller sources TLS
certs.
**HTTP-01 (default)** — per-app HTTPS listener + per-app Certificate.
New apps require zero platform-side config: deploy the HTTPRoute,
controller does the rest. Inheriting children's hostnames pick up
per-listener certs the same way the owner's own apps do.
**DNS-01 (opt-in)** — single wildcard Certificate (`<apex>` +
`*.<apex>`) covering every published app under the apex, plus per-child
SANs (`<child-apex>` + `*.<child-apex>`) for every inheriting tenant.
Pick this for clusters where many apps share the apex and Let's Encrypt
rate limits matter. The DNS provider account must be able to write TXT
records under every apex level the parent serves — for deeply-nested
inherited children that requires zone delegation or a wide-scope
provider credential.
| `publishing.certificates.dns01.provider` | Required
`publishing.certificates.dns01.<provider>` keys |
| --- | --- |
| `cloudflare` (default) | `cloudflare.secretName`,
`cloudflare.secretKey` |
| `route53` | `route53.region`, `route53.secretName` (and `accessKeyID`
if not on IRSA) |
| `digitalocean` | `digitalocean.secretName` |
| `rfc2136` | `rfc2136.nameserver`, `rfc2136.tsigKeyName`,
`rfc2136.secretName` |
The platform chart wires every provider's keys into `_cluster.dns01-*`;
the per-tenant gateway chart and the cluster-wide `cluster-issuers.yaml`
both read them.
## Strengthens admission coverage on apps.cozystack.io/*
The `cozystack-tenant-host-policy` probe in `gateway.bats` initially
looked like a flake — the apply that was supposed to be VAP-rejected
just succeeded. Investigation traced it to the custom REST handler at
`pkg/registry/apps/application/rest.go`. genericapiserver hands every
storage method a callback into the validating admission chain:
`createValidation` for Create, `updateValidation` for Update,
`deleteValidation` for Delete. `Update` was correctly invoking its
callback since
|
||
|
Aleksei Sviridkin
|
0994821bda |
feat(monitoring): toggle ingress↔HTTPRoute on _cluster.gateway-enabled
Adds Gateway API equivalents for the two ingress endpoints in the
monitoring chart — Grafana (web UI) and Alerta (alert console) — gated
on the same _cluster.gateway-enabled flag the keycloak / dashboard /
cozystack-api templates already use.
* Grafana's ingress is configured inline via Grafana CRD's spec.ingress
block; that block is now wrapped with {{ if ne gateway-enabled true }}
and a standalone HTTPRoute attaching to the tenant Gateway is rendered
in the alternate branch.
* Alerta uses a standalone Ingress object; the same Ingress↔HTTPRoute
swap is wired through an if/else.
Both routes attach to Gateway cozystack in the publishing namespace
(default tenant-root), matching the convention established by the rest
of the system charts.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Myasnikov Daniil
|
ed0297eb87 |
refactor(build): standardize image tagging to fix concurrent PR push conflicts
Every package Makefile tagged images via `settag` which, on a non-release commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every image's `:latest` manifest in the OCIR registry. Two outliers (metallb, redis-operator) used hardcoded version tags and collided across PRs even when `settag` would not have. Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro driven by three env vars: - IMAGE_TAG — build-unique tag, always pushed - PUBLISH_VERSIONED — also push :<component-version> - PUBLISH_FLOATING — also push :latest PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_* on, preserving the previous release output and adding a uniform :<git-tag> tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0 already off. Image consumers reference digests pinned into values.yaml by each Makefile, so the tag is decorative for cluster-side pulls; the build-unique tag is sufficient for E2E. Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor, matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing in the repo or downstream values.yaml consumed it. The same information is preserved via the new uniform `:<git-tag>` plus the existing `:<component-version>` on release. ubuntu-container-disk builds a separate image per Kubernetes version in a loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop iterations within a single build do not push to the same tag. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
3977aef509 |
feat(monitoring): add GPU observability dashboards and recording rules (#2418)
## What this PR does Adds GPU observability to Cozystack: five Grafana dashboards, a set of VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests for wiring the NVIDIA GPU Operator and DCGM Exporter on Talos when running GPU workloads directly in pods. ### Components 1. **Dashboards** — `dashboards/gpu/*.json`, registered in `packages/system/monitoring/dashboards-infra.list`. All follow the project convention: the Prometheus data source is selected through the `$ds_prometheus` template variable. - `gpu-performance` — per-GPU utilization (NVML, tensor pipe, graphics engine, memory copy), VRAM, power, temperature, and health (XID errors, power/thermal throttling). All metrics are shown per-GPU, filterable by the `$Hostname` selector. - `gpu-fleet` — cluster-wide inventory, capacity, utilization, and per-node power draw as percentage of TDP. - `gpu-tenants` — per-namespace allocation and 24-hour GPU-hour/kWh accounting; utilization and power are shown per-node (DCGM cannot attribute hardware metrics to namespaces). - `gpu-quotas` — allocated vs. requested GPUs, pending pods, and a per-namespace requests/limits table. All counts derive from the `namespace:gpu_count:allocated` recording rule, which excludes `Failed`/`Succeeded` pod carryover. - `gpu-efficiency` — cluster and per-GPU efficiency: tensor saturation, util-per-watt, and per-GPU power/thermal throttle fractions. 2. **Recording rules and alert** — `packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`. Four rule groups: - `gpu.recording.cluster.1m` — cluster-wide totals consumed by overview panels. - `gpu.recording.node.1m` — per-node hardware aggregates (grouped by `Hostname`) plus per-namespace allocation counts from kube resource requests; filters `cozy-*`/`kube-*`. - `gpu.recording.efficiency.1m` — per-GPU tensor saturation and util-per-watt, plus per-GPU power/thermal throttle fractions bounded via `clamp_max(..., 1)`. - `gpu.recording.throttle.validation.5m` — houses the `GPUThrottleFractionOverOne` warning alert that fires if the pre-clamp throttle fraction exceeds 1.0. This guards against DCGM counter-unit drift between GPU families; the `/1e9` divisor is verified on NVIDIA A10 with DCGM 3.x. Rules are safe on clusters without DCGM — they evaluate to empty series when no matching metrics are scraped. 3. **Reference manifests** under `packages/system/gpu-operator/examples/`. These are **not** templates; they document one working configuration: - `values-native-talos.yaml` — Cozystack Package values for the native-pod scenario. - `dcgm-custom-metrics.yaml` — ConfigMap extending the DCGM CSV with profiling, ECC, throttling, and energy counters used by the dashboards and recording rules. - `nvidia-driver-compat.yaml` — DaemonSet that stages `libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc tree into a location the GPU Operator validator inspects. Workaround for [NVIDIA/gpu-operator#1687](https://github.com/NVIDIA/gpu-operator/issues/1687). Requires `pod-security.kubernetes.io/enforce: privileged` on the target namespace when the cluster enforces baseline/restricted PodSecurityStandards. - `README.md` — explains the two deployment paths (sandbox vs. native), which DCGM metrics each dashboard depends on, and how the pieces fit together. 4. **Cross-validation test** — `hack/check-gpu-recording-rules.bats` enforces that every recording-rule reference in tracked dashboards resolves, and that every DCGM metric referenced by the rules is declared in either the upstream default CSV snapshot (`hack/dcgm-default-counters.csv`) or the custom CSV shipped under `examples/`. ### Why reference manifests, not templates The out-of-the-box `packages/system/gpu-operator/values-talos.yaml` targets **sandbox workloads** — GPUs passed through to KubeVirt VMs via VFIO, driver disabled on the host, device plugin disabled. That is the sensible default for Cozystack today. Running GPU workloads directly in pods on Talos is also possible but requires a different configuration that makes assumptions not every user shares: - the NVIDIA Talos system extension must be installed on GPU nodes, - the operator's own driver and toolkit components must be disabled because the extension provides them, - the validator's hardcoded paths need a workaround until NVIDIA/gpu-operator#1687 lands upstream. Shipping those as active templates would silently impose those assumptions. Shipping them as reference files next to the package lets operators opt in with eyes open and understand the moving parts before applying them. ### Scope note The dashboards and recording rules are useful independently of the reference manifests — any cluster that scrapes DCGM Exporter (however it is installed) will populate them. The reference manifests are self-contained and do not change any default behavior of the `gpu-operator` package. ### Screenshots <img width="2824" height="1055" alt="2026-04-27_12-36-52" src="https://github.com/user-attachments/assets/71f85de7-1c80-4817-b36c-aeb1c2bdd39a" /> <img width="2913" height="1255" alt="2026-04-27_12-36-39" src="https://github.com/user-attachments/assets/7a54daca-0727-4510-938c-080d31bdc229" /> <img width="2932" height="1251" alt="2026-04-27_12-36-30" src="https://github.com/user-attachments/assets/f6c6c215-3d05-400c-b21c-29f267c93f23" /> <img width="2864" height="1222" alt="2026-04-27_12-36-00" src="https://github.com/user-attachments/assets/e9d34bab-ade5-434b-92c3-f36712cf68d9" /> <img width="2958" height="1226" alt="image" src="https://github.com/user-attachments/assets/0047a5f8-12a5-4cda-8676-041de139cb76" /> ### Release note ```release-note feat(monitoring): add GPU observability — five Grafana dashboards (performance, fleet, tenants, quotas, efficiency), VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests under packages/system/gpu-operator/examples/ for running GPU workloads natively in pods on Talos. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added five GPU dashboards for performance, efficiency, fleet overview, quotas/allocation, and tenant usage (utilization, power, throttling, allocation, billing). * Added cluster recording rules and a throttling validation alert; new Prometheus-derived metrics and per-GPU rankings/timeseries. * **Documentation** * Added GPU Operator examples and README documenting native Talos setup and required DCGM counters. * **Tests** * Added validation tests ensuring dashboard ↔ recording-rule and DCGM metric consistency. * **Chores** * Registered new GPU dashboards in infra listings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
2b6e20cc3f |
fix(platform): migrate ACME HTTP-01 to ingressClassName API
ClusterIssuer solver referenced IngressClass "nginx" which does not exist on cozystack clusters — real classes are named after tenant namespaces (e.g. tenant-root). Cert issuance only worked because every requesting Ingress overrode the ClusterIssuer via the legacy acme.cert-manager.io/http01-ingress-class annotation. Switch both sides to the modern cert-manager API (available since cert-manager 1.12; cozystack ships 1.19.3): - ClusterIssuer: http01.ingress.ingressClassName, value parameterized from _cluster.expose-ingress (default "tenant-root") - Ingress annotation: http01-ingress-ingressclassname These must migrate together — mixing ingressClassName (ClusterIssuer) with the old http01-ingress-class annotation triggers cert-manager's "fields ingressClassName and class cannot be set at the same time" validation and breaks issuance. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Arsolitt
|
6d9066f074 |
feat(monitoring): add GPU fleet and tenants dashboards
- gpu-fleet: cluster-wide admin view — inventory, capacity (total / allocated / free), per-node utilization and power, throttling, temperatures, XID errors. - gpu-tenants: per-namespace view — live allocation, utilization, tensor saturation, power, and 24h GPU-hours / kWh integrations for billing inputs. Register both under gpu/* in dashboards-infra.list so they ship as GrafanaDashboard CRs and fall under the bats cross-check introduced earlier on this branch. Update examples/README to spell out which DCGM counters each of the five gpu/* dashboards actually needs on top of the upstream default CSV — gpu-performance needs profiling and throttling counters, gpu-efficiency needs profiling, gpu-tenants needs only DCGM_FI_PROF_PIPE_TENSOR_ACTIVE for its tensor panel, and gpu-fleet and gpu-quotas work on the default counter set alone. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Arsolitt
|
c1b9a06a36 |
feat(monitoring): expand GPU dashboards — efficiency and quotas
Revise gpu-performance and add two new dashboards, registered in dashboards-infra.list: - gpu-efficiency (GPU Efficiency Score) — utilization vs. capacity and workload efficiency signals. - gpu-quotas (GPU Quotas & Allocation) — per-namespace requested vs. used GPUs for tenant capacity planning. All three dashboards use the $ds_prometheus template variable, per the project convention. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Arsolitt
|
d1d19e9978 |
feat(monitoring): add GPU performance Grafana dashboard
Add the gpu/gpu-performance dashboard and register it in the infra dashboard list. The dashboard provides: - Cluster overview: total/allocated GPUs, average utilization, aggregate power draw. - Utilization: GPU util (NVML), tensor pipe active (realistic load for LLM/AI workloads), graphics engine active, memory copy util. - Memory: VRAM used/free per GPU. - Power and temperature per GPU. - Health: XID errors, power and thermal throttling. The dashboard relies on DCGM_FI_* metrics plus the cluster:gpu_* and namespace:gpu_* recording rules added to monitoring-agents. The JSON follows the cozystack convention — Prometheus data source is selected via the $ds_prometheus template variable. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Andrei Kvapil
|
78ca1f0789 |
Fix system postgresql images to 17.7-standard-trixie (#2364)
<!-- Thank you for making a contribution! Here are some tips for you: - Start the PR title with the [label] of Cozystack component: - For system components: [platform], [system], [linstor], [cilium], [kube-ovn], [dashboard], [cluster-api], etc. - For managed apps: [apps], [tenant], [kubernetes], [postgres], [virtual-machine] etc. - For development and maintenance: [tests], [ci], [docs], [maintenance]. - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same [label] as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note Fix system postgresql images to 17.7-standard-trixie ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Database Helm charts now preserve and reuse existing PostgreSQL image settings, normalizing tags to the standard-trixie variant when appropriate. * Migration configuration bumped to target version 38. * **Bug Fixes** * Migration logic updated to correctly locate and adjust CNPG Cluster resources and to log per-cluster patch/skip decisions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |