The downstream-coordination checklist still directed contributors at an
"externalIPs to exposureClass" migration; exposureClass was removed in
3dbc7d366 and the native replacement key is publishing.loadBalancerClass.
Last leftover of the ExposureClass removal this PR sweeps.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
## What this PR does
Fixes a v1.5.0 upgrade regression where a tenant ends up with **two**
SeaweedFS workload sets.
**Root cause.** v1.5.0 bumped the vendored SeaweedFS chart `4.0.405 →
4.31.0`. Upstream changed resource naming: before 4.31 the chart named
workloads after the **chart** (`seaweedfs-*`, ignoring the release
name), and 4.31 names them after the **release**. The data-plane release
is `<name>-system`, so every StatefulSet wanted to become
`seaweedfs-system-*`. StatefulSet names are immutable, so Helm could not
rename in place — it stood up a second, empty set beside the running one
while the data stayed on the original `data1-seaweedfs-volume-*` PVCs.
Where the node count let both sets run, they shared one `seaweedfs-s3`
Service (identical pod labels) and one `seaweedfs-db` filer metadata
database, so the empty set answered reads and the two master sets mixed
volume IDs from independent sequences into one metadata table.
**Fix.** Pin `fullnameOverride` in `system/seaweedfs` values so the
rendered names stop following the release name, which is the convention
already used by `cozy-proxy`, `flux-operator`, `grafana-operator`,
`linstor-scheduler` and `victoria-metrics-operator`. The names go back
to what every pre-1.5 cluster already runs, so the upgrade adopts the
running workloads **and their volumes** in place. No data migration, no
platform migration.
**The one case that cannot be adopted.** A tenant installed *fresh* on
1.5.x wrote its data under the renamed workloads, so its volumes are not
the chart-named ones. Pinning the chart name there would rename the
workloads *away* from that data and bring up an empty cluster beside it.
Helm cannot move data between PVCs, so `extra/seaweedfs` refuses to
render for such a tenant and points at the recovery runbook, which
re-binds the existing PVs onto the adopted PVC names (no copying — the
claim is renamed, the volume is not touched). The guard is fail-closed:
it fires only when renamed volumes are present and no adopted ones are.
Two naming subtleties the guard has to respect, both verified by
rendering the shipped 1.5 chart:
- Before 4.31 the chart ignored the release name, so an instance running
under a non-default name (`SeaweedFS` is a user-creatable kind) *also*
stored its data on `data1-seaweedfs-volume-*`. Such a tenant is adopted,
not blocked.
- On 4.31 the fullname helper appends the chart name when the release
name does not contain it, and the component helper truncates to 56
characters. A fresh 1.5.x instance named `archive` therefore wrote to
`data1-archive-system-seaweedfs-volume-*`, and a long name loses the
chart name from its tail entirely. The guard matches volumes by shape
and by the chart labels on the StatefulSet instead of reconstructing
those names, so both are caught.
**On the duplicate set.** Once the fix reconciles, the
`seaweedfs-system-*` StatefulSets and Deployments are no longer part of
the release and Helm removes them on its own. What it cannot remove —
PVCs it did not template, and cert-manager Secrets that carry no owner
reference — is inert, and the runbook clears it explicitly. A
pre-upgrade migration was considered and dropped: it would duplicate
what Helm already does, while its only unique action (deleting PVCs)
cannot distinguish "the duplicate never served" from "it served and is
currently down", and any transient delete failure under `set -e` would
fail the platform's pre-upgrade hook and block delivery of this very
fix.
## Testing
`helm unittest` — `system/seaweedfs` 21 tests, `extra/seaweedfs` 17
tests. The guard's `lookup` branches are covered by mocking the
Kubernetes provider (the pattern already used in `core/platform` and
`system/cozystack-api`): adopted-only, renamed-only, both sets, zoned
volumes, a non-default instance name, a truncated long name, and an
unrelated `data1-*-volume-*` PVC that must not trip the guard. The
s3-consumer suite now re-creates the name/fullname divergence
explicitly, so it keeps guarding that regression regardless of the
shipped default.
The remaining regression test for this class of bug is an upgrade E2E
(seed a pre-4.31 named set, upgrade, assert a single set with data
intact) — no fresh-install E2E or `helm template` can reproduce an
upgrade over pre-existing state. That belongs in the chainsaw upgrade
lane and is a follow-up.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(seaweedfs): adopt existing SeaweedFS workloads and volumes in place when upgrading past the 4.31 chart rename, instead of standing up a second, empty cluster beside them
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an audit & recovery runbook for SeaweedFS 4.31 rename impacts.
* **Bug Fixes**
* Stabilized SeaweedFS workload/PVC naming across upgrades, including
non-default instance names.
* Added preflight safeguards to prevent upgrade scenarios that could
split or wedge existing data.
* Updated post-delete cleanup to remove secrets from both legacy and
current naming schemes.
* **Tests**
* Expanded Helm unit coverage for naming guards, cleanup behavior, and
S3 service name consumers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Consolidates the etcd `v1alpha2` transition fix for **in-cluster 1.5 →
1.6 upgrades** into a single PR, rebased on current `main`. Supersedes
#3265 and #3261 — their commits are carried here (authorship preserved),
so those PRs can be closed once this lands. Fresh installs use the new
shapes directly and are unaffected, so fresh-install CI does not catch
these; the 1.5 → 1.6 e2e upgrade path is the authoritative regression
guard.
## What this PR does
### 1. Keep the legacy `etcd-headless` Service alive during adoption —
`packages/extra/etcd`
`etcd-migrate` adopts legacy clusters **in place**: the Pods keep their
original `spec.subdomain: etcd-headless` and are dialed at
`etcd-<i>.etcd-headless.<ns>.svc` until they roll onto the operator's
native `<member>.etcd.<ns>.svc` domain. The v1alpha2 operator only
creates the native `etcd` Service and the legacy `etcd-headless` Service
is pruned, so those per-pod names stop resolving (`no such host`),
`MemberList` fails, and `status.readyMembers` never populates — the
`EtcdCluster` never goes `Ready` even though etcd is healthy and in
quorum. We ship a chart-managed transitional headless `etcd-headless`
Service (selector mirrors the operator's native `etcd` Service via
`etcd-operator.cozystack.io/cluster`, `publishNotReadyAddresses: true`)
— the DNS counterpart of the legacy `*.etcd-headless.<ns>.svc` SAN
already kept for this window. Removable together with that SAN once
members roll onto the native subdomain.
### 2. Survive the immutable controller-Deployment selector on upgrade —
`packages/system/etcd-operator` (#3242)
#2859 replaced the upstream etcd-operator chart with the
cozystack-authored one, changing `Deployment.spec.selector.matchLabels`.
`spec.selector` is immutable, so `helm upgrade` cannot patch the
existing Deployment and the whole HelmRelease upgrade fails (`field is
immutable`). A **pre-upgrade hook**
(`templates/pre-upgrade-selector-fix.yaml`: ServiceAccount + Role +
RoleBinding + Job) deletes the Deployment **only** when its live
selector is the pre-1.6 one, so Helm recreates it cleanly. No-op when
the selector already matches, never runs on fresh install.
### 3. Raise the operator's memory cold-start floor —
`packages/system/etcd-operator`
Steady-state working set is ~250Mi; the static `limits.memory: 128Mi`
OOMKills a Pod that starts before the VPA admission webhook rewrites it.
Raise the floor to `256Mi` (and VPA `minAllowed` to match) as defense in
depth so the operator never depends on VPA timing to avoid crashing.
### 4. Make migration 50 (etcd adoption) robust in-cluster —
`packages/core/platform/images/migrations/migrations/50` (was #3261)
Exact server/peer cert-SAN match, Secret-gated wait on the adoption
Secret, and in-cluster (IPv6-safe) kubeconfig handling, so the migration
script drives the adoption reliably from inside the cluster. Covered by
`hack/migration-50-etcd-adopt.bats`.
### 5. Hardening / review fixes (this PR's original scope)
- **Hook `runAsUser: 65532`.** `pre-upgrade-selector-fix.yaml` set
`runAsNonRoot: true` but no numeric `runAsUser`; `clastix/kubectl`'s
image user is the non-numeric name `nonroot`, which the kubelet cannot
verify against `runAsNonRoot` — so the hook Pod fails admission and
silently blocks the very upgrade it exists to unblock. Adds `runAsUser:
65532`, matching the postgres-operator webhook-ready hook that runs the
same image.
- **Digest-pin the kubectl image.** The values comment claimed
digest-pinning but shipped a floating `v1.32` tag. Reuses the digest
postgres-operator vendors, adds the `renovate` annotation, and templates
`repo:tag@digest`.
- **Tests.** `etcd-operator/tests/selector-fix-hook_test.yaml` (hook
wiring, weight ordering, namespaced least-privilege RBAC,
numeric-non-root security context, digest-pinned image),
`etcd-operator/tests/deployment_test.yaml` (256Mi cold-start floor),
`extra/etcd/tests/etcd-cluster_test.yaml` (transitional `etcd-headless`
Service). Each assertion was mutation-tested.
### 6. Derive the default S3 endpoint from the provisioned bucket —
`packages/system/backupstrategy-controller`
Derive the default S3 endpoint (and per-driver scheme / TLS /
`secure_connection`) from the provisioned bucket Secret instead of
requiring it to be hand-set, so etcd (and other) backup strategies get a
working endpoint by default. Docs in
`docs/operations/backup-classes.md`; covered by
`tests/endpoint_form_test.yaml`.
### Verification
- `helm unittest` green on current `main`: etcd-operator **18/18**,
extra/etcd **18/18**, backupstrategy-controller **11/11**.
- Behaviours 1 & 2 were reproduced and confirmed live on a 1.5.2 →
1.6.0-rc.1 adoption (3-node cluster) — recreating the `etcd-headless`
Service took the adopted cluster to `readyMembers=3 / Available=True`.
```release-note
fix(etcd): complete the v1alpha2 transition on in-cluster 1.5→1.6 upgrades — keep the legacy etcd-headless Service alive so adopted members stay resolvable, delete the pre-1.6 operator Deployment via a pre-upgrade hook to get past the immutable selector, raise the operator's memory floor so it does not OOM before the VPA scales it, and make the etcd adoption migration robust in-cluster.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Improved backup storage endpoint handling across supported backup
drivers, including provisioned and external S3 storage.
* Added compatibility support for legacy etcd pod discovery during
migration.
* Added an automated upgrade safeguard for etcd operator deployments.
* **Bug Fixes**
* Improved certificate SAN detection and etcd migration authentication.
* Increased the etcd operator’s minimum startup memory to prevent early
restarts.
* **Documentation**
* Clarified backup endpoint, TLS, and driver-specific behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The `backupstrategy-controller.endpoint` helper changed how the S3 endpoint is
resolved (for a provisioned bucket it derives from the COSI system Secret and
forces https://, with `backupStorage.endpoint` demoted to a fallback), but
docs/operations/backup-classes.md still described the old single-verbatim-value
behavior and directly contradicted the code:
- the "Endpoint format per driver" intro said the templates adapt "the single
backupStorage.endpoint value";
- it claimed drivers "pull from backupStorage.endpoint in chart values, not
from the Secret" — now precisely backwards for provisioned buckets;
- the values-table `endpoint` row omitted the fallback/derive semantics.
Rewrite those to match the new helper + the values.yaml comment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Verified end-to-end on dev3 with Grafana v11.6.15: OAuth flow reaches
Grafana, id_token and userinfo response both carry the correct
`groups` claim (`[cozystack-cluster-admin, tenant-root-super-admin]`
for kvaps), `allowed_groups` in grafana.ini lists
`tenant-root-super-admin`, and login still fails with
`user not a member of one of the required groups`.
Grafana debug log shows `Attributes: map[]` — the extracted user's
Groups slice is empty. On v11.x+ Grafana does not auto-map the
`groups` claim onto user.Groups unless `groups_attribute_path` is
set explicitly (the field is documented as optional but the gate is
a silent no-op without it — a claim-name-string is not enough).
Fix: hard-wire `groups_attribute_path=groups` in the chart-forced
System-mode block alongside `allowed_groups`. "groups" points at
the top-level `groups` array from userinfo, matching the shape
Keycloak's oidc-group-membership-mapper emits (verified on the same
dev3 run).
Two helm-unittest cases updated to assert `groups_attribute_path`
lives next to `allowed_groups` in both the users-map-active and
empty-users branches (rejecting the silent no-op regression). Docs
also updated to name the requirement.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Symmetric with the Grafana-side allowed_groups gate — round-3 review
from lllamnyp on PR#3176 covered both surfaces, and #3044's original
System-mode wiring shipped without this guard. The rationale is
identical: audience binding stops REPLAY of a minted token across
tenant kube-apiservers, but nothing stopped Keycloak from ISSUING a
usable token to a caller outside the tenant.
Absent a claim-side gate, any authenticated cozy-realm user could
`kubectl oidc-login` against another tenant's cluster and land at
`system:authenticated`. RBAC default-denies named-resource access, but
`system:authenticated` still leaks discovery (kubectl api-resources,
OpenAPI schemata) — tenant-alice could enumerate tenant-bob's cluster's
CRD shape and built-in resource surface.
Fix: add a CEL `claimValidationRules` entry to the rendered
AuthenticationConfiguration (System mode only) that requires at least
one of the release's four tenant-scoped Keycloak groups
(`<namespace>-{view,use,admin,super-admin}`) to be present in the
token's `groups` claim. The `has(claims.groups) &&` guard is required
— CEL evaluation on a missing claim surfaces as HTTP 500 from the
authenticator instead of a clean 401.
CustomConfig mode is not touched — the tenant's own
AuthenticationConfiguration is authoritative and the tenant is
responsible for claim-side guards on their own issuer.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Round-3 review from lllamnyp: any user in the shared `cozy` Keycloak
realm could sign into any tenant's Grafana. The audience mapper stops
replay of a minted token across releases, but does not gate who may
mint one — tenant-alice could complete the OAuth flow against
tenant-bob's Grafana. With `spec.oidc.users` non-empty the earlier
`allow_sign_up=false` fix mostly closed the door (though tenant-alice
still hit `error 401` after landing in Grafana, revealing the release
exists); with empty users under mode: System — a config the schema
permits and the docs describe as hands-off — the login succeeded and
minted a Viewer account for every cozy-realm identity.
The fix uses the KeycloakRealmGroups the tenant chart already
provisions (`<namespace>-{view,use,admin,super-admin}` in
packages/apps/tenant/templates/keycloakgroups.yaml) — no new
directory objects owned by the Monitoring release. Two chart edits:
1. oidc-keycloak.yaml: add the platform-managed realm-default `groups`
scope to `defaultClientScopes` explicitly. EDP's KeycloakClient
does NOT merge realm-level defaults into `defaultClientScopes`, so
the scope must be listed here or the `groups` claim would be
missing from the token.
2. grafana.yaml: in System mode, unconditionally emit
`allowed_groups: <ns>-view <ns>-use <ns>-admin <ns>-super-admin`
in `auth.generic_oauth` and add `groups` to `scopes`. Grafana
rejects any login whose token does not carry at least one of the
groups.
The gate is chart-owned and does not depend on the users-map — with
`users: []` tenant members still get a self-service Viewer through
`auto_assign_org_role`, cross-tenant users get 401 in Keycloak's OAuth
callback. Docs updated to reflect the two-layered contract
(tenant-membership gate + optional users-map).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Address PR#3176 review round-2 gap-2: the users-Job's prune pass deletes
every non-admin Main-Org member whose email is not in the desired list.
When the desired list is empty every helm-upgrade wiped the whole org.
Three empty-users states used to fall into this trap:
* customConfig.secretRef: users[] is forbidden by
assertCustomSecretRefUsersEmpty, so the desired list was
structurally always empty and the chart wiped an org whose auth.ini
fragment the operator advertised as authoritative.
* System with users[] unset: the operator opted into OIDC but not
into the chart-managed users-map, yet the Job still pruned every
Main-Org member on their behalf.
* CustomConfig-inline with users[] unset: same shape as System.
The mode=None guard (from an earlier round) already had the same
shape; extend the render gate to cover the other empty-list doors —
now the Job renders only when `mode != None AND users non-empty`.
The trade-off: taking `users:` back to `[]` no longer prunes the last
entry — matching the already-documented `System → None` posture —
called out explicitly in the docs.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Caught by dev3 end-to-end test on the PR working tree
(cozystack-pr-test skill): the users-Job was rendered in every mode
including `None` on the rationale that a `mode: System → None`
switch should garbage-collect OIDC-provisioned local accounts. That
rationale ignored the upgrade path from the pre-PR chart: an
existing tenant that never had `spec.oidc` on their Monitoring CR
would receive `mode: None` on first render under the PR, the Job
would fire on every Flux reconcile of the outer HR, and its prune
pass would DELETE every non-admin Main-Org member — including users
the tenant added by hand through the Grafana UI on the pre-PR chart.
Silent breaking change with data loss.
Wrap the whole Job resource in `{{- if ne $mode "None" }} … {{- end }}`
so `mode: None` = "chart owns nothing in Grafana orgs, we don't
touch what's there". Operators who need to prune OIDC-provisioned
accounts after switching System | CustomConfig → None do it
themselves through the Grafana UI or admin API — same posture as
"we don't own the Keycloak directory objects" from the earlier
architectural change.
The `assertUsersPrologue` include stays outside the `if` so an
operator listing users in `mode: None` still gets the schema and
email-shape validation at render time — the Job body just doesn't
emit.
Test coverage:
- Flip the previous "mode=None still renders the users-Job so a
mode switch can prune stale members" case to
`mode=None does NOT render the users-Job (upgrade safety guard)`
as a regression lock.
- Add `mode=None with non-empty users still fails render` covering
the prologue-only-fires-on-schema-issue path.
Docs: rewrite the "Users and RBAC" section step 3 to name the
upgrade-safety motivation and the manual-cleanup handoff.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The failed-hook post-mortem paragraph in the users-Job troubleshooting
section still said the Job "stays around until the next helm-upgrade
for post-mortem", but the shipped Job spec carries
`ttlSecondsAfterFinished: 3600` — the Kubernetes TTL controller reaps
the Job + Pod one hour after the terminal Failed condition regardless
of when the next `helm upgrade` lands. Reword to name the 1h window
explicitly and instruct the operator to re-trigger the hook if they
missed the window.
The edit was authored alongside commit 1f664566 but slipped out of
that commit's file list; this is the cleanup.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Previous docs claimed "the chart forces skip_org_role_sync = true
regardless of mode" — a promise the code did not keep in
CustomConfig mode. The chart is now fixed; sync the docs so the
guarantee reads truthfully and the incompatibility surface is
explicit:
- `docs/oidc-grafana.md`: rewrite the CustomConfig section so both
chart-forced settings are named explicitly, the `merge` semantics
are documented (operator can't override), and the
`secretRef` + `users` incompatibility is called out with an
explicit failure path.
- `system/monitoring/values.yaml`: expand the `config`/`secretRef`
comments so operators reading the values doc see the merge behavior
and the secretRef limitation before they hit the render-time fail.
Add `enabled: "true"` to the "typical keys" list — Grafana defaults
`enabled: false` and an inline map without it is inert.
- `extra/monitoring`: regenerated values.schema.json + README.md via
`make generate` (cozyvalues-gen + hack/update-crd.sh) to pick up
the `oidc.users[].email` doc-string touch; mirrored openAPISchema
on the ApplicationDefinition follows.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Rewrite `docs/oidc-grafana.md` for the new authorization model:
- Selector example carries a `users:` map.
- System-mode "what the chart provisions" list drops the three
KeycloakRealmGroups + role_attribute_path, adds the audience
scope-only KeycloakClient defaults, and adds the users-Job step
with `skip_org_role_sync` + `oauth_allow_insecure_email_lookup`
wiring notes.
- Fresh "Users and RBAC" section describes the users-Job reconcile
contract: pre-provision -> add-to-org -> PATCH role -> prune.
Explains the "user not in list -> gets nothing" semantics
(replaces the default-Viewer fallback).
- CustomConfig section explicitly notes that `role_attribute_path`
in the operator's payload is now dead config: the chart forces
`skip_org_role_sync = true` regardless of mode.
- "Failure modes" gains "user logs in but sees no dashboards"
(missing from `users:`) and a users-Job troubleshooting entry.
- Drop the `GrafanaAdmin` deferral notes and the "Mode toggle
destroys chart-owned KeycloakRealmGroup membership" failure
mode — both moot now that the groups are gone.
- Removed the stale `KeycloakRealmUser` example that used
`tenant-acme-monitoring-admin` (the chart never rendered a
group by that name — the inner release is `monitoring-system`
and the example was a live footgun).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The MAJOR finding from the cozy-review agent: the platform-side
`oidc.grafanaAdmin: true` documented in the PR body and in
docs/oidc-grafana.md never actually reached the platform Grafana's
HelmRelease. `Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin`
was set in packages/core/platform/templates/bundles/system.yaml, but
`monitoring-system` (and `monitoring`) have no `install:` block in
packages/core/platform/sources/monitoring-application.yaml — only
`monitoring-rd` does. PackageReconciler skips components without an
`install:` block before ever reaching the assignment that copies
`pkg.Spec.Components[].Values` onto `hr.Spec.Values`
(internal/operator/package_reconciler.go around lines 197-201 and
292-293). The values-override dictionary was silently dropped, no
HR carried `oidc.grafanaAdmin: true`, and the platform Grafana
rendered with the chart default (`false`). The
`bundles_monitoring_grafana_admin_test.yaml` unit test only asserted
against the emitted Package CR spec — exactly the layer where the
value still existed but was about to be discarded — so the broken
propagation slipped past.
Trim the broken promise from Phase 1. The chart-side helper
(`monitoring.oidc.allowAssignGrafanaAdmin`) stays as-is so a future
phase that routes the override through the tenant-root chart's
monitoring HR values (or adds an `install:` block on the
`monitoring-system` component) lands on a green chart-side contract.
The helm-unittest that verifies chart-side emission stays too, but
with a docstring calling out that end-to-end wiring is deferred.
Everything else that promised the lever works — bundle line, docs,
platform bundle test — is either removed or rewritten to match
reality.
Also picks up two MINOR findings from the same review:
* Drop the misleading `secret-hash` annotation on the OIDC
client-secret Secret. The comment claimed it "rotates the
KeycloakClient CR and therefore the Grafana Deployment env when
the client-secret changes", but nothing downstream watches an
annotation on the Secret itself — no rotation is triggered. The
`lookup`-based fallback already keeps the value stable across
reconciles, so no rotation is even needed. Removing the dead
annotation avoids planting a broken assumption for future readers.
* Add `release.cozystack.io/helm-install-timeout: "15m"` on the
monitoring cozyrds ApplicationDefinition. Under `mode: System` the
chart now emits, on top of the existing Grafana + Alerta +
VMCluster + CNPG stack, four additional Keycloak CRs whose
reconciliation is asynchronous against an operator in
`cozy-keycloak` (not the tenant namespace). Flux's default 5m
install window was already tight for the pre-OIDC stack; 15m
matches the precedent set by tenant-rd/kubernetes-rd cozyrds.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The tenant-facing `Monitoring` CR exposed `spec.oidc.grafanaAdmin`
via `packages/extra/monitoring/values.schema.json`, which the
ApplicationDefinition mirrored into `openAPISchema`. A tenant with
create/patch on `Monitoring` in their own namespace could set the
flag on their release, promoting members of their
`<clientId>-admin` Keycloak group to server-level `GrafanaAdmin`
instead of the intended org-level Admin.
Drop the field from the tenant surface. The platform bundle still
flips it on for the `monitoring-system` component through
`Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin`
(the platform Grafana chart continues to read the value verbatim
in `packages/system/monitoring/values.yaml`); tenant Grafana
releases inherit the chart default `false` and stay at org-level
Admin.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Follow-up to the previous self-review-fix commit: the unconditional
`- cozystack.keycloak-operator` in dependsOn was itself a
CRITICAL cold-install regression on any cluster where
`authentication.oidc.enabled: false` (the default). The
keycloak-operator Package is only emitted by
`packages/core/platform/templates/bundles/system.yaml:112` when
that flag is on — a hard dep against it from the default variant
would leave `cozystack.monitoring-application` stuck on
DependenciesNotReady forever, cascading a
`cozystack-platform` HR timeout that hits every OIDC-off customer.
Fix by mirroring the cozystack-engine variant split:
- `default` variant keeps the baseline reconciliation edges
(no keycloak-operator dep).
- `oidc` variant additionally waits for
`cozystack.keycloak-operator` so the v1.edp.epam.com CRDs are
registered before the monitoring chart reconciles and the
`.Capabilities.APIVersions.Has` guard in
templates/grafana/oidc-keycloak.yaml can safely emit
KeycloakClient/Scope/Groups.
The bundle template swings between variants on
`.Values.authentication.oidc.enabled`, mirroring the
cozystack-engine invocation two lines above. The
`monitoring-system` component override
(oidc.grafanaAdmin: true) is preserved across both branches — the
values field is a no-op when mode: None (no auth.generic_oauth
block is emitted at all), so keeping it identical avoids two
divergent code paths.
Also widened the platform helm-unittest coverage to lock:
- default variant does NOT depend on keycloak-operator (permanent
DependenciesNotReady guard).
- oidc variant DOES.
- Bundle emits the right variant based on
authentication.oidc.enabled AND keeps the GrafanaAdmin uplift
on the monitoring-system component in either branch.
83 platform tests + 23 monitoring tests still green.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Address the two blocking-class findings from the cozy-review pass on
this PR:
(1) Missing PackageSource dependency on cozystack.keycloak-operator
(MAJOR cold-install race).
Without this edge, the monitoring chart can reconcile before the
v1.edp.epam.com CRDs are registered on a fresh cluster — the
`.Capabilities.APIVersions.Has` guard in
templates/grafana/oidc-keycloak.yaml then silently drops all
Keycloak-side objects for mode: System, and Grafana's
auth.generic_oauth block points at a client Keycloak never
provisions. Add the ordering edge, plus a
sources_monitoring_application_dependson_test.yaml regression
guard so a future edit cannot drop it silently.
(2) allow_assign_grafana_admin driven by `.Release.Name ==
monitoring-system` (MINOR fragile coupling).
Replace the release-name sniff with an explicit
`oidc.grafanaAdmin: false` values field, and flip it on for the
platform Grafana release via
Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin
in packages/core/platform/templates/bundles/system.yaml —
mirroring the pattern monitoring-agents already uses for
per-component overrides. Tenant Grafana releases inherit the
chart default false and stay at org-level Admin.
Two new helm-unittest cases in packages/system/monitoring/tests/
lock the values-driven behaviour on both sides
(grafanaAdmin=true → Admin uplift, default → tenant stays at org
Admin), plus bundles_monitoring_grafana_admin_test.yaml in
packages/core/platform/tests/ regression-guards the Package's
component override so the platform release keeps its uplift on
future edits.
Also document the mode-toggle group-membership loss in
docs/oidc-grafana.md — mode: System → CustomConfig/None deletes the
three chart-owned KeycloakRealmGroups; membership is not restored on
toggle-back. Not a bug, but the operator UX cost of the toggle
deserves a bullet in Failure modes.
Regenerated artifacts (cozyvalues-gen + hack/update-crd.sh):
packages/extra/monitoring/values.schema.json
packages/extra/monitoring/README.md
packages/system/monitoring-rd/cozyrds/monitoring.yaml
Test suites still green: packages/system/monitoring 23/23,
packages/core/platform 81/81.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
New docs/oidc-grafana.md parallel to docs/oidc-tenant.md from
cozystack/cozystack#3044. Covers:
- The three modes on the Monitoring CR (None / System / CustomConfig)
and how the selector maps to what the chart provisions.
- What System mode creates on the cozy realm: a per-instance
confidential KeycloakClient, a KeycloakClientScope with the
audience mapper, three KeycloakRealmGroups, and the persistent
client-secret Secret; plus the Grafana CR side (auth.generic_oauth
section, GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET env).
- Both CustomConfig payload paths (inline vs secretRef) and the XOR
guard.
- User-role mapping via role_attribute_path and how an operator adds
users to the three groups through a KeycloakRealmUser.
- Failure modes (assertSystemEnabled, XOR, wrong claims,
emailVerified prescriptive requirement) and what is out of scope
for Phase 1 (per-tenant realms, backend-logout-url, disabled login
form, CEL claimValidationRules).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Classify each tenant from its volume PVCs, re-bind the volumes of a tenant
installed fresh on 1.5.x so the adopted names pick them up, stop a split cluster
before upgrading it, and clear the leftovers the upgrade cannot remove itself.
The recovery steps select the renamed set by a precise filter and derive secret
names from the fullname helper, so they work for an instance running under a
non-default name and for pooled or zoned volumes.
Co-authored-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Cozystack is upstream for repositories that nothing keeps in sync with
it, and no CI job compares the two sides, so a change here breaks them
in silence. Only two narrow paths are automated, and both fire on a
release rather than on a PR.
It has already happened. The website docs generator works from an app
list hardcoded in its own Makefile, and packages/apps/opensearch was
never added to it, so an application we ship has no reference page at
all. The Terraform provider still offers a Kubernetes version this repo
removed. The plugin repo still sends operators to hack/ scripts that
were deleted, one of them rewritten as a Go command.
Add a Downstream Repositories trigger map to the contributing guide,
naming per repository which change here forces a change over there and
which file to touch, and a checklist in the PR template that links to
it. The template addresses AI agents directly: they author a large share
of PRs and will otherwise tick a box without opening the follow-up it
claims.
The coupling is wider than the obvious ones, and mostly invisible. talm
restates the node contract, overlapping ours but already diverged, so it
must be diffed rather than overwritten. cozyhr reads an annotation the
operator writes and silently merges the wrong values if it is renamed.
cozy-proxy is vendored in from its own repo, yet consumes labels
produced here and quietly stops handling external VM services if they
move. The telemetry server queries metric names emitted here, and a
rename just empties its dashboards. None of these fail loudly.
Guard the map with a unit test, because a map about silent drift must
not drift silently itself. It pins every in-repo path the map cites to a
file that exists, pins the map and the checklist to the same repository
list, and pins the workflow carve-out below to the map's location, to
the job that reads the flag, and to the output that carries it: cutting
any one of those three leaves the other two looking wired up. The header
states plainly what the suite cannot check, so a green tick is not
mistaken for a correct map: whether a coupling is real, and whether it
still holds over there, stays a human's job.
Editing the map alone is a docs-only PR, which skips the unit tests and
would exempt the map from its own guard. Rather than widen the docs-only
rule, which also gates the Talos image build, the plan step exports a
separate flag that only the unit-test job reads: a docs PR now runs the
tests and still builds nothing. It detects the map by its old path too,
since a rename is reported as the new path alone.
Document that gh pr create --body and --body-file replace the body
wholesale, which would otherwise drop the checklist from every
CLI-authored PR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The v0.2.15 syntax/semantics traps were tribal knowledge scattered across PR
review notes (the number-literal caveat lived only in #2826's body, which
evaporates on merge). Consolidate them into a durable subsection so the next
author finds them before hitting each one fresh: filter-as-list conditions,
no number-literals/backticks in JMESPath, error: fails hard on an unknown
GVK, no conditional Test-level skip, step-level-only finally, bounded/blocking
cleanup surfacing latent teardown bugs, projection-style failure diffs, and
the catch-script cwd.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconciles the talos-image-cache conflicts created when #3254 merged to
main: keep the Chainsaw _lib/ port (hack/e2e-chainsaw/_lib/talos-image-cache.sh,
a superset of #3254's logic + the node-join diagnose helper), drop the
BATS-era hack/e2e-apps/talos-image-cache.sh, and keep the _lib/-path
versions of the mirror manifest, install bats, and unit test. Also picks
up main's zizmor action-pinning (#3223, unblocks pre-commit/zizmor) and
Talos v1.13.6 (#3240).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Pre-release tags (vX.Y.Z-alpha.N / -beta.N / -rc.N) were cut by a manual
`git push origin HEAD:refs/tags/<tag>`. That leaves the immutable-tag work in
#2677 only partly enforceable: a repo ruleset cannot restrict `v*` tag
*creation* to CI while still letting maintainers hand-push pre-releases (fnmatch
has no way to match "v* except v*-rc.*"), so "do not hand-cut stable tags" stays
a matter of discipline backed only by an in-workflow guard.
Add cut-prerelease.yaml — a workflow_dispatch that becomes the sole entry point
for creating a pre-release tag:
- Generates the CozyStack CI app token (same app as promote-rc.yaml) and pushes
the tag as the app, so tags.yaml fires (a tag pushed with the default
GITHUB_TOKEN would not trigger it) and the push-event base_ref is populated
(tags.yaml's Get-base-branch step needs it).
- Validates ^v\d+\.\d+\.\d+-(alpha|beta|rc)\.\d+$ and rejects stable vX.Y.Z —
stable tags are cut write-once by the promote flow at a PR merge commit.
- Enforces dispatch from main or the matching release-X.Y branch, and refuses to
move an existing tag (write-once).
- Pushes HEAD:refs/tags/<tag> from the branch tip — the exact form
docs/release.md prescribes — on ubuntu-latest (no build runner needed).
With every pre-release cut by CI, repo admins can then enable a "Restrict
creations" tag ruleset on `v*` with the CI app as the sole bypass actor, turning
tag-creation discipline into hard enforcement.
Also update tags.yaml's two human-facing error messages and docs/release.md (the
RC and patch flows, the workflow list, and a new repo-side tag-ruleset
enforcement note) to point at the workflow instead of a manual push.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds `docs/vm-external-vlan.md`, a guide for attaching a `vm-instance`
VM directly to an external, physically-routed VLAN via a Linux bridge
and a `bridge`-type NetworkAttachmentDefinition.
The documented VM networking story is overlay-only (pod network +
KubeOVN VPC subnets). Attaching a VM to a real VLAN — so it shares a
broadcast domain with external hardware and takes an address from that
VLAN's subnet — is undocumented, and the obvious `macvlan` approach
silently fails with KubeVirt's default bridge binding: macvlan
demultiplexes ingress by the child interface's MAC while bridge binding
puts the guest's own MAC on the wire, so gateway replies reach the
parent interface and are dropped before the guest. The working recipe is
a Linux bridge + `bridge` CNI NAD.
The guide covers why bridge and not macvlan, the host bridge, the NAD
(one per tenant namespace), assigning a static guest address through
cloud-init (the chart has no `networkData`), the always-present
dual-homed pod NIC, and the post-recreate ARP staleness from KubeVirt
regenerating the guest MAC.
Docs-only; no chart or manifest changes. The recipe depends on the
`bridge` CNI plugin being present in `/opt/cni/bin`; #3195 makes
Cozystack's multus package stage the reference plugins there.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
docs(vm-instance): add a guide for attaching a VM to an external VLAN via a Linux bridge and a bridge-type NetworkAttachmentDefinition
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added a new guide for connecting a `vm-instance` directly to an
externally routed VLAN.
* Includes setup steps for creating a bridged VLAN interface, defining a
network attachment, and configuring the VM with a static IP.
* Notes important networking caveats such as dual-homing behavior, MAC
changes after recreation, and host-to-VM connectivity requirements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Implements the immutable-tag + rc-promotion flow from #2677. A **stable
release becomes a renamed release-candidate**: the bytes shipped as
`vX.Y.Z` are bit-for-bit the bytes built and e2e-tested as
`vX.Y.Z-rc.N`. No tag is ever force-moved, and stable is never rebuilt —
it is *promoted* by retagging the rc's existing images.
Layered onto the build matrix from #2937/#2983 (this PR is **stacked on
`refactor/build-matrix-2937`** and must merge after it).
## The five force-retag sites, removed
| Site | Before | After |
|---|---|---|
| `tags.yaml` api/apps/v1alpha1 tag | `git tag -f` / `push -f` |
write-once (create-if-absent, fail if it would move) |
| `tags.yaml` release-X.Y.Z branch | `git branch -f` / `push -f` |
compare-before-force (no-op if unchanged; staging branch only) |
| `pull-requests-release.yaml` stable tag | `git tag -f` / `push -f` |
write-once at the PR merge commit (force impossible by construction) |
| `pull-requests-release.yaml` maintenance branch | `updateRef
force:true` | fast-forward-only |
| `auto-release.yaml` patch tags | delete-recreate (cron) | **workflow
deleted** — stable only via explicit promote |
## Version decoupling (the enabler)
The operator baked its version into the image at build time, so an rc
image self-reported the rc string — blocking retag-promotion. It now
reads `COZYSTACK_VERSION` from the environment (threaded via
`cozystackOperator.platformVersion` → Deployment env, stamped by `make
manifests`), falling back to the build-time value when unset. The same
image bits can report any release name. The only runtime reader is the
telemetry metric `cozy_cluster_info{cozystack_version=...}`.
## Promotion flow
`promote-rc.yaml` (`workflow_dispatch`, `rc_tag=vX.Y.Z-rc.N`):
1. Validate the rc release exists and the stable tag does not.
2. `hack/promote-retag.sh` reads the rc's digest-pinned image refs from
`packages/*/*/values.yaml` and `skopeo copy`s each — **by digest** — to
`:vX.Y.Z` and `:latest`, verifying with `skopeo inspect`.
3. Rewrite the cosmetic `-rc.N` substring in vendored tags to the stable
version (the `@sha256` wins regardless), restamp the version-stamped
assets (operator manifests, cozypkg, openapi; the heavy Talos assets are
copied verbatim from the rc draft), open a `release-X.Y.Z` PR.
4. Merging the PR reuses the existing release-PR e2e and
`pull-requests-release.yaml` finalize to cut the **write-once** stable
tag at the merge commit and publish the release. Squash is disallowed
(the tag needs a real merge commit).
`nightly.yaml` cuts write-once `*-nightly.<date>` tags (gated by
`NIGHTLY_ENABLED`); `retention.yaml` keeps the newest 14 per line
(dry-run by default).
## Validation
**Local end-to-end — the core claim is proven, not asserted.** The whole
design rests on "copy-by-digest to a new tag preserves the digest, so
stable == the e2e-tested rc, bit-for-bit." This was exercised against a
real registry (`ttl.sh`) with real `skopeo`:
- Pushed a multi-arch image under a `:v1.4.0-rc.2` tag, built a temp
tree exercising all three `values.yaml` digest shapes (single `image:`
string, split `repository`/`tag`/`digest` map, and the
`platformSourceRef` OCI artifact), then ran the actual
`hack/promote-retag.sh v1.4.0`.
- **Result: `:v1.4.0-rc.2`, `:v1.4.0`, and `:latest` all resolve to the
identical digest** (`sha256:fd8d9aa6…`), across all 17 platform
manifests. The script's own `skopeo inspect` post-check passed, and an
independent re-inspection confirmed it.
Local validation caught (and this PR fixes) two bugs that only surface
against a real registry — never in static lint:
1. `skopeo copy --multi-arch all --all` — mutually exclusive flags,
fatal error; every retag would have failed on CI. Now `--multi-arch
all`.
2. The same digest was retagged twice when an image appeared in two
value shapes; the ref set is now deduped on the canonical `repo@digest`.
**Other local checks:** `go test ./pkg/version` + `go build`/`vet`;
`helm template` renders `COZYSTACK_VERSION` in all 3 operator variants
(omitted when unset); `make manifests` stamps the version into the
install assets; `shellcheck` clean on `promote-retag.sh`; `actionlint`
clean and `act -l` resolves the job graph on all workflows; the rc-tag
parser and nightly version-math unit-tested (accept/reject +
minor-vs-patch bumps).
**Still CI-only (cannot be exercised offline):** the full workflow
runtime — github-script API calls, OCIR auth, the rc-release /
staging-branch preconditions, the nightly `base_ref` push mechanics, and
self-hosted runners. The *logic* inside the steps is unit-tested; the
*orchestration* is not. Treat it as unproven until the workflows run.
## Out of scope / required follow-up (needs repo admin — not doable from
a PR)
- Tag-protection rules: `v*` and `api/apps/v1alpha1/*` create-only by
the CI app, no delete/update; "limit branches/tags updated in a single
push" = 1.
- Enable `NIGHTLY_ENABLED` (and `RETENTION_APPLY` when ready) repo
variables.
- Registry-side pruning of `*-nightly.*` image tags (no OCIR delete
parity yet — tracked TODO in `retention.yaml`).
## Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added nightly publishing (mirror-by-digest, disk build, e2e
validation) and nightly retention pruning.
* Added RC-to-stable promotion via digest retagging (no rebuild).
* Operator/installer and console now expose version/platform metadata,
including runtime override via `COZYSTACK_VERSION`.
* **Bug Fixes**
* Enforced write-once tag behavior and fast-forward-only maintenance
updates across release workflows.
* Made nightly mirroring/selection and retention pruning more selective
and safer.
* **Tests**
* Added Bats coverage for nightly mirroring and RC retagging, plus new
Helm/unit checks.
* **Documentation**
* Updated the release model to center RC promotion and tag immutability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Bring the security documentation in line with controls that are already
in place, and add the two coordinated-disclosure items flagged during the
incubation due-diligence review.
- self-assessment: DCO is enforced by an automated check; private intake
is GitHub Private Vulnerability Reporting; OpenSSF passing badge achieved
2026-04-01 (working toward silver); GHSA/PVR framing for advisory history;
incident-response now summarized publicly.
- SECURITY.md: route reports to Private Vulnerability Reporting with a
MAINTAINERS.md fallback; add a 90-day coordinated-disclosure window and a
CVSS-based remediation-target table.
- CODEOWNERS: document that it encodes the area-scoped Reviewer role, which
is intentionally distinct from the MAINTAINERS.md governance roster, and
that security reporting is not routed through this file.
- Add docs/security/incident-response.md as the public incident runbook.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: tym83 <6355522@gmail.com>
Add the CNCF security self-assessment required for the Incubation due
diligence review, alongside the existing design-level threat model. It
documents the trust boundaries, actors/actions, security functions,
secure development practices, and known gaps.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: tym83 <6355522@gmail.com>
Document the platform's actors, boundary-crossing actions, and security
goals/non-goals, organized by trust boundary (T0-T4) per the CNCF
TAG-Security Actors/Actions model. Records the primary trust-boundary
invariant (tenant RBAC grants no HelmRelease/core writes; the aggregated
API server fixes the chart reference server-side) and the two admission
enforcement points (core kube-apiserver vs cozystack-api). Every claim
cites a source file so the model can be re-verified as the code evolves.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Promotion mutated the registry at dispatch time — retagging ~37 images to
vX.Y.Z (+:latest), publishing the stable cozy-installer chart — before the
promote PR's e2e ran and before any merge. An abandoned promotion then left
stable-named bytes for a version that never shipped, and the (correct) image
write-once check wedged any later re-promotion to the same version.
Move every irreversible registry side effect into pull-requests-release.yaml
finalize, which runs only after the promote PR merged (= after its e2e passed),
making promotion atomic. promote-rc.yaml now only stages the release-X.Y.Z
branch, drafts the stable release, and opens the PR; an abandoned promotion
leaves nothing but an overwritable draft + branch.
Folded into the same restructure:
- Cut the api/apps/v1alpha1/vX.Y.Z Go-module tag in finalize, at the stable
tag's commit. It was gated in tags.yaml on `release_exists==false &&
is_stable`, unsatisfiable under promotion (the draft always pre-exists), so
promoted stables never got their submodule tag — a regression for Go
consumers. Finalize is where the stable tag is born, so it cuts the matching
submodule tag write-once (stable only).
- Gate :latest by max-semver. promote-retag.sh now moves :latest only when
MOVE_LATEST=1 (default off); finalize passes it the same make_latest decision
the release gets, so a patch on an older line moves neither the release's
`latest` nor the images' :latest. Added MOVE_LATEST bats coverage.
- Stamp platformVersion into the published stable chart's default values
(uncommitted, at package time) so `helm --install --version X.Y.Z` reports the
stable version in telemetry instead of the rc version baked into the operator
image. Not committed to the branch — that would leak platformVersion=vX.Y.Z
onto main and mis-stamp the next rc.
- promote-rc: serialize all promotions (static concurrency group, not per-rc);
label the PR full-e2e so the full suite runs; scope the rc->stable sed to the
depth-2 values.yaml glob plus the kubernetes .tag files (the build's own write
set); tolerate a leftover draft so re-promotion is not wedged.
- finalize: assert the PR author is cozystack-ci[bot], and run on an ephemeral
oracle-vm runner (needs skopeo/helm).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Prior wording overstated Phase 1: the rendered
AuthenticationConfiguration carries no claimValidationRules, so the
chart itself does not gate on email_verified. Rewrite the failure-mode
bullet to describe the layered guarantees operators actually rely on:
1. provisioning KeycloakRealmUsers with emailVerified: true,
2. Keycloak's realm-level duplicateEmails: false as an
impersonation-through-second-account guard,
3. the k8s upstream side-effect that rejects tokens explicitly
carrying email_verified: false.
Also flag CEL claimValidationRules as a reasonable follow-up hardening
path, but explicitly out of scope for Phase 1 per the design doc.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
The tenant kube-apiserver auto-applies
`claims.?email_verified.orValue(true) == true` when username.claim is
email, so a Keycloak user with `emailVerified: false` fails login with
`oidc: email not verified` even though token issuance succeeds.
Document the requirement in the operator guide alongside the other
System-mode failure modes.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Adversarial review flag: the previous rewording implied the cozy realm
has `email` in its client-scope defaults, which is misleading — the
per-cluster client my chart creates lists only `groups` and the
per-cluster audience scope as defaults. The `email` claim reaches the
token because the chart-generated `kubectl oidc-login` block passes
`--oidc-extra-scope=email` explicitly, not because of realm-side
default-scope hygiene.
State the actual mechanic: `email` is a built-in OIDC scope available
in every conformant issuer, requested by the client at auth time via
the kubeconfig's exec block. No realm-side configuration required.
For `CustomConfig`, note that the operator should verify their BYO
issuer emits `email` when the client requests the `email` scope
(every conformant provider does).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Two follow-ups from the final adversarial review:
- The chart-generated OIDC kubeconfig exec block requested only
`--oidc-extra-scope=groups`. On a Keycloak realm where `email` is
not a default client scope, kubectl-oidc-login would fetch a token
that lacks the `email` claim — apiserver would then fail to
identify the user, causing a silent auth failure. Add
`--oidc-extra-scope=email` explicitly so the flow does not depend
on realm-level default-scope hygiene.
- `docs/oidc-tenant.md` still said the `User:` subject "must match
the `preferred_username` claim emitted by the issuer" — obsolete
after the claim switch. Rewrite to say `email`, and add a note
that the chart requests the `email` scope explicitly.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fixes for two factual errors:
- The "How a user logs in" example named the Secret
`prod-oidc-kubeconfig`, but the cozystack-api prefixes the
cluster name with `kubernetes-` when materialising the
`HelmRelease`, so the actual Secret is
`kubernetes-prod-oidc-kubeconfig`. The NOTES.txt template was
already correct; only the doc example was off.
- The CRB hash formula was described as `<release>:<email>` but
the code uses `<release>-<email>` as the SHA input.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Add the operator-facing guide for the new selector — what the three
modes mean, what the chart provisions, how a user logs in, what the
failure modes are, and what is intentionally out of scope (per-tenant
realms, federation, cross-cluster SSO). The architectural rationale
points back to cozystack/community#24.
Update the chart NOTES.txt to print the OIDC kubeconfig retrieval
snippet, the per-cluster client/issuer values, and the listed
`users[]` roster — only when OIDC is on, otherwise the previous
admin-kubeconfig hint is unchanged.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
nightly.yaml publishes each build under 0.0.0-nightly.<date>.<run-id>.<attempt>
(the run-id + attempt suffix makes every nightly tag immutable), but the docs
still showed the old date-only 0.0.0-nightly.<YYYYMMDD> form in the mirror-stage
description and both the helm-install and oras-pull pin examples. Update them to
the full form; the floating `nightly` example is unchanged.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
"What CI does on a tag push" only described one of the four triggers;
promote-rc.yaml uses manual dispatch and pull-requests-release.yaml fires
on PR merge. Rename to "What CI does during the release process" so
maintainers find the right entry point regardless of which workflow they
are debugging.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on docs/release.md: the Regular and
Patch sections still told maintainers to push vX.Y.Z to start the build, while
the rest of the doc said the stable tag is created write-once at the merge
commit and stable comes only from rc promotion (with a 'can also push directly'
escape hatch). Rewrite both sections to the single promotion model — push an rc,
promote it via promote-rc.yaml, merge the promote PR, finalize creates the
write-once tag — drop the direct-push escape hatch, and add promote-rc.yaml to
the workflow list.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replace the nightly-rc-tag description with the new model: a nightly is
an installable copy of main on GHCR (mirror OCIR->GHCR by digest +
re-published packages artifact, cozy-installer chart and nocloud disk),
validated by the full e2e suite, with no GitHub release and no Go-module
tag. Document the helm install path and the dated/floating tag scheme,
and update the Tag-immutability section and workflow link list.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The release doc still described the removed force-retag flow and auto-release.yaml
cron. Update it to the model this PR ships: stable tags are created write-once at
the PR merge commit (never force-moved), the maintenance branch is
fast-forward-only, stable patches come only from rc promotion (nightly-rc.yaml +
promote-rc.yaml + retention.yaml), and the api/apps/v1alpha1 Go-module tag is cut
only for stable. Replaces the "Automated patch tag" and "Force-retagging"
sections (the latter becomes "Tag immutability") and refreshes the See-also list.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Add a guide for bridging a vm-instance VM onto a physically-routed VLAN
via a Linux bridge and a bridge-type NetworkAttachmentDefinition. The
overlay-only default does not cover the case where a VM must share a
broadcast domain with external hardware and take an address from that
VLAN's subnet.
Explain why macvlan is incompatible with KubeVirt bridge binding (macvlan
demultiplexes ingress by the child MAC while bridge binding puts the
guest MAC on the wire, so replies are dropped) and why a Linux bridge
works, then walk through the host bridge, the NAD, and assigning a static
guest address through cloud-init. Cover the dual-homed NIC layout, the
absence of networkData, and the post-recreate ARP staleness.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Two post-vendor cleanups from the #2963 review:
- **Pin the console image builder to pnpm 11.1.2.** The `Containerfile`
ran `corepack prepare pnpm@latest`, so the container build could drift
to a newer pnpm major than the `pnpm@11.1.2` pinned in `package.json`
and used by CI. `--frozen-lockfile` guards the lockfile, but pinning the
builder makes the image build reproducible and consistent with CI.
- **Repoint a stale doc link.** `docs/storage-immutability.md`
referenced the now-vendored `cozystack/cozystack-ui#6`; it now points at
the in-tree console path.
## Verification
- No source change; `pnpm typecheck` + `pnpm test` unaffected.
- Builder pnpm version now matches `package.json`'s `packageManager`
field.
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Clarified the storage immutability documentation by updating the UI
reference to an in-repo path.
* **Chores**
* Standardized the build environment to use a specific pnpm version,
helping keep installs and builds consistent.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The console Containerfile ran `corepack prepare pnpm@latest`, so the image build could drift to a newer pnpm major than the `pnpm@11.1.2` pinned in package.json and used by CI. `--frozen-lockfile` guards the lockfile, but pinning the builder makes the image build reproducible and consistent with CI.
Also repoints docs/storage-immutability.md off the now-vendored cozystack-ui#6 link to the in-tree console path (follow-up from #2963).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- scope create-github-app-token to the current repo with read-only
contents permission (was org-wide, inheriting the app's blanket
installation permissions); both token steps only read releases/assets
- fail-fast the Talos image upload when the disk is missing
(if-no-files-found: error) so drift fails in build-talos, not later
as a confusing e2e download error
- download the flux install script to a tempfile before executing it,
so a failed curl is not masked by the piped `sudo bash` exiting 0
- fix the regular-PR artifact-source comment (Talos disk comes from
build-talos, pr.patch from finalize) and the TIA scope wording in
docs/agents/e2e-testing.md
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.
Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.
The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Rewrite docs/agents/e2e-testing.md for the Chainsaw suite: declarative
asserts replace the until-get/wait backstops, Chainsaw cleanup replaces
test-level EXIT traps, catch blocks replace the `|| { ...; false; }` idiom,
and the TIA/diagnostics sections describe suites + `make test-chainsaw
CHAINSAW_SUITES`. Move the BATS->Chainsaw bullet out of 'in-flight' since it
is now the standard.
Fix references the migration left stale: hack/e2e-apps/ -> hack/e2e-chainsaw/
in the gemini styleguide and the nats/etcd backup READMEs. The
backup-postgres.bats / backup-foundationdb.bats files cited by the postgres
examples, the postgres README, and the foundationdb controller comment never
existed -- drop the false 'CI e2e drives this' claims and describe those
backup flows as the manual reference flows they are.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
