Add a node-local Vector DaemonSet that receives Talos machine and kernel
logs (pushed via machine.logging over the node loopback) and forwards them
to the tenant VictoriaLogs. Talos keeps its system and kernel logs inside
the OS and does not expose them as host files, so the tail-based Fluent Bit
in monitoring-agents never collects them.
The receiver stays on the pod network so cluster DNS resolves the vlinsert
service, and accepts Talos logs via a hostPort bound to hostIP 127.0.0.1,
avoiding the host-network DNS breakage a hostNetwork receiver would hit.
Shipped as an optional system package.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux
6.18.24. That kernel predates the fixes for two related KVM x86
shadow-paging use-after-free bugs in `arch/x86/kvm/mmu/mmu.c`:
CVE-2026-46113, fixed upstream in
`0cb2af2ea66ad8ff195c156ea690f11216285bdf` and released in 6.18.30, and
CVE-2026-53359, fixed in `81ccda30b4e83d8f5cc4fd50503c44e3a33abfeb` and
released in 6.18.38. A worker's own kernel runs KVM whenever a tenant
enables nested virtualization inside the VM, and either bug lets a
nested guest escape into the worker.
This bumps the worker pin to Talos v1.13.6, which ships Linux 6.18.38.
Stable kernel branches are cumulative, so v1.13.6 is the first v1.13
patch release carrying both fixes. The pinned v1.13.0 ships 6.18.24 and
carries neither; v1.13.5 ships 6.18.36 and carries only CVE-2026-46113.
Nested virtualization stays available to tenants; the kernel bump is the
fix, not a capability removal.
The image-factory schematic is unchanged and resolves for the new
version: both the `openstack-amd64.raw.xz` artifact and the
`installer/<schematicID>:v1.13.6` manifest are served by the public
factory. Talos v1.13 supports Kubernetes 1.31 through 1.36, so the
chart's Talos-to-Kubernetes support window is unaffected and no tenant
`spec.version` needs to move.
Applying this rolls each worker pool once. The new pin changes the
rendered worker spec, which rotates the content hash in the
`KubevirtMachineTemplate` and `talos-reconcile` Job names, and CAPI
replaces the worker VMs to deliver the new kernel. The pinned hashes in
the chart tests are recomputed to match.
A second commit fixes the support-matrix lookup itself. The matrix dict
was keyed on the full patch version, so every Talos patch bump dropped
out of the dict and silently disabled the render-time guard — exactly
the mismatch the guard exists to catch. The lookup now derives the minor
from `talos.version`. The supported-Kubernetes window is unchanged;
unknown minors are still skipped, leaving compatibility to whoever
overrode the pin.
### Screenshots
Not applicable — no UI changes.
### Release note
```release-note
fix(kubernetes): bump tenant worker Talos to v1.13.6, picking up the Linux 6.18.38 kernel fixes for CVE-2026-46113 and CVE-2026-53359 (KVM shadow-paging use-after-free, nested-guest escape into the worker VM). Nested virtualization remains available. Applying this upgrade rolls each tenant worker pool once, as the worker template is renamed to carry the new image.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Updated the default Talos worker release to v1.13.6 for Kubernetes
cluster provisioning.
- Improved compatibility checks to apply consistently across Talos patch
releases within the same minor version.
- **Bug Fixes**
- Corrected generated resource naming expectations when storage settings
or GPU configuration change.
- **Tests**
- Added coverage confirming Kubernetes clusters render successfully with
Talos v1.13.6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Bumps the platform Talos profiles from v1.13.5 to v1.13.6.
CVE-2026-53359 and CVE-2026-46113 are related KVM x86 shadow-paging
use-after-free bugs that let a guest escape to the host, and the fix is
complete only once both upstream patches (`81ccda30b4e8`,
`0cb2af2ea66a`) are applied. On the 6.18 stable line CVE-2026-46113
landed in 6.18.30 and CVE-2026-53359 in 6.18.38. Talos v1.13.5 runs
Linux 6.18.36, so it carries only the first of the two. Talos v1.13.6
runs Linux 6.18.38 and carries both.
Nested virtualization stays enabled. The bugs are fixed in the kernel,
so there is no reason to take the feature away from workloads that rely
on it — bumping the kernel is the complete fix.
Profiles are regenerated with `hack/gen-profiles.sh v1.13.6`. Alongside
the Talos version and the installer reference, this refreshes the pinned
system-extension digests: the firmware extensions move from the 20260519
to the 20260622 build, and `intel-ucode` keeps its `20260512` tag but
resolves to a new digest because upstream re-pushed that tag as an OCI
image index (identical config and layer blobs underneath).
References: [Talos v1.13.6 release
notes](https://github.com/siderolabs/talos/releases/tag/v1.13.6)
(`Linux: 6.18.38`)
### Screenshots
Not applicable, no UI changes.
### Release note
```release-note
fix(talos): bump Talos Linux to v1.13.6, which ships Linux 6.18.38 with the kernel fixes for the CVE-2026-53359 and CVE-2026-46113 KVM guest-to-host escapes
```
flux-shard-operator was the last image built with the pre-#2937 cache
pattern: --cache-from a :latest tag that PR builds no longer publish and
--cache-to type=inline, which never caches multistage builder layers. Use
the shared cache-args macro so it reads (and, on main, writes) the same
mode=max :buildcache ref in CACHE_REGISTRY as every other image.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Talos v1.13.5 ships Linux 6.18.36, which carries the fix for
CVE-2026-46113 but not the one for CVE-2026-53359. The two are related
KVM x86 shadow-paging bugs allowing a guest to escape to the host, and
both patches are needed for the mitigation to hold. Talos v1.13.6 ships
Linux 6.18.38, the first release on the 6.18 stable line that has both.
Profiles regenerated with hack/gen-profiles.sh v1.13.6. Alongside the
Talos version and the installer reference, this refreshes the pinned
system-extension digests: the firmware extensions move from the 20260519
to the 20260622 build, and intel-ucode stays on its 20260512 tag but
resolves to a new digest because upstream re-pushed that tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux 6.18.24.
That kernel predates both fixes for a pair of related KVM x86 shadow-paging
use-after-free bugs: CVE-2026-46113, fixed in 6.18.30, and CVE-2026-53359,
fixed in 6.18.38. A worker's own kernel runs KVM whenever a tenant enables
nested virtualization inside the VM, and either bug lets a nested guest
escape into the worker.
Talos v1.13.6 ships Linux 6.18.38. Stable kernel branches are cumulative, so
it is the first v1.13 patch release carrying both fixes: the pinned v1.13.0
ships 6.18.24 and carries neither, while v1.13.5 ships 6.18.36 and carries
only CVE-2026-46113. Nested virtualization stays available to tenants — the
kernel bump is the fix.
The new pin changes the rendered worker spec, so the content hash in the
KubevirtMachineTemplate and talos-reconcile Job names rotates and each
worker pool rolls once on upgrade. Pinned hashes in the chart tests are
recomputed accordingly.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Talos<->Kubernetes support window is declared per Talos minor release,
and patch releases never move it. Keying the chart's matrix dict on the full
patch version meant every Talos patch bump fell out of the dict and silently
disabled the render-time guard — precisely the mismatch the guard exists to
catch.
Derive the minor from talos.version before the lookup, so the guard keeps
firing across patch releases of a known minor. Unknown minors are still
skipped, leaving compatibility to the operator who overrode the pin.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Addresses #3181, where platform log ingestion could fail **silently**:
fluent-bit shipped to `vlinsert-generic` while the monitoring
`HelmRelease` reported `Ready`, and a `vlinsert-generic` `ExternalName`
in `cozy-monitoring` dead-ended at a `tenant-root` VictoriaLogs that did
not exist when the root tenant had monitoring disabled. Logs were
dropped with no failing signal.
Three complementary layers make the failure loud instead of silent:
1. **Readiness gate** —
`packages/extra/monitoring/templates/helmrelease.yaml` now sets a
`poller` `waitStrategy` and a `healthCheckExprs` entry on the
`VLCluster` `status.updateStatus`. A missing/unhealthy log store makes
the `monitoring-system` release **not-ready** instead of flipping
`Ready` the moment the CR is applied. Mirrors the existing seaweedfs
CNPG pattern.
2. **No empty-log-store hole** —
`packages/system/monitoring/templates/vlogs/vlogs.yaml` fails the render
on an empty `logsStorages`. With no entry no `VLCluster` exists, the
readiness gate has nothing to match, and fluent-bit would silently drop
logs — so this is now a loud misconfiguration.
3. **No silent fallback** — the `cozy-monitoring → tenant-root`
`vlinsert`/`vminsert` `ExternalName` redirects in `cozystack-basics` are
created only when the root tenant hosts monitoring. The template reads
the live `Tenant/root` `spec.monitoring` via `lookup` (source of truth,
so it cannot drift from the tenant's real state) and falls back to a new
platform value `monitoring.rootEnabled` (via
`_cluster.monitoring-enabled`) when `lookup` is empty (dry-run / first
render). The default preserves the historical always-create behaviour.
Covered by helm-unittest across all four affected charts (141 tests):
readiness-gate fields, empty-vs-populated `logsStorages` contract,
`ExternalName` present/absent fallback contract, and the platform wiring
into the `cozystack-values` Secret. Also adds the missing `test:` target
to `packages/extra/monitoring/Makefile` so its suite runs in CI.
**Known limitation / follow-up:** the legacy platform monitoring release
installed directly into `cozy-monitoring` via the `cozystack.monitoring`
PackageSource cannot carry the readiness gate, because the engine's
`ComponentInstall`/HelmRelease builder and the vendored
`helm-controller` API v1.4.3 lack `waitStrategy`/`healthCheckExprs`.
That source is not referenced by any active bundle; full coverage there
needs a helm-controller bump plus engine support.
### Release note
```release-note
fix(monitoring): surface missing platform VictoriaLogs as not-ready and stop the dangling vlinsert-generic ExternalName from silently dropping logs (#3181)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a platform setting (`monitoring.rootEnabled`) to control whether
root-level monitoring is enabled.
* Monitoring redirect ExternalName services are now rendered only when
root monitoring is enabled (defaulting to enabled).
* **Bug Fixes**
* Ensures an explicitly set `monitoring.rootEnabled=false` is preserved
when generating platform configuration.
* Prevents incomplete monitoring deployments by failing rendering when
`logsStorages` is empty.
* Improves VictoriaMetrics readiness by delaying “ready” until the
operator reports an operational status.
* **Tests**
* Added/expanded Helm-unittest suites covering monitoring wiring,
redirects, readiness gating, and `logsStorages` requirements.
* Added a `make test` target to run chart unittests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Match the full rendered worker-disk URL shape (version + artifact + end anchor) like the sibling override assertion, so the trailing-slash normalization case cannot pass on a partial match.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
When the `monitoring` tenant module is disabled, the
VictoriaMetrics/VictoriaLogs storage PVCs (created by vm-operator) and
the `alerta-tls` / `grafana-ingress-tls` cert-manager secrets were left
orphaned in the tenant namespace.
This adds a `post-delete` cleanup hook
(`packages/extra/monitoring/templates/hooks/cleanup.yaml`) that:
- deletes the VM/VL storage PVCs scoped to this release via `-l
apps.cozystack.io/application.name={{ .Release.Name }}-system`. That
label is stamped on the `storage.volumeClaimTemplate.metadata.labels` of
the VMCluster/VLCluster CRs (`packages/system/monitoring`); vm-operator
copies claim-template labels onto the PVCs (`IntoSTSVolume`), so the
selector matches and stays scoped to this monitoring release.
`managed-by=vm-operator` is intentionally **not** used — vm-operator
does not reliably stamp it onto the storage PVCs;
- deletes the `alerta-tls` and `grafana-ingress-tls` secrets by name
(cert-manager TLS secrets carry only the cluster-wide
`controller.cert-manager.io/fao=true` label, so a label sweep would be
unsafe — they are removed by name).
CNPG PostgreSQL PVCs (`alerta-db-*`, `grafana-db-*`) are intentionally
**not** targeted — CNPG cleans those up itself, and they do not carry
`apps.cozystack.io/application.name`, so the selector leaves them
untouched.
**Upgrade path:** the claim-template label only lands on PVCs created
after the change, so **migration 50**
(`packages/core/platform/images/migrations/migrations/50`) backfills
`apps.cozystack.io/application.name` onto the pre-existing VM/VL storage
PVCs of every cozystack-managed monitoring cluster, so the cleanup
reclaims storage on already-deployed clusters too.
A `test` Makefile target and helm-unittest suites are added for both the
cleanup hook and the claim-template label. The cleanup Job is hardened:
non-root, read-only root fs, dropped capabilities, `seccompProfile:
RuntimeDefault`, resource requests/limits, `activeDeadlineSeconds: 120`,
`backoffLimit: 0`, `--wait=false` on both deletes; RBAC scoped to PVC
get/list/delete and the two named secrets. Failures are logged but never
block teardown.
> [!WARNING]
> **Disabling the monitoring module now permanently deletes its
VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they
hold.** This is intentional and fixes the storage leak. Back up before
disabling if the data matters.
Fixes#3091.
### Release note
```release-note
fix(monitoring): garbage-collect the orphaned VictoriaMetrics/VictoriaLogs storage PVCs and the alerta/grafana TLS secrets left behind when a tenant's monitoring module is disabled; a migration backfills the release label onto pre-existing storage PVCs so already-deployed clusters are covered too. WARNING: disabling monitoring now permanently removes its storage PVCs and the data they hold.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a Helm **post-delete** monitoring cleanup job to best-effort
remove managed PVCs and two specific TLS secrets.
* Included dedicated cleanup RBAC and hardened pod security settings for
the cleanup job.
* **Tests**
* Added Helm chart tests validating hook weight/policies, job settings,
exact deletion commands, and negative cases.
* Added a `make test` target to run Helm unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Makes the optional `keycloak-kms-proxy` PII-encryption feature
production-usable.
**Vault auth** — new `encryption.kms.vault.auth` switch:
- `kubernetes` — **recommended**. The proxy logs in with its own
dedicated ServiceAccount (`keycloak-kms-proxy`; bind the Vault role to
`system:serviceaccount:<ns>:keycloak-kms-proxy`), so no credential is
stored in a Secret.
- `approle` — `KKP_VAULT_ROLE_ID` / `KKP_VAULT_SECRET_ID` from
`encryption.kms.vault.appRole.secretName`.
- `token` (default, unchanged) — Vault token from `tokenSecretName`.
The proxy pod runs under a dedicated ServiceAccount (never the namespace
default), with its token mounted only for Kubernetes auth.
**Shared DEK set** — `encryption.deksetSecretName` → `KKP_DEKSET_FILE`
(mounted read-only), and `encryption.replicas` is now configurable.
Without a dekset the proxy mints an ephemeral DEK (only safe on a fresh,
empty database, single replica). A shared wrapped DEK set — minted by
the proxy's backfill tool — is required to migrate an existing populated
Keycloak and to run HA; a guard rejects `replicas > 1` without one.
Render-time validation requires each mode's settings and rejects unknown
values. helm-unittest covers all auth modes, the ServiceAccount/token
toggle, and the dekset/replicas paths. Uses `keycloak-kms-proxy` v0.2.0.
### Release note
```release-note
feat(keycloak): Vault Kubernetes/AppRole auth, a dedicated ServiceAccount, and a shared DEK set (migration + HA) for the optional PII-encryption proxy
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Expanded Keycloak KMS proxy Vault authentication to support **token**,
**AppRole**, and **Kubernetes** modes.
* Added **multi-replica/HA** support via configurable DEK-set reuse,
including selecting a custom DEK-set key.
* Introduced a dedicated proxy ServiceAccount with **conditional token
automount** for Kubernetes auth.
* **Bug Fixes**
* Added stricter validation for missing/unsupported Vault auth settings.
* Prevented Vault credential environment variables from leaking across
auth modes.
* Updated rendering behavior and test expectations for proxy resources
(including KEK/ServiceAccount documents).
* **Chores**
* Updated the Keycloak KMS proxy container image version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Re-review on #3094 (IvanHunters, MAJOR): the volumeClaimTemplate label lands on
PVCs only at creation — the StatefulSet controller never re-labels existing PVCs
and vm-operator's PVC reconcile is resize-only. So on an already-deployed cluster
(exactly the #3091 population) the pre-upgrade VM/VL storage PVCs stay unlabeled
and the cleanup hook's selector misses them.
Add migration 49 (mirrors the ClickHouse-keeper backfill in migration 48 for the
sibling #3072). It relabels the pre-existing vmstorage/vmselect/vlstorage PVCs of
every cozystack-managed VMCluster/VLCluster with apps.cozystack.io/application.name
read from the CR's own spec.managedMetadata, so only monitoring's clusters are
touched (a user's own VMCluster carries no such label) and the value always
matches what the cleanup hook selects. Best-effort like #48: a skipped relabel is
no worse than the pre-existing leak, so transient apiserver errors never abort the
platform upgrade. Bump platform targetVersion 49 -> 50.
Also correct the misleading comments (IvanHunters MINOR): spec.managedMetadata is
never applied to PVCs by vm-operator at any current version, so the claim-template
label is the only mechanism — do not imply a v0.71.0 crossover.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Re-review on #3094 (IvanHunters): the AND selector kept managed-by=vm-operator,
which on the shipped vm-operator v0.68.4 is not reliably present on the storage
PVCs (it lives on the StatefulSet selector/pod labels, not the claim template),
making the delete fragile. Match by apps.cozystack.io/application.name=<release>-system
alone: that label is stamped on the VM/VL claim templates (67b087ab) and copied
onto the PVCs by IntoSTSVolume even on v0.68.4, is already release-scoped, and is
absent from the CNPG db PVCs (grafana-db/alerta-db carry only policy.cozystack.io
+ cnpg.io/*), so those stay untouched. Regression guard now forbids re-adding
managed-by to the selector.
Also apply the review's discipline items: the cleanup Job keeps only
before-hook-creation (a Failed run survives for post-mortem; hook-succeeded stays
on SA/Role/RoleBinding, cf. #3072) and sets backoffLimit: 0 (best-effort, no retry).
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Re-review on #3094 (IvanHunters, lexfrei): the release-scoped selector added
in ba5797f4 was a silent no-op on the shipped vm-operator v0.68.4. The
apps.cozystack.io/application.name label lived only under the VMCluster/VLCluster
spec.managedMetadata.labels, and v0.68.4 does not propagate managedMetadata onto
PVCs (that lands only in v0.71.0). So the delete matched nothing and the #3091
storage leak stayed open.
Stamp the label where it actually reaches the PVCs: on
spec.{vmselect,vmstorage,vlstorage}.storage.volumeClaimTemplate.metadata.labels
of the VMCluster/VLCluster CRs (packages/system/monitoring). vm-operator copies
claim-template labels onto the PVCs (IntoSTSVolume) even on v0.68.4, so the
existing cleanup selector (managed-by=vm-operator,apps.cozystack.io/application.name=
<release>-system) now matches and stays scoped to this release. Verified by
rendering the CRs (label resolves to monitoring-system, matching the hook).
Also add the explicit subjects[].namespace on the cleanup RoleBinding (lexfrei
non-blocking) and a helm-unittest suite asserting the label lands on all three
storage claim templates. Corrected the misleading test comment about how the
label reaches the PVCs.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Address IvanHunters review on #3094: the PVC delete selector was
release-agnostic (-l managed-by=vm-operator), matching the VM storage
PVCs of every VMCluster/VLCluster in the namespace. If a namespace ever
hosted more than one monitoring release, tearing one down would delete
the other's VictoriaMetrics/VictoriaLogs storage.
Scope the selector to this release via apps.cozystack.io/application.name=
<release>-system, the ownership label the monitoring chart stamps on the
VM CRs' managedMetadata and vm-operator propagates onto the PVCs — mirroring
the merged seaweedfs hook's release scoping. deletecollection is still not
needed: kubectl delete -l lists then deletes individually.
Tests: assert the release-scoped selector and add a regression guard that
the delete never falls back to the bare managed-by=vm-operator selector.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Address review on #3094:
- Add --wait=false to both kubectl delete calls so PVC finalizers
(pvc-protection on pods still terminating) can never block the
post-delete hook and stall the HelmRelease teardown.
- Add activeDeadlineSeconds: 120 on the cleanup Job as a backstop.
- Tests: assert the non-blocking flag, plus negative assertions that the
PVC delete stays label-scoped (no --all) and the secret delete stays
name-scoped (no -l/--all).
deletecollection RBAC verb intentionally not added: kubectl delete -l
lists then deletes individually, so the existing delete verb suffices.
BREAKING CHANGE: disabling monitoring now permanently removes its
VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they hold.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
When the monitoring tenant module is disabled, the vm-operator storage PVCs
(VictoriaMetrics/VictoriaLogs) and the alerta/grafana TLS secrets were left
orphaned. Add a post-delete cleanup hook that deletes the PVCs by the
managed-by=vm-operator label and the two TLS secrets by name, plus a test
target and helm-unittest suite (the package had neither). CNPG PVCs are not
targeted (CNPG cleans them). The cleanup Job is hardened (non-root, read-only
root fs, dropped caps, resource limits); failures are logged, never blocking
teardown.
NOTE: disabling monitoring now permanently removes its storage PVCs and the
data they hold.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Platform log ingestion could fail silently: fluent-bit shipped to
vlinsert-generic while the monitoring HelmRelease reported Ready, and a
vlinsert-generic ExternalName in cozy-monitoring dead-ended at a
tenant-root VictoriaLogs that did not exist when the root tenant had
monitoring disabled. Logs were dropped with no failing signal.
Three complementary changes:
- Readiness (do not stay silently Ready): the monitoring-system
HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on
its VLCluster (status.updateStatus), so a missing/unhealthy log store
makes the release not-ready instead of flipping Ready as soon as the
CR is applied. Mirrors the seaweedfs CNPG pattern.
- No empty-log-store hole: system/monitoring rejects an empty
logsStorages at render time (helm fail). With no entry no VLCluster is
created, the readiness gate has nothing to match, and fluent-bit would
silently drop logs — so this is made a loud misconfiguration rather
than a documented footgun.
- No silent fallback: the cozy-monitoring -> tenant-root
vlinsert/vminsert ExternalName redirects in cozystack-basics are now
created only when the root tenant hosts monitoring, gated on
_cluster.monitoring-enabled — fed from a new platform value
monitoring.rootEnabled (default true, preserving the historical
always-create behaviour). monitoring.rootEnabled is the single
render-time source of truth: changing it re-renders cozystack-basics
(helm-controller watches the cozystack-values Secret it flows into). A
live `lookup` of Tenant/root spec.monitoring is deliberately NOT used:
cozystack-basics is an operator-generated HelmRelease with drift
detection off, so it re-renders only on install/upgrade/values-change
— never on a reconcile tick or when a lookup result changes — and a
lookup would go stale in both directions. Operators must keep
monitoring.rootEnabled in sync with Tenant/root spec.monitoring.
Covered by helm-unittest across all four charts: readiness gate fields,
empty-vs-populated logsStorages contract, ExternalName present/absent
and null-guard contract, and the platform wiring into the
cozystack-values Secret. The CEL health predicates are exercised live,
not in helm-unittest (which cannot evaluate CEL). Adds the missing
`test:` target to packages/extra/monitoring/Makefile so its suite runs
in CI.
Known limitation (follow-up): the legacy cozystack.monitoring
PackageSource that installs system/monitoring directly into
cozy-monitoring cannot carry the readiness gate, because the engine's
ComponentInstall/HelmRelease builder and the vendored helm-controller
API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not
referenced by any active bundle; full coverage needs a helm-controller
bump plus engine support.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Refs #3231 (parts 1 + 2 addressed here; part 3 — intra-file
`cozy_wait_linstor_pool_free` gate + orphan
`default-chk-clickhouse-test-keeper-cluster1-0-2-0` PVC cleanup —
deferred to a separate follow-up PR against
`hack/e2e-apps/run-kubernetes.sh`).
**Do not auto-close the issue on merge** — the hardening work is a
distinct concern and belongs on its own tracker entry.
## Root cause chain (recap from the issue)
`packages/apps/kubernetes/templates/cluster.yaml` hardcoded
`MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's
`minReplicas`. The two render-side
`kubernetes-oidc-{system,customconfig}` bats set `minReplicas: 0` and
delete their `Kubernetes` CR without waiting for readiness — but the
chart still spun up 2 × 20 GiB `replicated` (DRBD) worker `DataVolume`s
each. Their create/delete churn overlapped `kubernetes-previous`, which
then saw slow LINSTOR provisioning (my cozyreport from run 28843794670
shows a ~6-minute gap between the `Kubernetes` CR being created and the
CDI `ExternalProvisioning` event actually firing for the worker root
disk); the tenant node registered late; Cilium's CNI init missed the
12-minute node-join deadline; and the test failed with `cni plugin not
initialized`.
## Changes
Two edits, applied together — the chart edit is a no-op without the test
edit, and vice-versa (dropping `nodeGroups` falls back to the chart
helper's default `md0`, which itself has `minReplicas: 0`, so both
pieces are needed for zero workers to actually render).
### 1. `packages/apps/kubernetes/templates/cluster.yaml`
```diff
spec:
clusterName: {{ $.Release.Name }}
- replicas: 2
```
The field is **omitted entirely** so the cluster-autoscaler owns it
alone. CAPI's MachineDeployment defaulting webhook seeds `spec.replicas`
from the `cluster-api-autoscaler-node-group-min-size` annotation on
Create (documented at
https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling
— *"if it's a new MachineDeployment or MachineSet, use min size"*), and
Server-Side-Apply field-manager ownership then keeps chart upgrades from
overwriting whatever value the autoscaler patched in the meantime.
Emitting the field from the chart at all — even with the
semantically-correct `{{ $group.minReplicas }}` — would keep the
helm-controller ↔ cluster-autoscaler field-ownership fight alive:
helm-controller applies with SSA + force-conflicts
(https://fluxcd.io/flux/components/helm/helmreleases/), so every routine
`helm upgrade` re-asserts the chart's chosen value and drains whatever
the autoscaler had patched in the meantime.
### 2. `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats`
Drop the explicit `nodeGroups:` block and the paired `storageClass:`
from the render-side CR bodies. The tests only assert on the
KamajiControlPlane and the cozy-realm Keycloak objects — worker plumbing
was never part of their coverage.
After the chart fix, the chart helper's default `md0` (with
`minReplicas: 0`) renders a MachineDeployment with `spec.replicas: 0` —
the resource is still emitted for API-surface completeness but never
creates a KubevirtMachine, never provisions a DataVolume, and never
touches DRBD.
## Verification
### helm-unittest
```
Charts: 1 passed, 1 total
Test Suites: 16 passed, 16 total
Tests: 171 passed, 171 total
```
No suite asserted on `MachineDeployment.spec.replicas` directly, so
nothing needed updating.
### helm template — verify the omit
With `tests/values/common.yaml` (which has `minReplicas: 0`), the
rendered MD carries the `cluster-api-autoscaler-node-group-min-size:
"0"` annotation and **no `spec.replicas` field**. CAPI's defaulting
webhook will fill in `spec.replicas: 0` on Create — no worker
DataVolumes provisioned until scale-out.
With a user CR that sets `minReplicas: 2` (the shape the
`kubernetes-latest` / `kubernetes-previous` bats CRs use), the rendered
MD carries the `cluster-api-autoscaler-node-group-min-size: "2"`
annotation and again **no `spec.replicas` field**. CAPI's defaulting
webhook will fill in `spec.replicas: 2` on Create — byte-identical
topology to the pre-PR chart's fresh-install state (2 workers), but
without the recurring drain-on-upgrade cycle.
### Render-side render walk
`helm template` of the OIDC bats CR bodies (after this PR): 1
MachineDeployment with **no chart-managed `spec.replicas`** and
`min-size: "0"` on the autoscaler annotation. CAPI seeds `replicas: 0`
on Create, 0 KubevirtMachines emitted, 0 DataVolume provisioning
triggered. Test assertions on KamajiControlPlane / KeycloakClient /
KeycloakClientScope / AuthenticationConfiguration Secret are unchanged.
## Upgrade impact
Existing customers stop losing workers on `helm upgrade`. Prior to this
PR the chart re-asserted `MachineDeployment.spec.replicas: 2` on every
Helm action, so a customer running with the autoscaler at, say,
`replicas: 8` would see 6 workers drained on every routine Cozystack
platform bump and then wait for the autoscaler to scale back up in
response to unschedulable Pods. With `spec.replicas` no longer emitted
by the chart, SSA field-manager ownership preserves the autoscaler's
value across upgrades unconditionally.
Fresh installs with `nodeGroups: {}` (the schema default) transition
from "2 workers by default" to "0 workers until scale-out". The chart
helper's default md0 carries `minReplicas: 0`, which is what CAPI's
defaulting webhook seeds `spec.replicas` from. The `values.yaml` /
README / helper docstring / regenerated Go type doc + openAPISchema call
this out explicitly; the release-note repeats it below. See finding 5 in
the paired self-review for the specific `addons.ingressNginx.enabled:
true` interaction operators need to be aware of on fresh installs.
## Test plan
- [x] `helm unittest packages/apps/kubernetes/` — 171/171 passing.
- [x] `helm template` with `tests/values/common.yaml` (minReplicas: 0) —
MD.replicas: 0.
- [x] `helm template` with a `minReplicas: 2` override — MD.replicas: 2
(byte-identical to pre-PR).
- [ ] E2E: `kubernetes-latest` unchanged (spawns its two workers as
before), `kubernetes-oidc-{system,customconfig}` render zero workers,
`kubernetes-previous` no longer flakes on LINSTOR pool pressure
originating from the earlier OIDC suite.
## Notes
- Part 3 of #3231 (intra-file `cozy_wait_linstor_pool_free` gate between
kube sub-tests and the orphan clickhouse keeper PVC cleanup) is **not**
in this PR — worth a separate targeted change to
`hack/e2e-apps/run-kubernetes.sh`, and it's a hardening-only improvement
rather than a root-cause fix. Happy to send it as a follow-up if wanted.
## Release note
```release-note
fix(apps/kubernetes): stop the chart from managing `MachineDeployment.spec.replicas`; the cluster-autoscaler owns the field alone. CAPI's defaulting webhook now seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create, and SSA field-manager ownership keeps subsequent `helm upgrade` actions from overwriting whatever value the autoscaler patched. Two behavioural changes worth calling out:
- Existing customers: `helm upgrade` no longer drains workers on every platform bump — the autoscaler-held `spec.replicas` is preserved unconditionally. Prior to this change the chart hardcoded `replicas: 2`, which drained back to 2 on every reconcile action.
- Fresh installs with `nodeGroups: {}` (the schema default) no longer come up with two workers by default. The chart helper's default md0 carries `minReplicas: 0`, and CAPI's defaulting webhook seeds `spec.replicas` from that. Operators enabling `addons.ingressNginx.enabled: true` on such a CR must either supply an explicit nodeGroup with `roles: [ingress-nginx]` and `minReplicas >= 1`, or accept that the cluster-autoscaler will bring the default md0 up in response to the ingress-nginx controller Pods becoming Pending on install.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Rendered `MachineDeployment` manifests no longer set `spec.replicas`,
avoiding chart upgrades overwriting autoscaler-managed sizing.
* **Documentation**
* Clarified `nodeGroups: {}` behavior: a default `md0` is emitted with
`minReplicas: 0`/`ingress-nginx` role, and workers are provisioned only
after an autoscaler-triggering unschedulable Pod or manual scaling;
updated related storage and ingress guidance.
* **Tests**
* Strengthened default node group rendering assertions and adjusted OIDC
e2e manifests to reduce CI provisioning churn.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Tenant worker VMs boot from a Talos raw disk image that CDI streams over
HTTP from the hardcoded public Talos Image Factory, and the in-guest
talos-reconcile Job pulls its installer image from the same hardcoded host.
Neither was overridable, so air-gapped / rate-limited / flaky-egress
environments could not point workers at a mirror (the README flagged this
as a Phase-2 gap), and CI had no way to avoid bulk-pulling the OS image
from the public internet per worker.
Add two chart values, both defaulting to the public factory so rendered
output is byte-identical and behaviour is unchanged:
- talos.imageFactoryURL (default https://factory.talos.dev) - base URL for
the worker OS disk image streamed in by CDI.
- talos.installerRepository (default factory.talos.dev/installer) - OCI
repository prefix for the in-guest installer image.
Regenerated values.schema.json, the aggregated-API Go types, the
kubernetes-rd cozyrds schema and the README via `make generate`; added
unittests for the override and trailing-slash trimming.
Refs #3231
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The release workflow publishes the OCI tag as 0.2.0 (docker/metadata-action
strips the leading v from the git tag), so the chart default of v0.2.0 does
not exist and pulls fail. Pin the real tag.
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Assert KKP_DEKSET_FILE honors a non-default deksetSecretKey, closing the
last uncovered branch of the dekset wiring.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Wire encryption.deksetSecretName -> KKP_DEKSET_FILE (mounted read-only) and
make encryption.replicas configurable. Without a dekset the proxy mints an
ephemeral DEK, which only works on a fresh database and a single replica.
A shared wrapped DEK set is required to migrate an existing populated
Keycloak (the proxy must reuse the DEK the backfill wrapped rows with) and
to run more than one replica; a guard rejects replicas>1 without one.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
v0.2.0 is the first release carrying the Kubernetes/AppRole auth modes
this chart wires, so encryption.kms.vault.auth is now usable with the
default image.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Lock in the SA-object token toggle (off by default, on only for kubernetes
auth), complementing the pod-level assertions.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Address review: the proxy pod ran under the namespace default ServiceAccount,
which the keycloak StatefulSet also uses. Binding a Vault Kubernetes-auth role
to default would hand the keycloak pod itself Transit access, defeating the
isolation the proxy exists to provide. Add a dedicated keycloak-kms-proxy
ServiceAccount, run the proxy pod under it, and mount its token only in the
kubernetes auth mode (off otherwise, to minimise the credential surface).
Docs name the SA to bind the Vault role to.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Extend encryption.kms.vault.auth with "kubernetes" (recommended): the
proxy logs in with its own ServiceAccount token, so no credential is
stored in a Secret. "approle" and "token" remain. Render-time
validation requires kubernetes.role and rejects unknown methods;
helm-unittest covers the new path and the mutual absence of the other
modes' env vars.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
sircthulhu's post-approval note on PR#3233: both the cluster.yaml
comment and the nodegroups_default_test.yaml comment attribute the
upgrade-safety guarantee to Server-Side-Apply field-manager
ownership, but this chart ships as a HelmRelease through Flux
helm-controller, which applies via the Helm SDK three-way merge —
not SSA. The stated conclusion is correct but for the wrong reason
and would send a future maintainer chasing a non-existent field-
manager fight when debugging drift.
Rewrite both comments around the two independent CAPI mechanisms
that actually carry the guarantee:
1. Create — the MachineDeployment defaulting webhook seeds
spec.replicas from the min-size autoscaler annotation.
2. Update — calculateMachineDeploymentReplicas preserves the
existing live value whenever the incoming spec has
replicas: nil (clamping into [min, max] on drift).
The chart guarantee reduces to "leave the field out of the rendered
manifest and let CAPI's update-time defaulting keep the live value
stable" — no field-manager ownership involved.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The notContains checks matched whole array elements, so a name-only
content never equaled the valueFrom entry and the assertion could never
fail. Add any:true so "AppRole stores no static token" and, symmetrically,
"token mode renders no AppRole vars" are actually verified (confirmed by
mutation-testing the template).
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Symmetry with the zero-minReplicas neighbour, which already covers
both min-size and max-size. CAPI's calculateMachineDeploymentReplicas
defaults to 1 (not to minSize) if either annotation is missing on a
new MachineDeployment, so both keys are load-bearing on the
Create-path guarantee and worth asserting in both regression cases.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The keycloak-kms-proxy Vault Transit backend only supported a static
token. Add an encryption.kms.vault.auth switch — "token" (default,
unchanged) or "approle" — wiring KKP_VAULT_ROLE_ID / KKP_VAULT_SECRET_ID
/ KKP_VAULT_APPROLE_MOUNT from a Secret. With AppRole the proxy logs in
and re-authenticates on demand, so no long-lived token is stored.
Requires a keycloak-kms-proxy image with AppRole support (> v0.1.0).
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
## What this PR does
Bumps the LINSTOR storage stack to current upstream, picking up the
published security fixes the issue enumerates:
- `linstor-server` (piraeus-server) `1.33.2 → 1.33.3` (latest 1.33
patch; v1.34.0 is rc-only). Pulls in the 1.33.3 DRBD fixes (clone-status
check before connection states arrive; the secondary-after-mkfs path is
now guarded upstream too) plus refreshed Debian packages. Note: 1.33.3
also disables DRBD volume shrink upstream (it never worked correctly) —
no effect on cozystack, which does not expose volume shrink.
- `linstor-csi` `v1.10.6 → v1.11.2`. Its go.mod requires Go ≥ 1.26, so
the CSI builder moves `golang:1.25 → golang:1.26` — the right lever for
the Go stdlib / `moby/spdystream` CVEs.
Both cozystack images are rebuilt multi-arch (linux/amd64 + linux/arm64)
and pinned by digest in `values.yaml`. All five piraeus-server patches
re-apply cleanly against v1.33.3 in the Dockerfile glob order, and both
linstor-csi patches re-apply against v1.11.2.
Adds helm-unittest coverage (the package shipped none) asserting the
controller / CSI / satellite images stay digest-pinned
(`<image>:<tag>@sha256:<64-hex>`), version-agnostic so it survives the
release-prepare re-tag.
**Verified on a dev cluster:** a rolling restart of all satellites,
controller, and CSI to 1.33.3 / v1.11.2 kept every node Online with zero
faulty resources; a replicated PVC provisioned, mkfs'd, and reached
UpToDate across 3 nodes with no DRBD "device held open" abort (the
cozystack secondary-after-mkfs patch handled the path). The csi v1.11.2
"no RWX without DRBD layer" change is a no-op for cozystack, which
always uses the DRBD layer.
Closes#2881
### Release note
```release-note
chore(linstor): bump linstor-server to 1.33.3 and linstor-csi to v1.11.2
Upgrade impact: an empty tenant Kubernetes worker `nodeGroups[*].storageClass` now defaults to the application-level (DRBD `replicated`) storageClass instead of the management-cluster default, because linstor-csi v1.11.2 rejects ReadWriteMany volumes on a non-DRBD storage class. This changes the rendered worker template, so existing tenant clusters that relied on the empty-storageClass default roll their worker MachineDeployments once on the first reconcile after upgrade — workers are reprovisioned on the application storageClass and container images are re-pulled. The per-nodeGroup override and the both-empty cluster-default escape hatch are preserved.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated LINSTOR Server to **v1.33.3** and LINSTOR CSI to **v1.11.2**
(image tags/digest pins refreshed)
* Updated the LINSTOR CSI build toolchain to **Go 1.26**
* Added a **test** make target to run Helm unit tests
* **Documentation**
* Updated the LINSTOR Server patches README to reference **v1.33.3**
* **Tests**
* Added an image-pinning validation suite to confirm controller and
satellite images use **sha256** digests
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Worker-disk storageClass default (companion to the bump)
linstor-csi v1.11.2 rejects ReadWriteMany volumes that are not on a
DRBD-layer storage class. Worker VMs live-migrate, so their disks need
RWX — the access mode comes from the chosen StorageClass's CDI
StorageProfile (a DRBD class yields RWX), it is not set on the worker
DataVolume by the chart. With the bump, a worker nodeGroup left on a
non-DRBD cluster-default StorageClass would fail provisioning, so an
empty per-nodeGroup storageClass now defaults to the application
storageClass (replicated/DRBD) instead of the cluster default. The
per-nodeGroup override and the both-empty cluster-default escape-hatch
are preserved.
Upgrade impact: the empty-storageClass default ships on `md0`, and the
KubevirtMachineTemplate name is content-addressed over the rendered
worker body, so this flip renames the template and triggers a one-time
CAPI rolling replacement of worker pools on the first reconcile after
upgrade. Two hash-pin tests guard the rename so it stays an intentional,
reviewed change. linstor-csi v1.11.2 also defaults DRBD two-primaries to
kubevirt-only — a no-op for cozystack, where kubevirt is the target.
Structural follow-up to the previous commit's `replicas: {{ $group.minReplicas }}`.
Emitting the field from the chart at all — even with the semantically-
correct value — keeps the helm-controller ↔ cluster-autoscaler field-
ownership fight alive: helm-controller applies with SSA + force-conflicts
(https://fluxcd.io/flux/components/helm/helmreleases/), so every routine
`helm upgrade` action re-asserts `replicas: {{ $group.minReplicas }}` and
drains whatever the autoscaler had patched in the meantime, with a real
disruption window on running clusters.
Drop the field from the manifest entirely. CAPI's MachineDeployment
defaulting webhook seeds `spec.replicas` from the
`cluster-api-autoscaler-node-group-min-size` annotation on Create (see
https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling),
so fresh installs still start with the intended minimum. From that
point on the autoscaler owns the field via SSA field-manager tracking
and chart apply never touches it again.
Behavioural summary:
* Fresh install with `minReplicas: 0` → CAPI defaults `replicas: 0`
on Create → no worker DataVolumes provisioned until scale-out.
* Fresh install with `minReplicas: 2` → CAPI defaults `replicas: 2`
on Create → byte-identical to pre-#3231 behaviour.
* Existing customer (any minReplicas) → next chart upgrade does not
touch spec.replicas — the autoscaler-held value is preserved
unconditionally.
Regression tests updated to assert:
* `notExists: spec.replicas` on the emitted MachineDeployment (the
exact contract this commit establishes).
* Autoscaler `min-size` / `max-size` annotations continue to carry
the nodeGroup's declared bounds (so a future refactor cannot
accidentally drop the wiring the defaulting webhook depends on).
helm-unittest suite stays at 173/173.
Ref: cozystack/cozystack#3231.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Follow-ups from the self-review of #3233:
- `values.yaml` / helper docstring / regenerated README + schema +
cozyrds openAPISchema + Go type doc: the previous "one
ingress-nginx worker" contract was already misleading (helper
default carried `minReplicas: 0` but the chart hardcoded
`spec.replicas: 2`), and the replicas fix makes the docstring
strictly false — an empty-nodeGroups install now provisions zero
workers until either the cluster-autoscaler responds to an
unschedulable Pod or the operator explicitly scales the group.
Update the docstrings to describe the actual behaviour and to
spell out the ingress-nginx-addon implication so operators are
not surprised at addon-install time.
- `tests/nodegroups_default_test.yaml`: two regression cases pinning
`MachineDeployment.spec.replicas == $group.minReplicas` for
`minReplicas: 0` (the render-side OIDC bats path — zero worker
DataVolumes provisioned) and for `minReplicas: 2` (the
kubernetes-latest / kubernetes-previous bats path — byte-
identical to pre-PR behaviour). Without these a future refactor
can silently re-hardcode `replicas: 2` and turn no tests red.
helm-unittest suite grows from 171 to 173, both new cases pass.
Ref: cozystack/cozystack#3231.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
Implements the immutable-tag + rc-promotion flow from #2677. A **stable
release becomes a renamed release-candidate**: the bytes shipped as
`vX.Y.Z` are bit-for-bit the bytes built and e2e-tested as
`vX.Y.Z-rc.N`. No tag is ever force-moved, and stable is never rebuilt —
it is *promoted* by retagging the rc's existing images.
Layered onto the build matrix from #2937/#2983 (this PR is **stacked on
`refactor/build-matrix-2937`** and must merge after it).
## The five force-retag sites, removed
| Site | Before | After |
|---|---|---|
| `tags.yaml` api/apps/v1alpha1 tag | `git tag -f` / `push -f` |
write-once (create-if-absent, fail if it would move) |
| `tags.yaml` release-X.Y.Z branch | `git branch -f` / `push -f` |
compare-before-force (no-op if unchanged; staging branch only) |
| `pull-requests-release.yaml` stable tag | `git tag -f` / `push -f` |
write-once at the PR merge commit (force impossible by construction) |
| `pull-requests-release.yaml` maintenance branch | `updateRef
force:true` | fast-forward-only |
| `auto-release.yaml` patch tags | delete-recreate (cron) | **workflow
deleted** — stable only via explicit promote |
## Version decoupling (the enabler)
The operator baked its version into the image at build time, so an rc
image self-reported the rc string — blocking retag-promotion. It now
reads `COZYSTACK_VERSION` from the environment (threaded via
`cozystackOperator.platformVersion` → Deployment env, stamped by `make
manifests`), falling back to the build-time value when unset. The same
image bits can report any release name. The only runtime reader is the
telemetry metric `cozy_cluster_info{cozystack_version=...}`.
## Promotion flow
`promote-rc.yaml` (`workflow_dispatch`, `rc_tag=vX.Y.Z-rc.N`):
1. Validate the rc release exists and the stable tag does not.
2. `hack/promote-retag.sh` reads the rc's digest-pinned image refs from
`packages/*/*/values.yaml` and `skopeo copy`s each — **by digest** — to
`:vX.Y.Z` and `:latest`, verifying with `skopeo inspect`.
3. Rewrite the cosmetic `-rc.N` substring in vendored tags to the stable
version (the `@sha256` wins regardless), restamp the version-stamped
assets (operator manifests, cozypkg, openapi; the heavy Talos assets are
copied verbatim from the rc draft), open a `release-X.Y.Z` PR.
4. Merging the PR reuses the existing release-PR e2e and
`pull-requests-release.yaml` finalize to cut the **write-once** stable
tag at the merge commit and publish the release. Squash is disallowed
(the tag needs a real merge commit).
`nightly.yaml` cuts write-once `*-nightly.<date>` tags (gated by
`NIGHTLY_ENABLED`); `retention.yaml` keeps the newest 14 per line
(dry-run by default).
## Validation
**Local end-to-end — the core claim is proven, not asserted.** The whole
design rests on "copy-by-digest to a new tag preserves the digest, so
stable == the e2e-tested rc, bit-for-bit." This was exercised against a
real registry (`ttl.sh`) with real `skopeo`:
- Pushed a multi-arch image under a `:v1.4.0-rc.2` tag, built a temp
tree exercising all three `values.yaml` digest shapes (single `image:`
string, split `repository`/`tag`/`digest` map, and the
`platformSourceRef` OCI artifact), then ran the actual
`hack/promote-retag.sh v1.4.0`.
- **Result: `:v1.4.0-rc.2`, `:v1.4.0`, and `:latest` all resolve to the
identical digest** (`sha256:fd8d9aa6…`), across all 17 platform
manifests. The script's own `skopeo inspect` post-check passed, and an
independent re-inspection confirmed it.
Local validation caught (and this PR fixes) two bugs that only surface
against a real registry — never in static lint:
1. `skopeo copy --multi-arch all --all` — mutually exclusive flags,
fatal error; every retag would have failed on CI. Now `--multi-arch
all`.
2. The same digest was retagged twice when an image appeared in two
value shapes; the ref set is now deduped on the canonical `repo@digest`.
**Other local checks:** `go test ./pkg/version` + `go build`/`vet`;
`helm template` renders `COZYSTACK_VERSION` in all 3 operator variants
(omitted when unset); `make manifests` stamps the version into the
install assets; `shellcheck` clean on `promote-retag.sh`; `actionlint`
clean and `act -l` resolves the job graph on all workflows; the rc-tag
parser and nightly version-math unit-tested (accept/reject +
minor-vs-patch bumps).
**Still CI-only (cannot be exercised offline):** the full workflow
runtime — github-script API calls, OCIR auth, the rc-release /
staging-branch preconditions, the nightly `base_ref` push mechanics, and
self-hosted runners. The *logic* inside the steps is unit-tested; the
*orchestration* is not. Treat it as unproven until the workflows run.
## Out of scope / required follow-up (needs repo admin — not doable from
a PR)
- Tag-protection rules: `v*` and `api/apps/v1alpha1/*` create-only by
the CI app, no delete/update; "limit branches/tags updated in a single
push" = 1.
- Enable `NIGHTLY_ENABLED` (and `RETENTION_APPLY` when ready) repo
variables.
- Registry-side pruning of `*-nightly.*` image tags (no OCIR delete
parity yet — tracked TODO in `retention.yaml`).
## Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added nightly publishing (mirror-by-digest, disk build, e2e
validation) and nightly retention pruning.
* Added RC-to-stable promotion via digest retagging (no rebuild).
* Operator/installer and console now expose version/platform metadata,
including runtime override via `COZYSTACK_VERSION`.
* **Bug Fixes**
* Enforced write-once tag behavior and fast-forward-only maintenance
updates across release workflows.
* Made nightly mirroring/selection and retention pruning more selective
and safer.
* **Tests**
* Added Bats coverage for nightly mirroring and RC retagging, plus new
Helm/unit checks.
* **Documentation**
* Updated the release model to center RC promotion and tag immutability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Closes#3231.
`packages/apps/kubernetes/templates/cluster.yaml` hardcoded
`MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's
`minReplicas`. The two render-side `kubernetes-oidc-{system,customconfig}`
bats — which set `minReplicas: 0` and delete their `Kubernetes` CR
without waiting for readiness — still provisioned two 20 GiB
`replicated` (DRBD) worker `DataVolume`s each. Their create/delete
churn overlapped the very next test, `kubernetes-previous`, and put
transient pressure on a LINSTOR satellite's ZFS pool: worker root
disks landed late, tenant nodes registered late, Cilium's CNI init
missed the 12-minute node-join deadline, and the test failed with
`cni plugin not initialized`.
Two changes, applied together — the chart fix is a no-op without
the test fix, and the test fix is a no-op without the chart fix,
because dropping `nodeGroups` falls back to the chart helper's
default `md0` (with `minReplicas: 0`).
**Chart**: `spec.replicas: {{ $group.minReplicas }}` so the field
matches the autoscaler annotations already emitted just above.
`minReplicas: 0` groups now genuinely scale from zero and do not
silently provision workers; groups with `minReplicas >= 2` (the
kubernetes-latest / kubernetes-previous bats CRs, the tests/values/
common.yaml default, the values.yaml default that gets templated
into the helper's `md0`) render byte-identical to before.
**Render-side OIDC tests**: drop the explicit `nodeGroups:` block
and `storageClass:` — falling back to the chart helper's default
`md0` (now with an effective `replicas: 0` after the chart change)
means the MachineDeployment renders but never creates a KubevirtMachine
and no DataVolume is provisioned. The tests only assert on the
KamajiControlPlane and the cozy-realm Keycloak objects, so removing
worker plumbing does not narrow their coverage.
Upgrade impact: any existing customer CR that already carries
`nodeGroups.md0.minReplicas: 0` (rare — the default in tests/values
and the chart helper both use it, but real user CRs almost always
have `minReplicas >= 1`) will see the MD's `spec.replicas` reconcile
from `2` down to `0` on the next chart apply. If the cluster-
autoscaler is enabled it will scale back up as soon as an
unschedulable pod appears; if it is not, the group intentionally
becomes empty (which is what `minReplicas: 0` requests literally).
This matches the semantics the autoscaler annotations were already
advertising — the hardcoded `2` was contradicting them.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for the newer etcd snapshot workflow
(BackupClass/BackupJob/RestoreJob) with in-place restore.
* Updated etcd chart/operator packaging and migration flow for the
v1alpha2 operator release.
* **Bug Fixes**
* Switched readiness/condition handling to **Available**.
* Improved waits to cover all etcd member pods during restore and
scenario steps.
* **Documentation**
* Refreshed etcd backup/restore docs and example scenarios to match the
v1alpha2 APIs and parameters.
* **Tests**
* Updated e2e coverage to validate v1alpha2 operator runtime contracts
and replaced legacy backup-block tests with a snapshot round-trip test.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address non-blocking review feedback from lexfrei on
packages/system/dashboard/Makefile:34.
When COZYSTACK_VERSION is empty (a local build with no matching git tag) the
build-arg baked a bare `APP_VERSION=v` into the console bundle. Match the guard
the root Makefile already uses for platformVersion so an empty version yields an
empty APP_VERSION instead of "v". Dev-only cosmetics — the config.json path
overrides the displayed version at deploy time — but it removes the mismatch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- nightly.yaml: pass the registry tokens to `skopeo login` over stdin
(--password-stdin) instead of `-p` on argv, matching promote-rc/finalize and
keeping the token out of the process list.
- dashboard configmap: require digits-then-dot (^v?[0-9]+\.) when deriving the
displayed platform version from the console image tag. The old ^v?[0-9] let a
tagless, port-carrying ref (ghcr.io:443/.../console, whose last ":"-segment is
"443/.../console") pass as a version. Verified: v1.5.0 / 1.5.0 still surface;
the port ref and :latest no longer do.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The console config ConfigMap stripped the leading v (trimPrefix "v") while
the operator's COZYSTACK_VERSION and the vX.Y.Z release tags keep it, so the
dashboard displayed 1.4.0 where the operator reported v1.4.0. Keep the tag
verbatim, and build the baked VITE_APP_VERSION fallback as v$(COZYSTACK_
VERSION) too so the deploy-time and build-time version strings stay
consistent with each other and with the operator.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The COZYSTACK_VERSION env test pinned talos and hosted but not generic,
leaving the generic branch of the endpoint-env interleave (KUBERNETES_
SERVICE_HOST/PORT from cozystack.apiServerHost/Port) unguarded. Add a
generic + platformVersion case so a refactor can't silently drop
COZYSTACK_VERSION or mis-render the generic endpoint block.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on packages/system/etcd-operator/templates/deployment.yaml: add an explicit seccompProfile (RuntimeDefault) to the controller-manager pod securityContext.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>