The Job in templates/talos/talos-reconcile-job.yaml is the only
producer of the TalosConfigTemplate that each worker
MachineDeployment names in spec.template.spec.bootstrap.configRef;
cluster.yaml renders the reference but deliberately not the object.
Its loop read .Values.nodeGroups directly, while the
MachineDeployment loop reads the effective set through the
kubernetes.nodeGroups helper.
For a cluster that supplies no nodeGroups the two disagree: the
helper's else-branch emits the built-in md0 group, so the md0
MachineDeployment renders, but the raw map is empty and no Job
renders at all. Every Machine the cluster-autoscaler adds to md0
then blocks indefinitely on a TalosConfigTemplate that nothing will
ever create, and the KamajiControlPlane spec.network.certSANs patch
the same Job performs is skipped too.
Range over the helper so the Job set tracks the MachineDeployment
set exactly. No change for a cluster that declares its own groups,
and md0 stays removable: the helper's if-branch keeps a
user-supplied map authoritative.
The gap survived because every helm-unittest fixture and e2e suite
declares md0 explicitly. tests/nodegroups_default_test.yaml does
render the empty-nodeGroups case but lists only
templates/cluster.yaml, so it never looked at the Job. The new
suite pins both halves of the helper contract and fails without
this fix.
Fixes#3504
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 4e9c5ce8ca)
The preserve-old-KubevirtMachineTemplate block re-emitted any KMT owned
by the CAPI Cluster and referenced by a MachineSet. A kubernetes-nodes
pool's KMT lives in the same namespace under the same Cluster, so the
parent would re-emit it with a foreign meta.helm.sh/release-name and its
next upgrade would fail Helm ownership validation. Restrict the block to
KMTs whose release-name annotation matches the parent release.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
quay.io serves only currently-tagged manifests: when a rolling tag moves,
the superseded digest immediately returns MANIFEST_UNKNOWN. The pinned
`centos:stream9` digest has died twice in six days (#3303, #3386), each
time breaking `make build` on main and on every PR that triggers a full
rebuild.
Repinning is not a fix -- the newly pinned digest is always the live tag,
so it is next in line to be retired. Move the runtime stage to a registry
that provably retains superseded manifests instead, which makes the digest
pin durable rather than decorative. Pinning policy is unchanged.
Verified equivalent: after `dnf install -y e2fsprogs xfsprogs nfs-utils`,
both bases provide mkfs.ext4, mkfs.xfs, blkid, mount, umount, fsck.ext4,
fsck.xfs and mount.nfs at identical paths. e2fsprogs and xfsprogs are
identical NVRs. The driver is built CGO_ENABLED=0, so the base libc is not
linked. The pinned digest is the OCI index covering amd64, arm64, ppc64le
and s390x -- the same architecture set as the previous base.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
quay.io/centos/centos:stream9 is a rolling tag; upstream deletes the pinned
digest whenever it publishes a new stream9 build. The digest re-pinned in
ce4b263ca (sha256:a93e3316...) has now itself been deleted, so every
kubevirt-csi-driver image build fails with:
quay.io/centos/centos:stream9@sha256:a93e3316...: not found
failed to resolve source metadata ...
make: *** [image-kubevirt-csi-driver] Error 1
breaking the "Build packages/apps/kubernetes" job on main and every PR.
Re-pin to the current stream9 manifest-list digest
(sha256:3714c89f1903dac5a5e1eb6c02d84189ff6d962a0c18df01feba0a5406856926),
resolved via `docker buildx imagetools inspect quay.io/centos/centos:stream9`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The System-mode AuthenticationConfiguration rendered a claimValidationRule
ranging over claims.groups, which is statically typed `any` in the apiserver
CEL environment. The .exists() comprehension rejects a range of type any, so
the tenant kube-apiserver failed config compilation at startup and CrashLooped
— the control plane never booted and cluster teardown then hung the e2e
cleanup budget. Wrap the range in dyn() so CEL treats it as dynamic; the has()
guard still short-circuits when groups is absent and the gate semantics are
unchanged. Strengthen the oidc unit test to pin dyn() (mutation-verified).
Verified on a live tenant apiserver (v1.32).
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Symmetric with the Grafana-side allowed_groups gate — round-3 review
from lllamnyp on PR#3176 covered both surfaces, and #3044's original
System-mode wiring shipped without this guard. The rationale is
identical: audience binding stops REPLAY of a minted token across
tenant kube-apiservers, but nothing stopped Keycloak from ISSUING a
usable token to a caller outside the tenant.
Absent a claim-side gate, any authenticated cozy-realm user could
`kubectl oidc-login` against another tenant's cluster and land at
`system:authenticated`. RBAC default-denies named-resource access, but
`system:authenticated` still leaks discovery (kubectl api-resources,
OpenAPI schemata) — tenant-alice could enumerate tenant-bob's cluster's
CRD shape and built-in resource surface.
Fix: add a CEL `claimValidationRules` entry to the rendered
AuthenticationConfiguration (System mode only) that requires at least
one of the release's four tenant-scoped Keycloak groups
(`<namespace>-{view,use,admin,super-admin}`) to be present in the
token's `groups` claim. The `has(claims.groups) &&` guard is required
— CEL evaluation on a missing claim surfaces as HTTP 500 from the
authenticator instead of a clean 401.
CustomConfig mode is not touched — the tenant's own
AuthenticationConfiguration is authoritative and the tenant is
responsible for claim-side guards on their own issuer.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
quay.io/centos/centos:stream9 is a rolling tag; upstream deleted the pinned
digest (sha256:0c9db821...) when it published a new stream9 build, so every
build of image-kubevirt-csi-driver fails with "not found" — on open PRs and
on main. Re-pin to the current live stream9 digest (sha256:a93e3316...).
This is an emergency CI unblock, not a durable fix: stream9 is garbage-collected,
so upstream will delete this digest too on its next rebuild and the same
"not found" breakage recurs until the pin is bumped again — the same operation
Renovate's weekly, non-automerged digest PR performs. A durable fix (mirror the
base image into cozystack's own registry, or move to an immutable-digest base
such as UBI9) is out of scope for this hotfix and left as follow-up.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
|
[github.com/kubevirt/csi-driver](https://redirect.github.com/kubevirt/csi-driver)
| replace | digest | `51c6bca` → `27b52aa` |
---
### Configuration
📅 **Schedule**: (UTC)
- Branch creation
- At any time (no schedule defined)
- Automerge
- At any time (no schedule defined)
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/cozystack/cozystack).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI1OS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0ZWQiLCJkby1ub3QtbWVyZ2UvaG9sZCJdfQ==-->
Add optional nodeGroups[name].maxUnhealthy and nodeGroups[name].nodeStartupTimeout
fields that override the cluster-wide nodeHealthCheck defaults per worker node
group. When unset, a group inherits nodeHealthCheck.{maxUnhealthy,nodeStartupTimeout},
so existing values are unaffected. The MachineHealthCheck render resolves the
per-group value first, reuses the existing int-or-string validation, and names
the offending group in the fail() message when an override is invalid.
Delivers the per-group granularity of the standalone maxUnhealthy proposal on
top of the cluster-wide nodeHealthCheck from the Talos worker bootstrap PR.
Co-authored-by: mattia-eleuteri <mattia@hidora.io>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Migrates the E2E **app** test suite from BATS to [Kyverno
Chainsaw](https://github.com/kyverno/chainsaw) — a declarative,
Kubernetes-native E2E framework (CNCF, part of the Kyverno project;
successor to KUTTL) — and wires it into CI as the replacement for the
per-app BATS loop. This grows the original pilot (postgres + bucket)
into the full suite.
All 22 app suites are ported under `hack/e2e-chainsaw/` and the
`hack/e2e-apps/*.bats` files are removed:
`postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `qdrant`,
`clickhouse`, `kafka`, `etcd`, `openbao`, `harbor`, `foundationdb`,
`external-dns`, `kuberture`, `vminstance`, `gateway`,
`kubernetes-latest`, `kubernetes-previous`, `kubernetes-oidc-system`,
`kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure`.
**Approach**
- **Declarative** suites assert on `status.conditions` and concrete
fields. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait`
pair that appeared ~100 times collapses into a single `assert` that
polls existence and state together, with a structured diff on failure
and automatic `events`/`describe`/`podLogs` capture via `catch`
(previously only `harbor.bats` did this, by hand).
- **Imperative** suites keep their logic in `script` steps: `openbao`
init/unseal, `kuberture` external-dns split-horizon probes,
`vminstance`, the `gateway` admission/impersonation cases, and
`kubernetes-latest`/`previous`, which wrap the relocated
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` verbatim (Kamaji bring-up,
LB/NFS/ouroboros checks).
- `gateway` tests derive the tenant apex from the namespace
`namespace.cozystack.io/host` label at runtime, so they are
host-independent.
**CI**
- The `e2e` job now runs `chainsaw test hack/e2e-chainsaw/` via a new
`test-chainsaw` target and uploads the JUnit `chainsaw-report.xml`.
- The `chainsaw` binary is added to the e2e-sandbox image.
- `install-cozystack` and `test-openapi` stay BATS (cluster bootstrap +
OpenAPI checks).
- The per-app 3-retry loop is dropped — assertion polling replaces the
fixed-timeout flakiness it papered over.
**Validation**
Ran against a development cluster: the DB/app, storage, and VM suites
pass; the three suites that depend on platform features not present on
that cluster (`gateway`, `kuberture`, `external-dns`) and the two
heavyweight `kubernetes-*` suites are exercised by this PR's CI run on a
freshly installed platform.
Note for reviewers: service-port asserts use the `(ports[*].port)`
projection form rather than a number-literal filter (`` ports[?port ==
`N`] ``), because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal
comparisons.
**2026-07-09: reconciled with main (~520 commits of drift)**
- etcd suite re-ported to the v1alpha2 operator contracts (#2859):
`readyMembers=3` gate, pod-label/`/scale`/WorkloadMonitor/metrics/defrag
contracts, plus the `ETCD_E2E_S3_ROUNDTRIP`-gated backup round-trip
driving `examples/backups/etcd`
- 12 new `ingress-hostname-policy` gateway cases ported (apex derived at
runtime from the tenant-root namespace label)
- post-branch bats drift folded in: `mariadb-single` webhook guard,
kafka `c1.small` presets, bucket/harbor 2m BucketClaim fail-fast, bucket
readonly-denial promoted to a hard fail (cosi-driver v0.3.1), qdrant
PVC-reclaim guard (#3059), kuberture fail-loud negations,
SC-fallback-default test (#2872 B1)
- 4 suites that only exist on main since the branch was cut are ported:
`kubernetes-oidc-system`, `kubernetes-oidc-customconfig`,
`securitygroup`, `serviceexposure`
- CI seams: `nightly.yaml` converted to the chainsaw invocation,
`run-kubernetes.sh` reconciled (Talos/CABPT waits, tenant drain, LINSTOR
pool wait, talos-image-cache under `_lib/`), `e2e-capture-dataplane.sh`
replicated in the Chainsaw global catch
### Release note
```release-note
fix(kubernetes): tenant Kubernetes teardown no longer hangs when the cluster has no working nodes — the pre-delete hook now bounds its wait for the in-tenant HelmReleases and force-clears their Flux finalizers on timeout (#3271)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* End-to-end testing is now driven by a unified Chainsaw suite (platform
apps, Kubernetes, networking, security groups, service exposure).
* Test Impact Analysis selects affected suites, with an option to run
the full suite.
* CI produces a JUnit report artifact and collects suite-scoped
diagnostics on failure.
* **Bug Fixes**
* CI E2E now fails fast and emits clearer Kubernetes/HelmRelease and
sandbox bucket state diagnostics.
* Improved teardown reliability with bounded wait + finalizer handling
to avoid hangs.
* **Documentation**
* Updated E2E testing docs and local guidance to reflect the new
Chainsaw-based workflow and suite paths.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux
6.18.24. That kernel predates the fixes for two related KVM x86
shadow-paging use-after-free bugs in `arch/x86/kvm/mmu/mmu.c`:
CVE-2026-46113, fixed upstream in
`0cb2af2ea66ad8ff195c156ea690f11216285bdf` and released in 6.18.30, and
CVE-2026-53359, fixed in `81ccda30b4e83d8f5cc4fd50503c44e3a33abfeb` and
released in 6.18.38. A worker's own kernel runs KVM whenever a tenant
enables nested virtualization inside the VM, and either bug lets a
nested guest escape into the worker.
This bumps the worker pin to Talos v1.13.6, which ships Linux 6.18.38.
Stable kernel branches are cumulative, so v1.13.6 is the first v1.13
patch release carrying both fixes. The pinned v1.13.0 ships 6.18.24 and
carries neither; v1.13.5 ships 6.18.36 and carries only CVE-2026-46113.
Nested virtualization stays available to tenants; the kernel bump is the
fix, not a capability removal.
The image-factory schematic is unchanged and resolves for the new
version: both the `openstack-amd64.raw.xz` artifact and the
`installer/<schematicID>:v1.13.6` manifest are served by the public
factory. Talos v1.13 supports Kubernetes 1.31 through 1.36, so the
chart's Talos-to-Kubernetes support window is unaffected and no tenant
`spec.version` needs to move.
Applying this rolls each worker pool once. The new pin changes the
rendered worker spec, which rotates the content hash in the
`KubevirtMachineTemplate` and `talos-reconcile` Job names, and CAPI
replaces the worker VMs to deliver the new kernel. The pinned hashes in
the chart tests are recomputed to match.
A second commit fixes the support-matrix lookup itself. The matrix dict
was keyed on the full patch version, so every Talos patch bump dropped
out of the dict and silently disabled the render-time guard — exactly
the mismatch the guard exists to catch. The lookup now derives the minor
from `talos.version`. The supported-Kubernetes window is unchanged;
unknown minors are still skipped, leaving compatibility to whoever
overrode the pin.
### Screenshots
Not applicable — no UI changes.
### Release note
```release-note
fix(kubernetes): bump tenant worker Talos to v1.13.6, picking up the Linux 6.18.38 kernel fixes for CVE-2026-46113 and CVE-2026-53359 (KVM shadow-paging use-after-free, nested-guest escape into the worker VM). Nested virtualization remains available. Applying this upgrade rolls each tenant worker pool once, as the worker template is renamed to carry the new image.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Updated the default Talos worker release to v1.13.6 for Kubernetes
cluster provisioning.
- Improved compatibility checks to apply consistently across Talos patch
releases within the same minor version.
- **Bug Fixes**
- Corrected generated resource naming expectations when storage settings
or GPU configuration change.
- **Tests**
- Added coverage confirming Kubernetes clusters render successfully with
Talos v1.13.6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The pre-delete hook (delete.yaml) deleted the tenant's addon HelmReleases with
an unbounded `kubectl delete … --wait=true`. When the tenant has no working
nodes (LINSTOR/ZFS exhaustion, worker VMs that never joined, or a nodeGroup at
minReplicas: 0), those uninstalls cannot converge — in-tenant cilium stays
InstallFailed because cilium-operator has nothing to schedule on — so the hook
hung indefinitely, the parent Helm uninstall failed on the pre-cleanup Job, and
the Kubernetes CR was stuck deleting.
Bound the wait with --timeout=120s; on expiry, force-clear the Flux finalizer
from any child HelmRelease still present so it can be garbage-collected — safe
because the whole tenant (control plane torn down in the steps below) is being
destroyed, so there is nothing left to orphan. A healthy tenant's children
uninstall well within the timeout and never reach the force path.
Also unblocks the render-side kubernetes-oidc-{system,customconfig} Chainsaw
e2e suites, which deterministically hit this teardown hang under the Chainsaw
cleanup budget.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux 6.18.24.
That kernel predates both fixes for a pair of related KVM x86 shadow-paging
use-after-free bugs: CVE-2026-46113, fixed in 6.18.30, and CVE-2026-53359,
fixed in 6.18.38. A worker's own kernel runs KVM whenever a tenant enables
nested virtualization inside the VM, and either bug lets a nested guest
escape into the worker.
Talos v1.13.6 ships Linux 6.18.38. Stable kernel branches are cumulative, so
it is the first v1.13 patch release carrying both fixes: the pinned v1.13.0
ships 6.18.24 and carries neither, while v1.13.5 ships 6.18.36 and carries
only CVE-2026-46113. Nested virtualization stays available to tenants — the
kernel bump is the fix.
The new pin changes the rendered worker spec, so the content hash in the
KubevirtMachineTemplate and talos-reconcile Job names rotates and each
worker pool rolls once on upgrade. Pinned hashes in the chart tests are
recomputed accordingly.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Talos<->Kubernetes support window is declared per Talos minor release,
and patch releases never move it. Keying the chart's matrix dict on the full
patch version meant every Talos patch bump fell out of the dict and silently
disabled the render-time guard — precisely the mismatch the guard exists to
catch.
Derive the minor from talos.version before the lookup, so the guard keeps
firing across patch releases of a known minor. Unknown minors are still
skipped, leaving compatibility to the operator who overrode the pin.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Match the full rendered worker-disk URL shape (version + artifact + end anchor) like the sibling override assertion, so the trailing-slash normalization case cannot pass on a partial match.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Tenant worker VMs boot from a Talos raw disk image that CDI streams over
HTTP from the hardcoded public Talos Image Factory, and the in-guest
talos-reconcile Job pulls its installer image from the same hardcoded host.
Neither was overridable, so air-gapped / rate-limited / flaky-egress
environments could not point workers at a mirror (the README flagged this
as a Phase-2 gap), and CI had no way to avoid bulk-pulling the OS image
from the public internet per worker.
Add two chart values, both defaulting to the public factory so rendered
output is byte-identical and behaviour is unchanged:
- talos.imageFactoryURL (default https://factory.talos.dev) - base URL for
the worker OS disk image streamed in by CDI.
- talos.installerRepository (default factory.talos.dev/installer) - OCI
repository prefix for the in-guest installer image.
Regenerated values.schema.json, the aggregated-API Go types, the
kubernetes-rd cozyrds schema and the README via `make generate`; added
unittests for the override and trailing-slash trimming.
Refs #3231
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
sircthulhu's post-approval note on PR#3233: both the cluster.yaml
comment and the nodegroups_default_test.yaml comment attribute the
upgrade-safety guarantee to Server-Side-Apply field-manager
ownership, but this chart ships as a HelmRelease through Flux
helm-controller, which applies via the Helm SDK three-way merge —
not SSA. The stated conclusion is correct but for the wrong reason
and would send a future maintainer chasing a non-existent field-
manager fight when debugging drift.
Rewrite both comments around the two independent CAPI mechanisms
that actually carry the guarantee:
1. Create — the MachineDeployment defaulting webhook seeds
spec.replicas from the min-size autoscaler annotation.
2. Update — calculateMachineDeploymentReplicas preserves the
existing live value whenever the incoming spec has
replicas: nil (clamping into [min, max] on drift).
The chart guarantee reduces to "leave the field out of the rendered
manifest and let CAPI's update-time defaulting keep the live value
stable" — no field-manager ownership involved.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Symmetry with the zero-minReplicas neighbour, which already covers
both min-size and max-size. CAPI's calculateMachineDeploymentReplicas
defaults to 1 (not to minSize) if either annotation is missing on a
new MachineDeployment, so both keys are load-bearing on the
Create-path guarantee and worth asserting in both regression cases.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Structural follow-up to the previous commit's `replicas: {{ $group.minReplicas }}`.
Emitting the field from the chart at all — even with the semantically-
correct value — keeps the helm-controller ↔ cluster-autoscaler field-
ownership fight alive: helm-controller applies with SSA + force-conflicts
(https://fluxcd.io/flux/components/helm/helmreleases/), so every routine
`helm upgrade` action re-asserts `replicas: {{ $group.minReplicas }}` and
drains whatever the autoscaler had patched in the meantime, with a real
disruption window on running clusters.
Drop the field from the manifest entirely. CAPI's MachineDeployment
defaulting webhook seeds `spec.replicas` from the
`cluster-api-autoscaler-node-group-min-size` annotation on Create (see
https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling),
so fresh installs still start with the intended minimum. From that
point on the autoscaler owns the field via SSA field-manager tracking
and chart apply never touches it again.
Behavioural summary:
* Fresh install with `minReplicas: 0` → CAPI defaults `replicas: 0`
on Create → no worker DataVolumes provisioned until scale-out.
* Fresh install with `minReplicas: 2` → CAPI defaults `replicas: 2`
on Create → byte-identical to pre-#3231 behaviour.
* Existing customer (any minReplicas) → next chart upgrade does not
touch spec.replicas — the autoscaler-held value is preserved
unconditionally.
Regression tests updated to assert:
* `notExists: spec.replicas` on the emitted MachineDeployment (the
exact contract this commit establishes).
* Autoscaler `min-size` / `max-size` annotations continue to carry
the nodeGroup's declared bounds (so a future refactor cannot
accidentally drop the wiring the defaulting webhook depends on).
helm-unittest suite stays at 173/173.
Ref: cozystack/cozystack#3231.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Follow-ups from the self-review of #3233:
- `values.yaml` / helper docstring / regenerated README + schema +
cozyrds openAPISchema + Go type doc: the previous "one
ingress-nginx worker" contract was already misleading (helper
default carried `minReplicas: 0` but the chart hardcoded
`spec.replicas: 2`), and the replicas fix makes the docstring
strictly false — an empty-nodeGroups install now provisions zero
workers until either the cluster-autoscaler responds to an
unschedulable Pod or the operator explicitly scales the group.
Update the docstrings to describe the actual behaviour and to
spell out the ingress-nginx-addon implication so operators are
not surprised at addon-install time.
- `tests/nodegroups_default_test.yaml`: two regression cases pinning
`MachineDeployment.spec.replicas == $group.minReplicas` for
`minReplicas: 0` (the render-side OIDC bats path — zero worker
DataVolumes provisioned) and for `minReplicas: 2` (the
kubernetes-latest / kubernetes-previous bats path — byte-
identical to pre-PR behaviour). Without these a future refactor
can silently re-hardcode `replicas: 2` and turn no tests red.
helm-unittest suite grows from 171 to 173, both new cases pass.
Ref: cozystack/cozystack#3231.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Closes#3231.
`packages/apps/kubernetes/templates/cluster.yaml` hardcoded
`MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's
`minReplicas`. The two render-side `kubernetes-oidc-{system,customconfig}`
bats — which set `minReplicas: 0` and delete their `Kubernetes` CR
without waiting for readiness — still provisioned two 20 GiB
`replicated` (DRBD) worker `DataVolume`s each. Their create/delete
churn overlapped the very next test, `kubernetes-previous`, and put
transient pressure on a LINSTOR satellite's ZFS pool: worker root
disks landed late, tenant nodes registered late, Cilium's CNI init
missed the 12-minute node-join deadline, and the test failed with
`cni plugin not initialized`.
Two changes, applied together — the chart fix is a no-op without
the test fix, and the test fix is a no-op without the chart fix,
because dropping `nodeGroups` falls back to the chart helper's
default `md0` (with `minReplicas: 0`).
**Chart**: `spec.replicas: {{ $group.minReplicas }}` so the field
matches the autoscaler annotations already emitted just above.
`minReplicas: 0` groups now genuinely scale from zero and do not
silently provision workers; groups with `minReplicas >= 2` (the
kubernetes-latest / kubernetes-previous bats CRs, the tests/values/
common.yaml default, the values.yaml default that gets templated
into the helper's `md0`) render byte-identical to before.
**Render-side OIDC tests**: drop the explicit `nodeGroups:` block
and `storageClass:` — falling back to the chart helper's default
`md0` (now with an effective `replicas: 0` after the chart change)
means the MachineDeployment renders but never creates a KubevirtMachine
and no DataVolume is provisioned. The tests only assert on the
KamajiControlPlane and the cozy-realm Keycloak objects, so removing
worker plumbing does not narrow their coverage.
Upgrade impact: any existing customer CR that already carries
`nodeGroups.md0.minReplicas: 0` (rare — the default in tests/values
and the chart helper both use it, but real user CRs almost always
have `minReplicas >= 1`) will see the MD's `spec.replicas` reconcile
from `2` down to `0` on the next chart apply. If the cluster-
autoscaler is enabled it will scale back up as soon as an
unschedulable pod appears; if it is not, the group intentionally
becomes empty (which is what `minReplicas: 0` requests literally).
This matches the semantics the autoscaler annotations were already
advertising — the hardcoded `2` was contradicting them.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for the newer etcd snapshot workflow
(BackupClass/BackupJob/RestoreJob) with in-place restore.
* Updated etcd chart/operator packaging and migration flow for the
v1alpha2 operator release.
* **Bug Fixes**
* Switched readiness/condition handling to **Available**.
* Improved waits to cover all etcd member pods during restore and
scenario steps.
* **Documentation**
* Refreshed etcd backup/restore docs and example scenarios to match the
v1alpha2 APIs and parameters.
* **Tests**
* Updated e2e coverage to validate v1alpha2 operator runtime contracts
and replaced legacy backup-block tests with a snapshot round-trip test.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The claim mapping switched to email in de0299f33, but two header
comments still described the earlier design:
oidc-authn-config.yaml:15 — "Username claim is preferred_username"
oidc-rbac-job.yaml:28 — "matched against preferred_username"
The rendered config (oidc-authn-config.yaml:49) maps
username.claim: email, the CRB subject is rendered from .email
(oidc-rbac-job.yaml:264), and the email-claim contract is pinned by
tests/oidc_test.yaml:260-261. Only the comments were stale; no code
change.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Gemini flagged that the `values.yaml` @field docstring on
`oidc.users[].email` still described the claim mapping as
`preferred_username`, misleading operators after the claim was
switched to `email` for real (see the earlier fix and its
end-to-end verification on dev3). Rewrite the description to
match the actual mechanic: `email` claim, requested via
`--oidc-extra-scope=email` in the chart-generated kubeconfig,
emitted by every conformant OIDC provider.
`make generate` re-propagates the update through README.md,
types.go and the cozyrds embedded schema.
Note on the other finding in the same review — Gemini also
flagged `audiences` as being incorrectly nested under `issuer`
in the AuthenticationConfiguration Secret. That is a
false positive: per the kube-apiserver v1beta1 spec
(apiserver.config.k8s.io/v1beta1, `JWTAuthenticator.Issuer`),
`audiences` IS a child of `issuer`, not a sibling of `jwt[i]`.
The current shape is correct and verified end-to-end on dev3
(alice/bob tokens carry `aud: [tenant-test-kubernetes-test3,
kubernetes]` and apiserver validates them successfully with
CRB-driven RBAC).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Four findings from the gemini re-review:
- `.Values.oidc.customConfig.secretRef.name` and
`.Values.oidc.customConfig.config` — replaced direct chained
access with `dig "customConfig" "…" "" .Values.oidc` so a CR that
explicitly nulls `customConfig` doesn't nil-panic. `.Values.oidc`
itself has a kubebuilder default of `{}` so it stays reachable.
- `.Values.controlPlane.apiServer.extraArgs` in the collision guard
— wrapped with `| default list` so an explicit null renders as an
empty range rather than a nil deref. (`dig` on `.Values` itself
doesn't work because it's `chartutil.Values`, not
`map[string]interface{}`.)
- `base64 -w 0` in the OIDC kubeconfig patch — replaced with
`base64 | tr -d '\n'`. Both work on the current alpine/k8s image
(BusyBox 1.37 supports `-w`), but the pipe form is portable back
to older BusyBox releases and to macOS/BSD base64 (which does not
know `-w`).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Same class of upgrade regression: a customer who wired their own
structured AuthenticationConfiguration by hand via
`controlPlane.apiServer.extraArgs: [--authentication-config=...]`
would hit a duplicate-flag scenario the moment they toggle
`spec.oidc.mode` — kube-apiserver would receive two conflicting
--authentication-config= arguments and the last-wins behaviour is
undefined between apiserver versions.
Add a second render-time guard that fails when a legacy
--authentication-config= is present in the user-supplied extraArgs
AND spec.oidc.mode is not None, pointing at the CustomConfig
migration path (customConfig.config or customConfig.secretRef.name).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Reviewer flagged a real customer-upgrade regression: the previous
values.yaml docstring on `controlPlane.apiServer.extraArgs` documented
`--oidc-issuer-url`/`--oidc-client-id` as a valid usage. An existing
customer who wired OIDC by hand through that passthrough (following
the doc) would see their tenant kube-apiserver BOOT-FAIL the moment
they toggle `spec.oidc.mode` to `System` or `CustomConfig`:
kube-apiserver refuses to boot with both `--authentication-config`
(injected by the new spec.oidc) and any legacy `--oidc-*` flag —
`--authentication-config cannot be used with any of --oidc-*`.
Add a render-time guard: when `spec.oidc.mode != None`, scan the
user-supplied `controlPlane.apiServer.extraArgs` for any flag with
prefix `--oidc-` and fail with a clear message pointing at the
`spec.oidc.mode: CustomConfig` migration path (they can carry the
same issuer/audience inside `customConfig.config`).
When `spec.oidc.mode: None` the passthrough still works — customers
who manage OIDC entirely by hand keep working until they opt into
the new selector explicitly.
Also update the values.yaml docstring on `extraArgs` to steer OIDC
users to `spec.oidc.mode` and warn against the collision.
Two new helm-unittest cases: the collision fails, and mode=None with
legacy flags renders normally (backward compat).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Two follow-ups from the final adversarial review:
- The chart-generated OIDC kubeconfig exec block requested only
`--oidc-extra-scope=groups`. On a Keycloak realm where `email` is
not a default client scope, kubectl-oidc-login would fetch a token
that lacks the `email` claim — apiserver would then fail to
identify the user, causing a silent auth failure. Add
`--oidc-extra-scope=email` explicitly so the flow does not depend
on realm-level default-scope hygiene.
- `docs/oidc-tenant.md` still said the `User:` subject "must match
the `preferred_username` claim emitted by the issuer" — obsolete
after the claim switch. Rewrite to say `email`, and add a note
that the chart requests the `email` scope explicitly.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Real bug caught by an end-to-end OIDC login test on dev3: apiserver
accepts the JWT (auth succeeds) but the RBAC check fails with 403
because the CRB `User:` subject was `alice-oidc@test3.example` (from
`users[].email`) but the apiserver-visible identity was `alice-oidc`
(from `preferred_username`, which in the cozy realm is the Keycloak
*username*, not the email).
The design proposal example used `claim: preferred_username`, but
the operator-facing `users[].email` field explicitly asks for the
email address. In the cozy realm — verified — `preferred_username`
is the Keycloak username (`kvaps`, `i.okhotnikov`), so the two do
NOT coincide. Every user whose Keycloak username differs from their
email would silently 403 after a successful OIDC handshake.
Switch the AuthenticationConfiguration `claimMappings.username.claim`
to `email`, aligning the apiserver-visible identity with the
operator-declared `users[].email`. Every conformant OIDC provider
emits `email` — cozy realm does (verified), and any BYO
`CustomConfig` issuer that follows OIDC does too.
The switch is invisible to installers where `preferred_username`
already equals the email (a common Keycloak setup); for installers
where it doesn't, this fix is required for RBAC to work at all.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Caught by dev3 Job runtime: `kubectl patch --type=merge` fetches the
current object before computing the merged state, so the Role needs
`get` in addition to `patch` on the specific Secret name. Without it
the Job fails with `secrets "..." is forbidden: ... cannot get
resource "secrets"` and the OIDC kubeconfig stays on its placeholder.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Tighten the release Role — the `delete` verb on `keycloakclients` and
`keycloakclientscopes` now carries `resourceNames` naming exactly the
per-cluster client and its audience scope. Without the scope, the SA
could delete any Keycloak object in the tenant namespace.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Follow-up to the KCP cleanup extension: Kamaji's SSA field manager
takes ownership of `spec.apiServer.extraArgs` and
`spec.apiServer.extraVolumeMounts` too, not just
`spec.deployment.extraVolumes`. The previous normalize patch only
replaced extraVolumes; the two remaining fields still carried the
`--authentication-config=` flag and the corresponding mount even
after mode was toggled to None.
Extend the merge patch to REPLACE all three fields at once, sourced
from Helm values so user-supplied entries in
`controlPlane.apiServer.extra{Args,VolumeMounts,Volumes}` remain
intact — only the chart-owned OIDC-only entries drop off.
Verified on dev3: with the extended patch, mode=System → mode=None
leaves the tenant KCP with `extraArgs: []`,
`extraVolumeMounts: []`, and `deployment.extraVolumes` back to just
`talos-ca`, `talos-tls-cert` (or those + operator-supplied extras).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Follow-up to the KCP normalize commit: Helm-controller's SSA-based
apply also does not delete namespaced Secrets and cozy-realm
Keycloak resources when the new manifest omits them — the resources
stay as ghost owner-labeled-by-Helm objects. The dashboard picks
them up (via the cozyrds `secrets.include` list) and shows the
placeholder kubeconfig even for clusters where OIDC was explicitly
turned off.
Extend the `mode=None` normalize path to also `kubectl delete`:
- `<release>-oidc-kubeconfig` Secret
- `<release>-oidc-authn-config` Secret
- `<namespace>-<release>` KeycloakClient in the cozy realm
- `<namespace>-<release>-audience` KeycloakClientScope
The delete verbs are added to the release-owned Role scoped to the
specific resourceNames (Secrets) and the API group (Keycloak — the
EDP operator does not support resourceNames on cluster-scoped-ish
CRDs). No new privileges beyond what the chart itself would need to
render these objects.
Together with the previous KCP-normalize commit, `mode=System →
mode=None` now leaves ZERO chart-owned OIDC leftovers.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Kamaji's controller takes SSA field-ownership on
`KamajiControlPlane.spec.deployment.extraVolumes` when it syncs the
resource back from the TenantControlPlane, and Helm's subsequent
apply (with the same field-manager but a manifest that omits the
OIDC volume) then no longer removes it — the tenant apiserver Pod
keeps an unmounted-but-declared `authentication-config` volume
pointing at a Secret the chart already reaped. Harmless at runtime
(the volume is not mounted after the extraVolumeMounts fix), but the
CRD gets dirty and the operator wants zero leftovers.
Fix: the OIDC bootstrap Job is now rendered for ALL modes (not only
`System`/`CustomConfig`). When `mode=None` the Job takes the
normalize path — a single `kubectl patch --type=merge` that replaces
`spec.deployment.extraVolumes` with the chart baseline (talos-ca +
talos-tls-cert). We verified end-to-end that `kubectl patch
--type=merge` on this array REPLACES it deterministically (the
earlier Helm patch didn't).
For `mode=System`/`CustomConfig` the Job continues to apply
per-user ClusterRoleBindings inside the tenant cluster, prune stale
ones labelled by this release, and patch the pre-rendered
`<release>-oidc-kubeconfig` Secret with a real kubeconfig.
The Role gains `patch` on the specific `kamajicontrolplanes/<release>`
so the normalize path has just enough privilege; nothing else new.
The pre-delete cleanup Job stays conditional on `mode != None` — no
CRBs to clean up when OIDC was never enabled.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Two real bugs caught by the dev3 e2e test — both would have shipped
without live-cluster validation:
- `rm --force` in the bootstrap and cleanup Job scripts is rejected
by BusyBox `rm` (the alpine/k8s image uses BusyBox coreutils, not
GNU). The bootstrap Job on dev3 died at the prune step with
`rm: unrecognized option: force`. Switch to `rm -f` (works on
both GNU and BusyBox). Also swap
`grep --line-regexp --quiet --fixed-strings` for `grep -x -q -F`
for the same reason.
- Kamaji's `KamajiControlPlane` is a CRD, and Helm applies
JSON-merge-patch to CRDs. That patch does not remove array-typed
fields the new manifest simply omits — it leaves the live array
untouched. Toggling `oidc.mode: System` → `None` therefore left
the tenant kube-apiserver with a dangling `--authentication-config`
flag pointing at a Secret the chart had already reaped, and the
apiserver would refuse to start on the next kubelet restart.
Fix: always render `extraArgs` and `extraVolumeMounts` on the
KamajiControlPlane, emitting `[]` on the off-path so the patch
REPLACES the stale array. Verified end-to-end on dev3.
Adds two helm-unittest cases: the mode=None regression, and an
adjustment of the baseline default-values test.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Three items from the gemini-code-assist and coderabbitai passes that
survived the Phase-1 redesign:
- `oidc-rbac-job.yaml` (gemini HIGH): pre-delete cleanup uses
`kubectl --request-timeout=30s` in addition to `--timeout=30s`
on the delete verb, so an unreachable tenant apiserver cannot
stall the pre-delete hook on the HTTP round-trip itself. The
`|| echo WARNING` fallback still keeps a truly unreachable
cluster from blocking `helm uninstall`.
- `oidc-keycloak.yaml` + `_helpers.tpl` (gemini LOW): switch
`_cluster.{oidc-enabled,root-host}` reads to `dig` with an empty
map default so a missing `_cluster` key nil-safely emits an
empty string and hits the platform-OIDC guard instead of a
template panic on `.Values._cluster`.
- `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats`
(coderabbit MAJOR): add an existence-poll `timeout ... until
kubectl get secret ... ; do sleep ...; done` before the
`kubectl wait secret --for=jsonpath` line. Without it a runner
that reaches the wait before the HR has rendered the Secret
gets an immediate `NotFound` and the test flakes.
Adds one helm-unittest case covering the nil-safe `_cluster` path.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix caught by deploying to dev3: the EDP Keycloak operator
CRD v1 does not declare `standardFlow` (the canonical field is
`standardFlowEnabled`, default true), and `webUrl` is meaningful only
for clients with a real web UI (not for the kubectl oidc-login flow
which is keyed only by `redirectUris`).
Both lines violated the CRD's openAPIV3Schema and Helm refused the
install with `field not declared in schema`. Dropping the empty
`webUrl` and relying on `standardFlowEnabled`'s default (true) keeps
the manifest minimal and CRD-clean. helm-unittest did not catch this
because it does not validate templates against live CRD schemas.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix: the Role granted `["get", "patch"]` on the OIDC
kubeconfig Secret, but the bootstrap Job script never issues `kubectl
get` against the management apiserver — it only `kubectl patch`es the
pre-rendered Secret. The tenant-side `kubectl get clusterrolebindings`
uses the mounted admin kubeconfig, which is a separate trust path.
Tightens the SA's permission surface to exactly what the script uses.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix: the KeycloakClientScope rendered for the per-cluster
audience used `included.custom.audience` (raw string), but the
project convention — set in
`packages/system/keycloak-configure/templates/configure-kk.yaml` for
the legacy global `kubernetes-client` scope — is
`included.client.audience` (refers to a clientId in the same realm).
Both keys make Keycloak emit the audience claim with the same value
in our setup, but the `client.audience` form is the idiomatic one
when the audience IS a registered client in the realm.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix: the design proposal specifies
`name: authentication-config` and
`mountPath: /etc/kubernetes/authentication-config`, but the previous
revision had renamed both to `cozy-oidc-authn-config` and
`/etc/kubernetes/cozy-oidc` to dodge a collision with the volume
name used in `apiserver_authentication_test.yaml`. That test was
demonstrating the GENERIC user-supplied `extra*` passthrough — not
the OIDC use case — so rename the test fixture instead and restore
the design-spec names on the chart side.
The reservation guard against user-supplied volumes reusing
`authentication-config` only fires when `spec.oidc.mode` is not
None, so clusters with OIDC off retain the previous freedom to name
their own volumes anything.
Renames in `apiserver_authentication_test.yaml`: the user-supplied
volume name `authentication-config` becomes `tenant-byo-mount`; the
Secret name `tenant-authentication-config` and the apiserver flag
`--authentication-config=` are untouched.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fixes for four linked lifecycle issues:
- The post-install Job created `<release>-oidc-kubeconfig` via
`kubectl apply` from inside the script; the resulting Secret had
no Helm release labels and was orphaned on `helm uninstall`. Now
a release-owned placeholder Secret is rendered up front in
`oidc-kubeconfig-secret.yaml` and the Job patches its `data`
field at runtime (the tenant CA still has to come from the
Kamaji-issued admin kubeconfig, which appears asynchronously).
- The ServiceAccount/Role/RoleBinding for the bootstrap Job were
hook resources with `helm.sh/hook-delete-policy: before-hook-creation`,
so they leaked on `helm uninstall`. They are now regular release
resources (no hook annotations) and are reaped by `helm uninstall`
alongside everything else. Both the post-install bootstrap Job
and the pre-delete cleanup Job find a working SA.
- The previous design tried to add `hook-succeeded` to the SA, but
that would delete the SA right after the bootstrap hook
succeeded — leaving the pre-delete cleanup Job with no SA to
schedule. Release-owned resources sidestep this race entirely.
- The Role had `verbs: ["create"]` on `secrets` without
`resourceNames` (Kubernetes RBAC cannot scope `create` by name).
With the placeholder Secret pre-rendered the Job only needs
`get`/`patch` on the specific name — the broad `create` rule is
removed.
Also tightens the script: `set -euo pipefail` (was `set -eu`, which
silently swallowed `kubectl get` failures inside command
substitutions) and uses `kubectl patch --type=merge` against the
existing Secret instead of `kubectl create --dry-run | kubectl apply`.
Adds three helm-unittest cases for the placeholder Secret across all
three modes.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
cozystack-ci[bot]