Commit Graph
1128 Commits
Author SHA1 Message Date
Andrei Kvapil b99df11b36 fix(kubernetes): bump tenant worker Talos to v1.13.6 (#3269)
## What this PR does

Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux
6.18.24. That kernel predates the fixes for two related KVM x86
shadow-paging use-after-free bugs in `arch/x86/kvm/mmu/mmu.c`:
CVE-2026-46113, fixed upstream in
`0cb2af2ea66ad8ff195c156ea690f11216285bdf` and released in 6.18.30, and
CVE-2026-53359, fixed in `81ccda30b4e83d8f5cc4fd50503c44e3a33abfeb` and
released in 6.18.38. A worker's own kernel runs KVM whenever a tenant
enables nested virtualization inside the VM, and either bug lets a
nested guest escape into the worker.

This bumps the worker pin to Talos v1.13.6, which ships Linux 6.18.38.
Stable kernel branches are cumulative, so v1.13.6 is the first v1.13
patch release carrying both fixes. The pinned v1.13.0 ships 6.18.24 and
carries neither; v1.13.5 ships 6.18.36 and carries only CVE-2026-46113.
Nested virtualization stays available to tenants; the kernel bump is the
fix, not a capability removal.

The image-factory schematic is unchanged and resolves for the new
version: both the `openstack-amd64.raw.xz` artifact and the
`installer/<schematicID>:v1.13.6` manifest are served by the public
factory. Talos v1.13 supports Kubernetes 1.31 through 1.36, so the
chart's Talos-to-Kubernetes support window is unaffected and no tenant
`spec.version` needs to move.

Applying this rolls each worker pool once. The new pin changes the
rendered worker spec, which rotates the content hash in the
`KubevirtMachineTemplate` and `talos-reconcile` Job names, and CAPI
replaces the worker VMs to deliver the new kernel. The pinned hashes in
the chart tests are recomputed to match.

A second commit fixes the support-matrix lookup itself. The matrix dict
was keyed on the full patch version, so every Talos patch bump dropped
out of the dict and silently disabled the render-time guard — exactly
the mismatch the guard exists to catch. The lookup now derives the minor
from `talos.version`. The supported-Kubernetes window is unchanged;
unknown minors are still skipped, leaving compatibility to whoever
overrode the pin.

### Screenshots

Not applicable — no UI changes.

### Release note

```release-note
fix(kubernetes): bump tenant worker Talos to v1.13.6, picking up the Linux 6.18.38 kernel fixes for CVE-2026-46113 and CVE-2026-53359 (KVM shadow-paging use-after-free, nested-guest escape into the worker VM). Nested virtualization remains available. Applying this upgrade rolls each tenant worker pool once, as the worker template is renamed to carry the new image.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Updated the default Talos worker release to v1.13.6 for Kubernetes
cluster provisioning.
- Improved compatibility checks to apply consistently across Talos patch
releases within the same minor version.

- **Bug Fixes**
- Corrected generated resource naming expectations when storage settings
or GPU configuration change.

- **Tests**
- Added coverage confirming Kubernetes clusters render successfully with
Talos v1.13.6.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-10 11:34:48 +02:00
renovate[bot] 5b9b420ffe chore(deps): update container image & action digests 2026-07-10 05:29:23 +00:00
Aleksei Sviridkin b73860d4c9 fix(kubernetes): bump tenant worker Talos to v1.13.6
Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux 6.18.24.
That kernel predates both fixes for a pair of related KVM x86 shadow-paging
use-after-free bugs: CVE-2026-46113, fixed in 6.18.30, and CVE-2026-53359,
fixed in 6.18.38. A worker's own kernel runs KVM whenever a tenant enables
nested virtualization inside the VM, and either bug lets a nested guest
escape into the worker.

Talos v1.13.6 ships Linux 6.18.38. Stable kernel branches are cumulative, so
it is the first v1.13 patch release carrying both fixes: the pinned v1.13.0
ships 6.18.24 and carries neither, while v1.13.5 ships 6.18.36 and carries
only CVE-2026-46113. Nested virtualization stays available to tenants — the
kernel bump is the fix.

The new pin changes the rendered worker spec, so the content hash in the
KubevirtMachineTemplate and talos-reconcile Job names rotates and each
worker pool rolls once on upgrade. Pinned hashes in the chart tests are
recomputed accordingly.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-09 15:59:08 +03:00
Aleksei Sviridkin b420d3de02 fix(kubernetes): key Talos support matrix on minor version
The Talos<->Kubernetes support window is declared per Talos minor release,
and patch releases never move it. Keying the chart's matrix dict on the full
patch version meant every Talos patch bump fell out of the dict and silently
disabled the render-time guard — precisely the mismatch the guard exists to
catch.

Derive the minor from talos.version before the lookup, so the guard keeps
firing across patch releases of a known minor. Unknown minors are still
skipped, leaving compatibility to the operator who overrode the pin.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-09 15:47:22 +03:00
Aleksei Sviridkin c32ac68ab8 test(apps/kubernetes): anchor the trailing-slash imageFactoryURL assertion
Match the full rendered worker-disk URL shape (version + artifact + end anchor) like the sibling override assertion, so the trailing-slash normalization case cannot pass on a partial match.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-09 00:53:06 +03:00
Aleksei Sviridkin f60fe2270b Merge remote-tracking branch 'origin/main' into feat/kubernetes-talos-image-factory-override
Signed-off-by: Aleksei Sviridkin <f@lex.la>

# Conflicts:
#	packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
2026-07-09 00:46:33 +03:00
Myasnikov DaniilandClaude Opus 4.8 3eb182ddf1 feat(apps/kubernetes): make Talos image factory URL and installer repo overridable
Tenant worker VMs boot from a Talos raw disk image that CDI streams over
HTTP from the hardcoded public Talos Image Factory, and the in-guest
talos-reconcile Job pulls its installer image from the same hardcoded host.
Neither was overridable, so air-gapped / rate-limited / flaky-egress
environments could not point workers at a mirror (the README flagged this
as a Phase-2 gap), and CI had no way to avoid bulk-pulling the OS image
from the public internet per worker.

Add two chart values, both defaulting to the public factory so rendered
output is byte-identical and behaviour is unchanged:

- talos.imageFactoryURL (default https://factory.talos.dev) - base URL for
  the worker OS disk image streamed in by CDI.
- talos.installerRepository (default factory.talos.dev/installer) - OCI
  repository prefix for the in-guest installer image.

Regenerated values.schema.json, the aggregated-API Go types, the
kubernetes-rd cozyrds schema and the README via `make generate`; added
unittests for the override and trailing-slash trimming.

Refs #3231

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-08 18:11:23 +05:00
IvanHunters 6f5d826777 docs(apps/kubernetes): correct upgrade-safety mechanism from SSA to CAPI update-time defaulting
sircthulhu's post-approval note on PR#3233: both the cluster.yaml
comment and the nodegroups_default_test.yaml comment attribute the
upgrade-safety guarantee to Server-Side-Apply field-manager
ownership, but this chart ships as a HelmRelease through Flux
helm-controller, which applies via the Helm SDK three-way merge —
not SSA. The stated conclusion is correct but for the wrong reason
and would send a future maintainer chasing a non-existent field-
manager fight when debugging drift.

Rewrite both comments around the two independent CAPI mechanisms
that actually carry the guarantee:

  1. Create — the MachineDeployment defaulting webhook seeds
     spec.replicas from the min-size autoscaler annotation.
  2. Update — calculateMachineDeploymentReplicas preserves the
     existing live value whenever the incoming spec has
     replicas: nil (clamping into [min, max] on drift).

The chart guarantee reduces to "leave the field out of the rendered
manifest and let CAPI's update-time defaulting keep the live value
stable" — no field-manager ownership involved.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-08 10:08:26 +03:00
IvanHunters e23d89da03 test(apps/kubernetes): assert autoscaler max-size annotation on nonzero-minReplicas case
Symmetry with the zero-minReplicas neighbour, which already covers
both min-size and max-size. CAPI's calculateMachineDeploymentReplicas
defaults to 1 (not to minSize) if either annotation is missing on a
new MachineDeployment, so both keys are load-bearing on the
Create-path guarantee and worth asserting in both regression cases.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-08 09:54:47 +03:00
IvanHunters b3aac80830 fix(apps/kubernetes): omit chart-managed spec.replicas so cluster-autoscaler owns the field alone
Structural follow-up to the previous commit's `replicas: {{ $group.minReplicas }}`.
Emitting the field from the chart at all — even with the semantically-
correct value — keeps the helm-controller ↔ cluster-autoscaler field-
ownership fight alive: helm-controller applies with SSA + force-conflicts
(https://fluxcd.io/flux/components/helm/helmreleases/), so every routine
`helm upgrade` action re-asserts `replicas: {{ $group.minReplicas }}` and
drains whatever the autoscaler had patched in the meantime, with a real
disruption window on running clusters.

Drop the field from the manifest entirely. CAPI's MachineDeployment
defaulting webhook seeds `spec.replicas` from the
`cluster-api-autoscaler-node-group-min-size` annotation on Create (see
https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling),
so fresh installs still start with the intended minimum. From that
point on the autoscaler owns the field via SSA field-manager tracking
and chart apply never touches it again.

Behavioural summary:
  * Fresh install with `minReplicas: 0` → CAPI defaults `replicas: 0`
    on Create → no worker DataVolumes provisioned until scale-out.
  * Fresh install with `minReplicas: 2` → CAPI defaults `replicas: 2`
    on Create → byte-identical to pre-#3231 behaviour.
  * Existing customer (any minReplicas) → next chart upgrade does not
    touch spec.replicas — the autoscaler-held value is preserved
    unconditionally.

Regression tests updated to assert:
  * `notExists: spec.replicas` on the emitted MachineDeployment (the
    exact contract this commit establishes).
  * Autoscaler `min-size` / `max-size` annotations continue to carry
    the nodeGroup's declared bounds (so a future refactor cannot
    accidentally drop the wiring the defaulting webhook depends on).

helm-unittest suite stays at 173/173.

Ref: cozystack/cozystack#3231.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-07 15:54:46 +03:00
IvanHunters d30a2cfffb docs+test(apps/kubernetes): align nodeGroups default docs with actual zero-worker topology + regression-guard the replicas fix
Follow-ups from the self-review of #3233:

- `values.yaml` / helper docstring / regenerated README + schema +
  cozyrds openAPISchema + Go type doc: the previous "one
  ingress-nginx worker" contract was already misleading (helper
  default carried `minReplicas: 0` but the chart hardcoded
  `spec.replicas: 2`), and the replicas fix makes the docstring
  strictly false — an empty-nodeGroups install now provisions zero
  workers until either the cluster-autoscaler responds to an
  unschedulable Pod or the operator explicitly scales the group.
  Update the docstrings to describe the actual behaviour and to
  spell out the ingress-nginx-addon implication so operators are
  not surprised at addon-install time.

- `tests/nodegroups_default_test.yaml`: two regression cases pinning
  `MachineDeployment.spec.replicas == $group.minReplicas` for
  `minReplicas: 0` (the render-side OIDC bats path — zero worker
  DataVolumes provisioned) and for `minReplicas: 2` (the
  kubernetes-latest / kubernetes-previous bats path — byte-
  identical to pre-PR behaviour). Without these a future refactor
  can silently re-hardcode `replicas: 2` and turn no tests red.

  helm-unittest suite grows from 171 to 173, both new cases pass.

Ref: cozystack/cozystack#3231.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-07 15:51:01 +03:00
IvanHunters 6fd98dc510 fix(apps/kubernetes): honour nodeGroup minReplicas + drop nodeGroups from render-side OIDC tests
Closes #3231.

`packages/apps/kubernetes/templates/cluster.yaml` hardcoded
`MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's
`minReplicas`. The two render-side `kubernetes-oidc-{system,customconfig}`
bats — which set `minReplicas: 0` and delete their `Kubernetes` CR
without waiting for readiness — still provisioned two 20 GiB
`replicated` (DRBD) worker `DataVolume`s each. Their create/delete
churn overlapped the very next test, `kubernetes-previous`, and put
transient pressure on a LINSTOR satellite's ZFS pool: worker root
disks landed late, tenant nodes registered late, Cilium's CNI init
missed the 12-minute node-join deadline, and the test failed with
`cni plugin not initialized`.

Two changes, applied together — the chart fix is a no-op without
the test fix, and the test fix is a no-op without the chart fix,
because dropping `nodeGroups` falls back to the chart helper's
default `md0` (with `minReplicas: 0`).

**Chart**: `spec.replicas: {{ $group.minReplicas }}` so the field
matches the autoscaler annotations already emitted just above.
`minReplicas: 0` groups now genuinely scale from zero and do not
silently provision workers; groups with `minReplicas >= 2` (the
kubernetes-latest / kubernetes-previous bats CRs, the tests/values/
common.yaml default, the values.yaml default that gets templated
into the helper's `md0`) render byte-identical to before.

**Render-side OIDC tests**: drop the explicit `nodeGroups:` block
and `storageClass:` — falling back to the chart helper's default
`md0` (now with an effective `replicas: 0` after the chart change)
means the MachineDeployment renders but never creates a KubevirtMachine
and no DataVolume is provisioned. The tests only assert on the
KamajiControlPlane and the cozy-realm Keycloak objects, so removing
worker plumbing does not narrow their coverage.

Upgrade impact: any existing customer CR that already carries
`nodeGroups.md0.minReplicas: 0` (rare — the default in tests/values
and the chart helper both use it, but real user CRs almost always
have `minReplicas >= 1`) will see the MD's `spec.replicas` reconcile
from `2` down to `0` on the next chart apply. If the cluster-
autoscaler is enabled it will scale back up as soon as an
unschedulable pod appears; if it is not, the group intentionally
becomes empty (which is what `minReplicas: 0` requests literally).
This matches the semantics the autoscaler annotations were already
advertising — the hardcoded `2` was contradicting them.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-07 15:35:59 +03:00
Timofei Larkin 7b65cf2afd feat(platform): migrate to etcd-operator v1alpha2 (#2859)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note

```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for the newer etcd snapshot workflow
(BackupClass/BackupJob/RestoreJob) with in-place restore.
* Updated etcd chart/operator packaging and migration flow for the
v1alpha2 operator release.
* **Bug Fixes**
  * Switched readiness/condition handling to **Available**.
* Improved waits to cover all etcd member pods during restore and
scenario steps.
* **Documentation**
* Refreshed etcd backup/restore docs and example scenarios to match the
v1alpha2 APIs and parameters.
* **Tests**
* Updated e2e coverage to validate v1alpha2 operator runtime contracts
and replaced legacy backup-block tests with a snapshot round-trip test.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-06 19:04:59 +04:00
Andrey Kolkov cc5d688e15 feat(platform): migrate to etcd-operator v1alpha2
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
Ivan Okhotnikov 4e3a22992c docs(apps/kubernetes): fix stale preferred_username refs in OIDC template comments
The claim mapping switched to email in de0299f33, but two header
comments still described the earlier design:

  oidc-authn-config.yaml:15 — "Username claim is preferred_username"
  oidc-rbac-job.yaml:28    — "matched against preferred_username"

The rendered config (oidc-authn-config.yaml:49) maps
username.claim: email, the CRB subject is rendered from .email
(oidc-rbac-job.yaml:264), and the email-claim contract is pinned by
tests/oidc_test.yaml:260-261. Only the comments were stale; no code
change.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 7b8a330ad4 docs(apps/kubernetes): update users[].email docstring — email claim, not preferred_username
Gemini flagged that the `values.yaml` @field docstring on
`oidc.users[].email` still described the claim mapping as
`preferred_username`, misleading operators after the claim was
switched to `email` for real (see the earlier fix and its
end-to-end verification on dev3). Rewrite the description to
match the actual mechanic: `email` claim, requested via
`--oidc-extra-scope=email` in the chart-generated kubeconfig,
emitted by every conformant OIDC provider.

`make generate` re-propagates the update through README.md,
types.go and the cozyrds embedded schema.

Note on the other finding in the same review — Gemini also
flagged `audiences` as being incorrectly nested under `issuer`
in the AuthenticationConfiguration Secret. That is a
false positive: per the kube-apiserver v1beta1 spec
(apiserver.config.k8s.io/v1beta1, `JWTAuthenticator.Issuer`),
`audiences` IS a child of `issuer`, not a sibling of `jwt[i]`.
The current shape is correct and verified end-to-end on dev3
(alice/bob tokens carry `aud: [tenant-test-kubernetes-test3,
kubernetes]` and apiserver validates them successfully with
CRB-driven RBAC).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 0ae65f4dbe fix(apps/kubernetes): address gemini review — dig-based nil-safety + portable base64
Four findings from the gemini re-review:

  - `.Values.oidc.customConfig.secretRef.name` and
    `.Values.oidc.customConfig.config` — replaced direct chained
    access with `dig "customConfig" "…" "" .Values.oidc` so a CR that
    explicitly nulls `customConfig` doesn't nil-panic. `.Values.oidc`
    itself has a kubebuilder default of `{}` so it stays reachable.

  - `.Values.controlPlane.apiServer.extraArgs` in the collision guard
    — wrapped with `| default list` so an explicit null renders as an
    empty range rather than a nil deref. (`dig` on `.Values` itself
    doesn't work because it's `chartutil.Values`, not
    `map[string]interface{}`.)

  - `base64 -w 0` in the OIDC kubeconfig patch — replaced with
    `base64 | tr -d '\n'`. Both work on the current alpine/k8s image
    (BusyBox 1.37 supports `-w`), but the pipe form is portable back
    to older BusyBox releases and to macOS/BSD base64 (which does not
    know `-w`).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov b85cefc3e2 fix(apps/kubernetes): extend --oidc-* guard to catch --authentication-config too
Same class of upgrade regression: a customer who wired their own
structured AuthenticationConfiguration by hand via
`controlPlane.apiServer.extraArgs: [--authentication-config=...]`
would hit a duplicate-flag scenario the moment they toggle
`spec.oidc.mode` — kube-apiserver would receive two conflicting
--authentication-config= arguments and the last-wins behaviour is
undefined between apiserver versions.

Add a second render-time guard that fails when a legacy
--authentication-config= is present in the user-supplied extraArgs
AND spec.oidc.mode is not None, pointing at the CustomConfig
migration path (customConfig.config or customConfig.secretRef.name).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 9c73b6ef10 fix(apps/kubernetes): render-fail when spec.oidc.mode collides with legacy --oidc-* extraArgs
Reviewer flagged a real customer-upgrade regression: the previous
values.yaml docstring on `controlPlane.apiServer.extraArgs` documented
`--oidc-issuer-url`/`--oidc-client-id` as a valid usage. An existing
customer who wired OIDC by hand through that passthrough (following
the doc) would see their tenant kube-apiserver BOOT-FAIL the moment
they toggle `spec.oidc.mode` to `System` or `CustomConfig`:
kube-apiserver refuses to boot with both `--authentication-config`
(injected by the new spec.oidc) and any legacy `--oidc-*` flag —
`--authentication-config cannot be used with any of --oidc-*`.

Add a render-time guard: when `spec.oidc.mode != None`, scan the
user-supplied `controlPlane.apiServer.extraArgs` for any flag with
prefix `--oidc-` and fail with a clear message pointing at the
`spec.oidc.mode: CustomConfig` migration path (they can carry the
same issuer/audience inside `customConfig.config`).

When `spec.oidc.mode: None` the passthrough still works — customers
who manage OIDC entirely by hand keep working until they opt into
the new selector explicitly.

Also update the values.yaml docstring on `extraArgs` to steer OIDC
users to `spec.oidc.mode` and warn against the collision.

Two new helm-unittest cases: the collision fails, and mode=None with
legacy flags renders normally (backward compat).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 0bf26182da fix(apps/kubernetes,docs): explicit --oidc-extra-scope=email + doc alignment
Two follow-ups from the final adversarial review:

  - The chart-generated OIDC kubeconfig exec block requested only
    `--oidc-extra-scope=groups`. On a Keycloak realm where `email` is
    not a default client scope, kubectl-oidc-login would fetch a token
    that lacks the `email` claim — apiserver would then fail to
    identify the user, causing a silent auth failure. Add
    `--oidc-extra-scope=email` explicitly so the flow does not depend
    on realm-level default-scope hygiene.
  - `docs/oidc-tenant.md` still said the `User:` subject "must match
    the `preferred_username` claim emitted by the issuer" — obsolete
    after the claim switch. Rewrite to say `email`, and add a note
    that the chart requests the `email` scope explicitly.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov fe3cfef90a fix(apps/kubernetes): switch AuthenticationConfiguration username claim to email
Real bug caught by an end-to-end OIDC login test on dev3: apiserver
accepts the JWT (auth succeeds) but the RBAC check fails with 403
because the CRB `User:` subject was `alice-oidc@test3.example` (from
`users[].email`) but the apiserver-visible identity was `alice-oidc`
(from `preferred_username`, which in the cozy realm is the Keycloak
*username*, not the email).

The design proposal example used `claim: preferred_username`, but
the operator-facing `users[].email` field explicitly asks for the
email address. In the cozy realm — verified — `preferred_username`
is the Keycloak username (`kvaps`, `i.okhotnikov`), so the two do
NOT coincide. Every user whose Keycloak username differs from their
email would silently 403 after a successful OIDC handshake.

Switch the AuthenticationConfiguration `claimMappings.username.claim`
to `email`, aligning the apiserver-visible identity with the
operator-declared `users[].email`. Every conformant OIDC provider
emits `email` — cozy realm does (verified), and any BYO
`CustomConfig` issuer that follows OIDC does too.

The switch is invisible to installers where `preferred_username`
already equals the email (a common Keycloak setup); for installers
where it doesn't, this fix is required for RBAC to work at all.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 7257f8723e fix(apps/kubernetes): grant OIDC Secret get verb — kubectl patch --type=merge reads before writing
Caught by dev3 Job runtime: `kubectl patch --type=merge` fetches the
current object before computing the merged state, so the Role needs
`get` in addition to `patch` on the specific Secret name. Without it
the Job fails with `secrets "..." is forbidden: ... cannot get
resource "secrets"` and the OIDC kubeconfig stays on its placeholder.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 15efa44e7a fix(apps/kubernetes): scope Keycloak cleanup delete verb to the specific per-cluster client + scope
Tighten the release Role — the `delete` verb on `keycloakclients` and
`keycloakclientscopes` now carries `resourceNames` naming exactly the
per-cluster client and its audience scope. Without the scope, the SA
could delete any Keycloak object in the tenant namespace.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov de60efbd15 fix(apps/kubernetes): normalize Job also clears stale extraArgs and extraVolumeMounts
Follow-up to the KCP cleanup extension: Kamaji's SSA field manager
takes ownership of `spec.apiServer.extraArgs` and
`spec.apiServer.extraVolumeMounts` too, not just
`spec.deployment.extraVolumes`. The previous normalize patch only
replaced extraVolumes; the two remaining fields still carried the
`--authentication-config=` flag and the corresponding mount even
after mode was toggled to None.

Extend the merge patch to REPLACE all three fields at once, sourced
from Helm values so user-supplied entries in
`controlPlane.apiServer.extra{Args,VolumeMounts,Volumes}` remain
intact — only the chart-owned OIDC-only entries drop off.

Verified on dev3: with the extended patch, mode=System → mode=None
leaves the tenant KCP with `extraArgs: []`,
`extraVolumeMounts: []`, and `deployment.extraVolumes` back to just
`talos-ca`, `talos-tls-cert` (or those + operator-supplied extras).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 2705bfeba8 fix(apps/kubernetes): normalize Job force-deletes ALL chart-owned OIDC objects at mode=None
Follow-up to the KCP normalize commit: Helm-controller's SSA-based
apply also does not delete namespaced Secrets and cozy-realm
Keycloak resources when the new manifest omits them — the resources
stay as ghost owner-labeled-by-Helm objects. The dashboard picks
them up (via the cozyrds `secrets.include` list) and shows the
placeholder kubeconfig even for clusters where OIDC was explicitly
turned off.

Extend the `mode=None` normalize path to also `kubectl delete`:
  - `<release>-oidc-kubeconfig` Secret
  - `<release>-oidc-authn-config` Secret
  - `<namespace>-<release>` KeycloakClient in the cozy realm
  - `<namespace>-<release>-audience` KeycloakClientScope

The delete verbs are added to the release-owned Role scoped to the
specific resourceNames (Secrets) and the API group (Keycloak — the
EDP operator does not support resourceNames on cluster-scoped-ish
CRDs). No new privileges beyond what the chart itself would need to
render these objects.

Together with the previous KCP-normalize commit, `mode=System →
mode=None` now leaves ZERO chart-owned OIDC leftovers.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 0f60f0703a fix(apps/kubernetes): bootstrap Job normalizes stale KCP OIDC volume when mode=None
Kamaji's controller takes SSA field-ownership on
`KamajiControlPlane.spec.deployment.extraVolumes` when it syncs the
resource back from the TenantControlPlane, and Helm's subsequent
apply (with the same field-manager but a manifest that omits the
OIDC volume) then no longer removes it — the tenant apiserver Pod
keeps an unmounted-but-declared `authentication-config` volume
pointing at a Secret the chart already reaped. Harmless at runtime
(the volume is not mounted after the extraVolumeMounts fix), but the
CRD gets dirty and the operator wants zero leftovers.

Fix: the OIDC bootstrap Job is now rendered for ALL modes (not only
`System`/`CustomConfig`). When `mode=None` the Job takes the
normalize path — a single `kubectl patch --type=merge` that replaces
`spec.deployment.extraVolumes` with the chart baseline (talos-ca +
talos-tls-cert). We verified end-to-end that `kubectl patch
--type=merge` on this array REPLACES it deterministically (the
earlier Helm patch didn't).

For `mode=System`/`CustomConfig` the Job continues to apply
per-user ClusterRoleBindings inside the tenant cluster, prune stale
ones labelled by this release, and patch the pre-rendered
`<release>-oidc-kubeconfig` Secret with a real kubeconfig.

The Role gains `patch` on the specific `kamajicontrolplanes/<release>`
so the normalize path has just enough privilege; nothing else new.

The pre-delete cleanup Job stays conditional on `mode != None` — no
CRBs to clean up when OIDC was never enabled.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov f87f773399 fix(apps/kubernetes): always emit KCP extraArgs/extraVolumeMounts + rm -f for BusyBox
Two real bugs caught by the dev3 e2e test — both would have shipped
without live-cluster validation:

  - `rm --force` in the bootstrap and cleanup Job scripts is rejected
    by BusyBox `rm` (the alpine/k8s image uses BusyBox coreutils, not
    GNU). The bootstrap Job on dev3 died at the prune step with
    `rm: unrecognized option: force`. Switch to `rm -f` (works on
    both GNU and BusyBox). Also swap
    `grep --line-regexp --quiet --fixed-strings` for `grep -x -q -F`
    for the same reason.

  - Kamaji's `KamajiControlPlane` is a CRD, and Helm applies
    JSON-merge-patch to CRDs. That patch does not remove array-typed
    fields the new manifest simply omits — it leaves the live array
    untouched. Toggling `oidc.mode: System` → `None` therefore left
    the tenant kube-apiserver with a dangling `--authentication-config`
    flag pointing at a Secret the chart had already reaped, and the
    apiserver would refuse to start on the next kubelet restart.
    Fix: always render `extraArgs` and `extraVolumeMounts` on the
    KamajiControlPlane, emitting `[]` on the off-path so the patch
    REPLACES the stale array. Verified end-to-end on dev3.

Adds two helm-unittest cases: the mode=None regression, and an
adjustment of the baseline default-values test.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 5325468c96 fix(apps/kubernetes,e2e): address bot-review findings on Phase 1 rewrite
Three items from the gemini-code-assist and coderabbitai passes that
survived the Phase-1 redesign:

  - `oidc-rbac-job.yaml` (gemini HIGH): pre-delete cleanup uses
    `kubectl --request-timeout=30s` in addition to `--timeout=30s`
    on the delete verb, so an unreachable tenant apiserver cannot
    stall the pre-delete hook on the HTTP round-trip itself. The
    `|| echo WARNING` fallback still keeps a truly unreachable
    cluster from blocking `helm uninstall`.

  - `oidc-keycloak.yaml` + `_helpers.tpl` (gemini LOW): switch
    `_cluster.{oidc-enabled,root-host}` reads to `dig` with an empty
    map default so a missing `_cluster` key nil-safely emits an
    empty string and hits the platform-OIDC guard instead of a
    template panic on `.Values._cluster`.

  - `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats`
    (coderabbit MAJOR): add an existence-poll `timeout ... until
    kubectl get secret ... ; do sleep ...; done` before the
    `kubectl wait secret --for=jsonpath` line. Without it a runner
    that reaches the wait before the HR has rendered the Secret
    gets an immediate `NotFound` and the test flakes.

Adds one helm-unittest case covering the nil-safe `_cluster` path.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov fa1fdd8c0e fix(apps/kubernetes): drop EDP-Keycloak-unknown fields from KeycloakClient
Self-review fix caught by deploying to dev3: the EDP Keycloak operator
CRD v1 does not declare `standardFlow` (the canonical field is
`standardFlowEnabled`, default true), and `webUrl` is meaningful only
for clients with a real web UI (not for the kubectl oidc-login flow
which is keyed only by `redirectUris`).

Both lines violated the CRD's openAPIV3Schema and Helm refused the
install with `field not declared in schema`. Dropping the empty
`webUrl` and relying on `standardFlowEnabled`'s default (true) keeps
the manifest minimal and CRD-clean. helm-unittest did not catch this
because it does not validate templates against live CRD schemas.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 073c0c717a fix(apps/kubernetes): drop unused get verb from OIDC bootstrap Role
Self-review fix: the Role granted `["get", "patch"]` on the OIDC
kubeconfig Secret, but the bootstrap Job script never issues `kubectl
get` against the management apiserver — it only `kubectl patch`es the
pre-rendered Secret. The tenant-side `kubectl get clusterrolebindings`
uses the mounted admin kubeconfig, which is a separate trust path.

Tightens the SA's permission surface to exactly what the script uses.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 5fcb40f07e fix(apps/kubernetes): switch audience mapper to included.client.audience
Self-review fix: the KeycloakClientScope rendered for the per-cluster
audience used `included.custom.audience` (raw string), but the
project convention — set in
`packages/system/keycloak-configure/templates/configure-kk.yaml` for
the legacy global `kubernetes-client` scope — is
`included.client.audience` (refers to a clientId in the same realm).

Both keys make Keycloak emit the audience claim with the same value
in our setup, but the `client.audience` form is the idiomatic one
when the audience IS a registered client in the realm.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov d13e2c586f fix(apps/kubernetes): align OIDC mount + path with the design spec
Self-review fix: the design proposal specifies
`name: authentication-config` and
`mountPath: /etc/kubernetes/authentication-config`, but the previous
revision had renamed both to `cozy-oidc-authn-config` and
`/etc/kubernetes/cozy-oidc` to dodge a collision with the volume
name used in `apiserver_authentication_test.yaml`. That test was
demonstrating the GENERIC user-supplied `extra*` passthrough — not
the OIDC use case — so rename the test fixture instead and restore
the design-spec names on the chart side.

The reservation guard against user-supplied volumes reusing
`authentication-config` only fires when `spec.oidc.mode` is not
None, so clusters with OIDC off retain the previous freedom to name
their own volumes anything.

Renames in `apiserver_authentication_test.yaml`: the user-supplied
volume name `authentication-config` becomes `tenant-byo-mount`; the
Secret name `tenant-authentication-config` and the apiserver flag
`--authentication-config=` are untouched.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 48fb4fe520 fix(apps/kubernetes): release-own the OIDC SA/Role/RoleBinding + pre-render the kubeconfig Secret
Self-review fixes for four linked lifecycle issues:

  - The post-install Job created `<release>-oidc-kubeconfig` via
    `kubectl apply` from inside the script; the resulting Secret had
    no Helm release labels and was orphaned on `helm uninstall`. Now
    a release-owned placeholder Secret is rendered up front in
    `oidc-kubeconfig-secret.yaml` and the Job patches its `data`
    field at runtime (the tenant CA still has to come from the
    Kamaji-issued admin kubeconfig, which appears asynchronously).
  - The ServiceAccount/Role/RoleBinding for the bootstrap Job were
    hook resources with `helm.sh/hook-delete-policy: before-hook-creation`,
    so they leaked on `helm uninstall`. They are now regular release
    resources (no hook annotations) and are reaped by `helm uninstall`
    alongside everything else. Both the post-install bootstrap Job
    and the pre-delete cleanup Job find a working SA.
  - The previous design tried to add `hook-succeeded` to the SA, but
    that would delete the SA right after the bootstrap hook
    succeeded — leaving the pre-delete cleanup Job with no SA to
    schedule. Release-owned resources sidestep this race entirely.
  - The Role had `verbs: ["create"]` on `secrets` without
    `resourceNames` (Kubernetes RBAC cannot scope `create` by name).
    With the placeholder Secret pre-rendered the Job only needs
    `get`/`patch` on the specific name — the broad `create` rule is
    removed.

Also tightens the script: `set -euo pipefail` (was `set -eu`, which
silently swallowed `kubectl get` failures inside command
substitutions) and uses `kubectl patch --type=merge` against the
existing Secret instead of `kubectl create --dry-run | kubectl apply`.

Adds three helm-unittest cases for the placeholder Secret across all
three modes.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov b7c553e653 docs(oidc): Phase-1 operator guide + chart NOTES
Add the operator-facing guide for the new selector — what the three
modes mean, what the chart provisions, how a user logs in, what the
failure modes are, and what is intentionally out of scope (per-tenant
realms, federation, cross-cluster SSO). The architectural rationale
points back to cozystack/community#24.

Update the chart NOTES.txt to print the OIDC kubeconfig retrieval
snippet, the per-cluster client/issuer values, and the listed
`users[]` roster — only when OIDC is on, otherwise the previous
admin-kubeconfig hint is unchanged.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 5bf7219f73 test(apps/kubernetes): helm-unittest cases for OIDC modes
Cover the three branches of the new selector API:

  - `mode: None` (default) — every OIDC template is empty, the
    KamajiControlPlane has no extra args and the standard two talos
    volumes, the bootstrap Job is not rendered.
  - `mode: System` — per-cluster `KeycloakClient` + audience scope
    in the `cozy` realm, namespace-prefixed clientId, PKCE attribute
    on the client, soft-render when the EDP Keycloak operator CRDs
    are absent, the AuthenticationConfiguration Secret pinned to the
    cozy issuer, the chart-owned `--authentication-config` flag and
    `cozy-oidc-authn-config` mount injected into the KCP, the
    bootstrap Job carrying `policy.cozystack.io/allow-to-apiserver`,
    and one ClusterRoleBinding per `users[]` entry with the right
    ClusterRole (`cluster-admin` vs `view`).
  - `mode: CustomConfig` — inline config written verbatim; secretRef
    path mounted on the KCP without a chart-owned Secret; no
    Keycloak objects emitted; mutual-exclusion and missing-source
    guards both hard-fail.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov c5295d8bad feat(apps/kubernetes): bootstrap Job applies per-user CRBs and writes OIDC kubeconfig Secret
Add a post-install,post-upgrade Helm-hook Job that reconciles the
operator-supplied `users[]` list into the tenant cluster:

  - For every entry, apply a ClusterRoleBinding labelled by the
    release (managed-by=cozystack-oidc, instance=<release>).
    `role: admin` maps to `cluster-admin`; `role: view` maps to
    `view`. The subject is the literal `email` value, matched
    against the issuer's `preferred_username` claim.
  - Prune any orphan CRBs with the same labels — removing a user
    from `users[]` revokes their kubectl access on the next loop.

For `mode: System` the same Job also writes
`<release>-oidc-kubeconfig` on the management cluster: a ready
kubeconfig with the per-cluster apiserver endpoint, the tenant CA
extracted from the Kamaji-issued admin kubeconfig, and a
`kubectl oidc-login` exec block keyed to the per-cluster client.

A pre-delete cleanup Job removes the labelled CRBs before the
control plane is torn down (best-effort: the CRBs disappear with
the tenant cluster anyway).

Pod is labelled `policy.cozystack.io/allow-to-apiserver: "true"`
so Cilium lets it reach the konnectivity tunnel.

ServiceAccount/Role/RoleBinding are minimum-privilege — only the
specific admin-kubeconfig and OIDC kubeconfig Secrets are listed.

Renders nothing for `mode: None`.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov cef556ba4c feat(apps/kubernetes): wire structured AuthenticationConfiguration
Render a per-release Secret carrying the structured kube-apiserver
`AuthenticationConfiguration` (apiserver.config.k8s.io/v1beta1) and
mount it into the KamajiControlPlane.

For `mode: System` the Secret is chart-owned: one JWT entry pointing
at the `cozy` realm issuer with the per-cluster audience, plus the
standard `preferred_username`/`groups` claim mappings. For `mode:
CustomConfig` the chart either writes the operator-supplied inline
config into a Secret, or — when `secretRef.name` is set — mounts an
existing operator-owned Secret directly (mutually exclusive; the
chart fails the render if neither or both are set).

The `--authentication-config=` flag, mount, and volume on the
KamajiControlPlane are kept orthogonal to the user-supplied
`controlPlane.apiServer.extra*` knobs: the OIDC additions are
appended, and the chart-managed volume name `cozy-oidc-authn-config`
is reserved (only when `mode` is not None) so user-supplied volumes
cannot collide.

Switches from the legacy `--oidc-*` flag form to the file form so
multiple issuers, inline private-CA PEM, and future Phase-2 issuers
extend the same Secret without rewiring the chart.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 83980c8c47 feat(apps/kubernetes): render per-cluster Keycloak client + audience scope
For `oidc.mode: System`, emit a per-cluster public `KeycloakClient`
plus a per-cluster `KeycloakClientScope` inside the platform `cozy`
realm. The scope carries an `oidc-audience-mapper` that pins
`id_token.aud` to the per-cluster clientId, so a token minted for
cluster A is rejected by cluster B's apiserver.

The clientId is namespace-prefixed (`<ns>-<release>`) to stay unique
across the realm — two clusters with the same name in different
tenants would otherwise collide.

The client is `public: true`, redirects locked to localhost:8000 /
localhost:18000 (the kubelogin/oidc-login defaults), with PKCE
required via the standard Keycloak attribute. `defaultClientScopes`
includes both `groups` and the per-cluster audience scope.

A render-time guard fails fast when `oidc.mode: System` is set but
`_cluster.oidc-enabled` is not `"true"` — the Keycloak operator CRDs
are otherwise absent and the chart would silently no-op.

Nothing is rendered for `mode: CustomConfig` or `mode: None`.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 5950450758 feat(apps/kubernetes): add Kubernetes.spec.oidc selector API
Introduce a flat selector — `mode: System | CustomConfig | None` —
plus a `users[]` list and a `customConfig` block for tenant-supplied
AuthenticationConfiguration. Default `None` preserves today's
behavior.

The `System` mode trusts the platform `cozy` realm via a per-cluster
public client; `CustomConfig` accepts an inline `AuthenticationConfiguration`
or a Secret reference (mutually exclusive). The `users[]` list drives
per-user ClusterRoleBindings inside the tenant cluster, with two
roles: `admin` and `view`.

Regenerates deepcopy, JSON schema, README, and the kubernetes-rd
cozyrds embedded schema.

Implements Phase 1 API of the per-tenant OIDC design proposal
(cozystack/community#24).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Matthieu 50c4080ba3 [kubernetes] Add CPU/memory limits to csi-resizer sidecar
The csi-resizer container was the only sidecar in the kcsi-controller
Deployment without resources.limits, triggering a 'no CPU limit'
conformance warning. Align it with the other six CSI sidecars in the
same Deployment (limits 512m/512Mi, requests 125m/128Mi) so capacity
planning and audits stay uniform.

Signed-off-by: Matthieu <matthieu@hidora.com>
2026-07-03 19:04:30 +03:00
IvanHunters 1baf126b1a fix(tenant): inherit full ancestor label chain from parent namespace (#2912)
## Summary

`packages/apps/tenant/templates/namespace.yaml` labels every tenant
namespace
with `tenant.cozystack.io/<ancestor>` for each of its ancestors. These
labels
drive the `<ancestor>-egress` `CiliumClusterwideNetworkPolicy`
(`networkpolicy.yaml`): an ancestor tenant's pods may reach a namespace
only if
that namespace carries `tenant.cozystack.io/<ancestor>`.

The chain was derived by splitting `.Release.Namespace`, which only ever
reached
one level up. A direct child of `tenant-root` got `tenant-root` (its
parent
namespace *is* `tenant-root`), but a grandchild — `tenant-ktj-htdev`
created
under `tenant-ktj` — got only `tenant-ktj` and dropped `tenant-root`
entirely.
Without `tenant.cozystack.io/tenant-root` on a nested namespace, the
root
ingress controller cannot reach the proxy service in a nested
`kubernetes`
tenant, surfacing as `upstream timed out (110) while connecting to
upstream` in
the nginx logs.

## Fix

Derive the ancestor chain deterministically from the namespace name
instead of
from any cluster lookup. Every tenant descends from `tenant-root`, so
that label
is emitted unconditionally; the remaining ancestors are encoded in the
namespace
name itself, because `tenant.name` builds each child as
`<parent-namespace>-<word>` — so every progressive dash-prefix of the
name is a
real ancestor namespace (`tenant-ktj-htdev` yields `tenant-ktj` and
`tenant-ktj-htdev`). The logic lives in a new
`tenant.ancestorTenantLabels`
helper:

```yaml
labels:
  {{- include "tenant.ancestorTenantLabels" $tenantName | nindent 4 }}
```

This needs no cluster state, renders identically offline (so it is
unit-testable), and converges on every reconcile regardless of the order
in
which parent and child `HelmRelease`s reconcile — unlike a
parent-namespace
`lookup`, which helm-controller does not re-render on and which is
ordering-dependent. The `ownerReference` `lookup` is kept but made
non-fatal:
during a real reconcile the parent namespace always exists (the tenant
`HelmRelease` and its `cozystack-values` Secret live in it), so the
`ownerReference` is always emitted; when the lookup is empty (offline /
unit
test) it is simply omitted so the labels still render.

Migration `49` backfills the labels on existing clusters so they heal on
upgrade
regardless of per-tenant `HelmRelease` state (e.g. suspended or failed
releases), and `migrations.targetVersion` is bumped to `50` so the
runner
iterates it. The migration derives the same chain from each namespace
name,
is idempotent (`kubectl label --overwrite`) and best-effort
(per-namespace
failures are tolerated so a transient apiserver error cannot abort the
upgrade).

## Test plan

- [x] `helm template packages/apps/tenant` at depth 1/2/3 — emits
exactly the
expected ancestor set at each depth: depth-1 `{tenant-root,
tenant-ktj}`,
depth-2 adds `tenant-ktj-htdev`, depth-3 adds `tenant-ktj-htdev-foo`. No
      missing, no stray labels.
- [x] `helm unittest packages/apps/tenant` — new
`tests/namespace_ancestor_labels_test.yaml` pins the chain at depths
1/2/3
      plus a negative assertion; whole suite green.
- [x] `bats hack/cozystack-version-stamp.bats` — migration `49` stamps
the
version through the shared `stamp_cozystack_version` helper (all 10
tests
pass, including the guard that migrations >= 42 must use the helper).
- [x] Migration `49` ancestor derivation verified under `/bin/sh` to
match the
helm output byte-for-byte, and dry-runs cleanly under `set -euo
pipefail`.
- [ ] Live cluster: create a 3-level tenant (`root -> ktj -> htdev`),
verify
      `tenant-ktj-htdev` has all three `tenant.cozystack.io/*` labels.
- [ ] Live cluster: create a `kubernetes` tenant under `htdev` with
`addons.ingressNginx.hosts`, verify an external HTTP request reaches the
tenant ingress controller via the root-ingress proxy (no `upstream timed
      out`).

## Closes

- Closes #2810
- Closes #2171

```release-note
fix(tenant): tenant namespaces now carry the full `tenant.cozystack.io/<ancestor>` label chain (including `tenant-root`) at any nesting depth, restoring root-ingress reachability to nested tenants; a migration backfills the labels on existing clusters
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved tenant namespace label inheritance across multi-level tenant
hierarchies, ensuring all ancestor tenant labels are consistently
applied.
* **New Features**
* Standardized ancestor label generation to reliably include the full
tenant chain for derived tenant namespaces.
* **Tests**
* Added regression coverage for ancestor label presence and absence at
multiple hierarchy depths.
* **Chores**
* Added a migration to backfill missing ancestor labels for existing
tenant namespaces during platform upgrades.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-03 16:12:24 +03:00
IvanHunters 1ac9398a4b fix(tenant): emit full ancestor label chain on tenant namespaces
The tenant.cozystack.io/<ancestor> namespace labels drive the
<ancestor>-egress CiliumClusterwideNetworkPolicy, which lets an ancestor
tenant reach a descendant namespace only when the descendant carries
tenant.cozystack.io/<ancestor>. namespace.yaml derived that chain by
splitting .Release.Namespace, which only reached one level up: a direct
child of tenant-root got tenant-root, but a grandchild (tenant-ktj-htdev
under tenant-ktj) got only tenant-ktj and dropped tenant-root entirely.
The root ingress controller in tenant-root then could not reach nested
tenants, surfacing as upstream timeouts.

Derive the chain deterministically from the namespace name instead. Every
tenant descends from tenant-root, and tenant.name builds each child as
<parent-namespace>-<word>, so every progressive dash-prefix of the name is
a real ancestor namespace. This needs no cluster lookup, renders offline
(so it is unit-testable), and converges on every reconcile regardless of
parent/child HelmRelease ordering. The ownerReference lookup is made
non-fatal so the template still renders when the parent namespace lookup
is empty; in a real reconcile the parent always exists and the
ownerReference is always emitted.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-03 09:51:49 +03:00
Myasnikov DaniilandClaude Opus 4.8 2b1d3e904d fix(kubernetes): gate vmop cert-manager-crds dependency on certManager.enabled
The victoria-metrics-operator HelmRelease is gated on
monitoringAgents.enabled but listed cert-manager-crds in dependsOn
unconditionally, while the cert-manager-crds HelmRelease is only
created when certManager.enabled is true. The valid combination
monitoringAgents.enabled=true + certManager.enabled=false therefore
left vmop blocked on a non-existent dependency
("...cert-manager-crds ... not found"), cascading to monitoring-agents
and vertical-pod-autoscaler. Gate the dependency on the same condition
that creates it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-03 11:18:46 +05:00
scooby87 71ca2b1bae fix(mariadb): re-guard replication so single-replica passes the operator webhook (#3180)
## What this PR does

Fixes a regression where **single-replica MariaDB can no longer be
created**.

[PR #2279](https://github.com/cozystack/cozystack/pull/2279) made
`replication.enabled: true` unconditional in the MariaDB CR so the
operator would always create `<release>-primary` / `<release>-secondary`
Services (expected by the dashboard RBAC and the ApplicationDefinition).
But the vendored mariadb-operator's validating webhook
(`vmariadb-v1alpha1.kb.io`) **rejects `replication.enabled: true` when
`spec.replicas == 1`** — replication needs a primary + at least one
replica. As a result, single-replica installs never became Ready (the
HelmRelease failed at admission).

The webhook rule lives in the operator's Go code (not in the vendored
CRD — there is no CEL rule tying replication to replica count), so the
failing path is reproduced by an e2e test rather than a schema
assertion.

### Fix — re-guard replication, teach consumers about the bare service

The operator provisions a bare `<release>` Service in **both**
topologies; `-primary`/`-secondary` exist **only** when replication is
on (`replicas>1`). Instead of forcing replication for everyone, this PR
lets replication track the replica count and makes the service-name
consumers handle the single-replica bare service:

- **`mariadb.yaml`** — gate `replication` on `replicas>1`; restore
single-replica external access via `service.type: LoadBalancer`, and
keep `primaryService: LoadBalancer` for `replicas>1` only. (Effectively
reverts #2279's two hunks.)
- **`backup-cronjob.yaml`** — single-replica dumps target the bare
service (no `-secondary` exists without replication); `replicas>1` still
offloads to `-secondary`.
- **`dashboard-resourcemap.yaml`** — always grant RBAC on the bare
service, plus `-primary`/`-secondary` when `replicas>1`, matching what
the ApplicationDefinition lists.
- **`mariadb-rd`** — list the bare service in `services.include` so
single-replica installs are visible in the dashboard.
- **`hack/e2e-apps/mariadb.bats`** — new `Create single-replica MariaDB`
test that reproduces the webhook rejection and guards against
regression.

Verified: `helm template` at `replicas=1`/`2` renders the intended
CR/RBAC/backup host; `helm unittest` passes; `update-crd.sh` keeps the
RD stable; `select-e2e.sh` maps the change to the mariadb suite.

### Release note

```release-note
fix(mariadb): single-replica instances can be created again — replication is now enabled only for replicas>1 (matching the operator webhook), while the dashboard, RBAC, and backups correctly target the bare service for single-replica setups
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an end-to-end test covering MariaDB deployments with a single
replica, including validation of the main service, endpoints,
StatefulSet replica count, and metrics components.

* **Bug Fixes**
* Updated MariaDB chart behavior to correctly handle single-replica vs
multi-replica configurations for external access, replication settings,
and backup host selection.
* Refined dashboard RBAC and secret inclusion so access applies to the
main MariaDB service in addition to replica-specific services.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 22:54:14 +03:00
Алексей Артамонов a76c5929fd fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry
The post-delete cleanup Jobs in the harbor and mariadb charts hardcoded
`docker.io/clastix/kubectl:v1.32` — a moving tag pulled directly from
Docker Hub. This broke build reproducibility (the tag can be re-pushed),
was a supply-chain risk, and could not be satisfied on air-gapped /
mirrored clusters, where it failed at the worst moment: during uninstall,
when the post-delete hook runs and stalls release teardown.

Add a canonical `cozy-lib.image` helper that prefixes a registry-relative
image path with `cozy-lib.images-registry` when set and returns it
unchanged (no leading "/") when empty, so standard installs resolve from
the default registry and mirrored installs resolve internally. Consume it
from both cleanup hooks with the image digest-pinned to the immutable
multi-arch index sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c
(the same digest already pinned for postgres-operator's webhook hook).

Keep the pin fresh with a custom Renovate manager: the repo disables the
helm-values manager (enabledManagers), and the ref is built at render time
by the helper rather than being a Dockerfile/go.mod dependency, so the
built-in managers cannot see it. A regex manager over the hook templates
tracks clastix/kubectl by tag+digest — mirroring the existing extra/etcd
manager — so the supply-chain pin does not silently age.

Also harden `cozy-lib.images-registry` against a nil .Values._cluster so
the helper never crashes the render — relevant precisely because the hook
renders during teardown. Helm unit tests assert the rendered image (routed,
digest-pinned, no leading slash) by pattern rather than an exact digest, so
a Renovate bump keeps them green.

Closes #3088

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-02 17:44:59 +03:00
Алексей Артамонов c7e8f4d975 fix(mariadb): re-guard replication so single-replica passes the operator webhook
PR #2279 made `replication.enabled: true` unconditional so the operator would
always create -primary/-secondary services. But the mariadb-operator validating
webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1,
so single-replica MariaDB could no longer be created — the HelmRelease never
became Ready.

Re-guard replication on replicas>1 and teach the service-name consumers about
the bare <release> service the operator provisions in both topologies:

- mariadb.yaml: gate `replication` on replicas>1; restore single-replica
  external access via `service.type: LoadBalancer` and keep `primaryService`
  LoadBalancer for replicas>1 only.
- backup-cronjob.yaml: route single-replica dumps to the bare service
  (no -secondary exists without replication).
- dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus
  -primary/-secondary when replicas>1, matching the ApplicationDefinition.
- mariadb-rd: list the bare service in services.include so single-replica
  installs are visible in the dashboard.

Adds a single-replica e2e test that reproduces the webhook rejection and
guards against regression.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-02 15:48:29 +03:00
scooby87 fdbe6b299c fix(clickhouse)!: clean up orphaned keeper and data PVCs on delete (#3057) (#3072)
## What this PR does

After deleting a `ClickHouse` app CR, data PVCs were left orphaned in
the tenant
namespace and consumed storage permanently:

- the keeper PVC `default-chk-<release>-keeper-cluster1-0-0-0`, backing
a
`ClickHouseKeeperInstallation` reconciled by the
clickhouse-keeper-operator;
- the main ClickHouse data / log PVCs.

The keeper operator does not delete data PVCs when the CR is removed
(data-safety
default) and the CRD exposes no `persistentVolumeClaimRetentionPolicy`
equivalent,
so the volumes leaked.

This adds a `post-delete` cleanup hook (modeled on the existing MariaDB
hook at
`packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml`) that deletes
every PVC
carrying the release label `app.kubernetes.io/instance=<release>`. It
also adds
that label to the keeper `volumeClaimTemplates` in `chkeeper.yaml`: the
keeper
PVC previously had **no** release label (the chart set labels only on
the keeper
pod template, not the PVC), so the operator never stamped one and a
single
selector would have missed it — leaving the reported leak unfixed.

### Existing releases (label backfill migration)

The altinity clickhouse-keeper-operator only stamps
`volumeClaimTemplates.metadata.labels`
onto PVCs it creates **after** the chart change; it never re-labels PVCs
that
already exist. So for releases installed before this fix, the keeper PVC
stays
unlabeled and the cleanup selector would still miss it. To close #3057
for
existing customers too, this PR adds migration `48` (`48 -> 49`,
`packages/core/platform/images/migrations/migrations/48`) that
back-fills
`app.kubernetes.io/instance=<release>` onto pre-existing keeper PVCs,
and bumps
`migrations.targetVersion` to `49`. The migration:

- routes its version stamp through the shared `lib/cozystack-version.sh`
helper
(`stamp_cozystack_version 49`), per the `cozystack-version-stamp.bats`
convention;
- no-ops when the keeper CRD is absent, and tolerates a transient
apiserver error
on the keeper list (`|| true`) so it cannot abort a platform-wide
upgrade — the
backfill is best-effort (worst case is the pre-existing leak, no worse
than today);
- is idempotent (skips PVCs already labeled).

The main ClickHouse data/log PVCs have carried
`app.kubernetes.io/instance` since
2025-06-03 (well before the migration era), so they need no backfill.

### Hook failure handling

The cleanup hook does **not** mask failures: `kubectl delete` failing
makes the
Job exit non-zero. The **Job** uses `helm.sh/hook-delete-policy:
before-hook-creation`
only (no `hook-succeeded`) so a failed cleanup Job and its logs survive
for
debugging; the **ServiceAccount/Role/RoleBinding** keep
`before-hook-creation,hook-succeeded` so the RBAC artifacts are
reclaimed after a
successful uninstall instead of lingering in the namespace.

### Scope

The hook intentionally reclaims **only PVCs** of the release. Other
release
state (Secrets, ConfigMaps) is owned/garbage-collected through the
normal Helm
uninstall path and is out of scope here.

> [!WARNING]
> **This is destructive — deleting a ClickHouse app now permanently
deletes its data.**
> The hook removes **all** PVCs of the release: the keeper PVC **and**
the main
> ClickHouse data and log PVCs. After deletion nothing is left behind to
recover
> data from. This is intentional and matches the existing MariaDB
cleanup-hook
> behaviour and the general cozystack model (deleting a managed app
reclaims its
> storage). Back up before deleting.

Fixes #3057.

### Release note

```release-note
fix(clickhouse): garbage-collect the keeper and data/log PVCs when a ClickHouse app is deleted, so no orphaned volumes are left behind. A migration back-fills the release label onto keeper PVCs from releases installed before this fix so they are cleaned up too. WARNING: deleting a ClickHouse app now permanently removes all of its PVCs (keeper + data + logs) and the data they hold.
```
2026-07-02 15:31:01 +03:00
Aleksei Sviridkin 9e89ae84ac fix(kubernetes): default worker disk storageClass to the application storageClass (#3129)
## What this PR does

Worker node VMs in a tenant Kubernetes cluster set `evictionStrategy:
External`, so they live-migrate. Live migration requires the worker
system DataVolume to be ReadWriteMany, and linstor-csi only grants RWX
on a DRBD-backed StorageClass. When a node group leaves `storageClass`
empty, the worker DataVolume template previously omitted
`storageClassName` and fell through to the management cluster's default
StorageClass — which is operator-configured and not guaranteed to be
DRBD-backed. On any cluster whose default StorageClass is not a
replicated/DRBD class, the worker disk lands on a non-RWX class and live
migration breaks.

This changes the empty-`storageClass` fallback from the management
cluster default to the application-level `storageClass` (default
`replicated`, a DRBD/LINSTOR class): `{{- with .group.storageClass |
default $.Values.storageClass }}`. A per-node-group `storageClass` still
wins when set; when both the node group and the application
`storageClass` are empty, `storageClassName` is omitted so the cluster
default still applies (explicit opt-out preserved).

Doc and codegen surfaces (`types.go`, `values.yaml`,
`values.schema.json`, `README.md`, and the generated `kubernetes-rd`
ResourceDefinition) are regenerated in lockstep with the new field
description. Chart unit tests cover the full contract: fallback to the
application class, a custom application class, per-node-group override
winning, and the both-empty omit path.

**Upgrade impact — one-time worker-pool roll.** The
`KubevirtMachineTemplate` name embeds a hash of the rendered worker
template body, and the MachineDeployment references that name. On the
shipped default (`md0` leaves `storageClass` empty) the body changes
from "no `storageClassName`" to `storageClassName: replicated`, so on
upgrade the template is renamed and Cluster API performs a one-time
rolling replacement of the worker pool — for every existing tenant
cluster, including those whose default StorageClass already was
`replicated` (the roll is driven by the body hash, not by whether the
effective class differs). This is the same class of roll any
worker-template change triggers and is unavoidable with this approach.
Two test pins document and guard the hash so future body changes are
deliberate.

### Release note

```release-note
fix(kubernetes): worker node disks now fall back to the application-level storageClass (a replicated/DRBD class) instead of the management-cluster default when a node group leaves storageClass empty, so live migration of worker VMs gets the ReadWriteMany volume it requires. Upgrading rolls each tenant cluster's worker pool once.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved worker disk `storageClassName` fallback for empty
per-node-group values: it now reliably uses the app-level
replicated/DRBD StorageClass (rather than relying on the
management-cluster default).
* Ensured KubeVirt data volume templates and rendering behavior are
consistent across single- and multi-node-group setups.

* **Documentation**
* Refined `storageClass`/RWX/DRBD guidance across configuration docs and
schema descriptions, including the intended non-immutable behavior
(optional/undefaulted).

* **Tests**
* Updated template test expectations to cover the new fallback vs
omission cases, including naming assertions for rendered templates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 12:00:00 +03:00
Алексей Артамонов 7faa32508d fix(clickhouse): reclaim cleanup hook RBAC on successful uninstall
Address review [MINOR] on #3072: keep the cleanup Job on
before-hook-creation (so a failed Job and its logs survive for
post-mortem), but restore before-hook-creation,hook-succeeded on the
ServiceAccount/Role/RoleBinding so those artifacts do not linger in the
namespace after a successful uninstall.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-01 18:31:05 +03:00
Алексей Артамонов 2e65ce108b fix(clickhouse): guard migration when keeper CRD absent and quote hook scalars
Address CodeRabbit review on #3072:

- Migration 45 now skips the backfill when the
  clickhousekeeperinstallations CRD is not installed. The migration runs
  platform-wide via targetVersion, so on clusters that never deployed
  ClickHouse the kubectl get would otherwise fail under set -euo pipefail
  and abort the whole upgrade.
- Quote the templated name/label/serviceAccountName scalars in the cleanup
  hook so the source parses under YAMLlint; rendered output is unchanged.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-01 18:31:05 +03:00