Follow-ups from the self-review of #3233:
- `values.yaml` / helper docstring / regenerated README + schema +
cozyrds openAPISchema + Go type doc: the previous "one
ingress-nginx worker" contract was already misleading (helper
default carried `minReplicas: 0` but the chart hardcoded
`spec.replicas: 2`), and the replicas fix makes the docstring
strictly false — an empty-nodeGroups install now provisions zero
workers until either the cluster-autoscaler responds to an
unschedulable Pod or the operator explicitly scales the group.
Update the docstrings to describe the actual behaviour and to
spell out the ingress-nginx-addon implication so operators are
not surprised at addon-install time.
- `tests/nodegroups_default_test.yaml`: two regression cases pinning
`MachineDeployment.spec.replicas == $group.minReplicas` for
`minReplicas: 0` (the render-side OIDC bats path — zero worker
DataVolumes provisioned) and for `minReplicas: 2` (the
kubernetes-latest / kubernetes-previous bats path — byte-
identical to pre-PR behaviour). Without these a future refactor
can silently re-hardcode `replicas: 2` and turn no tests red.
helm-unittest suite grows from 171 to 173, both new cases pass.
Ref: cozystack/cozystack#3231.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Closes#3231.
`packages/apps/kubernetes/templates/cluster.yaml` hardcoded
`MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's
`minReplicas`. The two render-side `kubernetes-oidc-{system,customconfig}`
bats — which set `minReplicas: 0` and delete their `Kubernetes` CR
without waiting for readiness — still provisioned two 20 GiB
`replicated` (DRBD) worker `DataVolume`s each. Their create/delete
churn overlapped the very next test, `kubernetes-previous`, and put
transient pressure on a LINSTOR satellite's ZFS pool: worker root
disks landed late, tenant nodes registered late, Cilium's CNI init
missed the 12-minute node-join deadline, and the test failed with
`cni plugin not initialized`.
Two changes, applied together — the chart fix is a no-op without
the test fix, and the test fix is a no-op without the chart fix,
because dropping `nodeGroups` falls back to the chart helper's
default `md0` (with `minReplicas: 0`).
**Chart**: `spec.replicas: {{ $group.minReplicas }}` so the field
matches the autoscaler annotations already emitted just above.
`minReplicas: 0` groups now genuinely scale from zero and do not
silently provision workers; groups with `minReplicas >= 2` (the
kubernetes-latest / kubernetes-previous bats CRs, the tests/values/
common.yaml default, the values.yaml default that gets templated
into the helper's `md0`) render byte-identical to before.
**Render-side OIDC tests**: drop the explicit `nodeGroups:` block
and `storageClass:` — falling back to the chart helper's default
`md0` (now with an effective `replicas: 0` after the chart change)
means the MachineDeployment renders but never creates a KubevirtMachine
and no DataVolume is provisioned. The tests only assert on the
KamajiControlPlane and the cozy-realm Keycloak objects, so removing
worker plumbing does not narrow their coverage.
Upgrade impact: any existing customer CR that already carries
`nodeGroups.md0.minReplicas: 0` (rare — the default in tests/values
and the chart helper both use it, but real user CRs almost always
have `minReplicas >= 1`) will see the MD's `spec.replicas` reconcile
from `2` down to `0` on the next chart apply. If the cluster-
autoscaler is enabled it will scale back up as soon as an
unschedulable pod appears; if it is not, the group intentionally
becomes empty (which is what `minReplicas: 0` requests literally).
This matches the semantics the autoscaler annotations were already
advertising — the hardcoded `2` was contradicting them.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for the newer etcd snapshot workflow
(BackupClass/BackupJob/RestoreJob) with in-place restore.
* Updated etcd chart/operator packaging and migration flow for the
v1alpha2 operator release.
* **Bug Fixes**
* Switched readiness/condition handling to **Available**.
* Improved waits to cover all etcd member pods during restore and
scenario steps.
* **Documentation**
* Refreshed etcd backup/restore docs and example scenarios to match the
v1alpha2 APIs and parameters.
* **Tests**
* Updated e2e coverage to validate v1alpha2 operator runtime contracts
and replaced legacy backup-block tests with a snapshot round-trip test.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from gemini-code-assist on packages/system/etcd-operator/templates/deployment.yaml: add an explicit seccompProfile (RuntimeDefault) to the controller-manager pod securityContext.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The claim mapping switched to email in de0299f33, but two header
comments still described the earlier design:
oidc-authn-config.yaml:15 — "Username claim is preferred_username"
oidc-rbac-job.yaml:28 — "matched against preferred_username"
The rendered config (oidc-authn-config.yaml:49) maps
username.claim: email, the CRB subject is rendered from .email
(oidc-rbac-job.yaml:264), and the email-claim contract is pinned by
tests/oidc_test.yaml:260-261. Only the comments were stale; no code
change.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Gemini flagged that the `values.yaml` @field docstring on
`oidc.users[].email` still described the claim mapping as
`preferred_username`, misleading operators after the claim was
switched to `email` for real (see the earlier fix and its
end-to-end verification on dev3). Rewrite the description to
match the actual mechanic: `email` claim, requested via
`--oidc-extra-scope=email` in the chart-generated kubeconfig,
emitted by every conformant OIDC provider.
`make generate` re-propagates the update through README.md,
types.go and the cozyrds embedded schema.
Note on the other finding in the same review — Gemini also
flagged `audiences` as being incorrectly nested under `issuer`
in the AuthenticationConfiguration Secret. That is a
false positive: per the kube-apiserver v1beta1 spec
(apiserver.config.k8s.io/v1beta1, `JWTAuthenticator.Issuer`),
`audiences` IS a child of `issuer`, not a sibling of `jwt[i]`.
The current shape is correct and verified end-to-end on dev3
(alice/bob tokens carry `aud: [tenant-test-kubernetes-test3,
kubernetes]` and apiserver validates them successfully with
CRB-driven RBAC).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Four findings from the gemini re-review:
- `.Values.oidc.customConfig.secretRef.name` and
`.Values.oidc.customConfig.config` — replaced direct chained
access with `dig "customConfig" "…" "" .Values.oidc` so a CR that
explicitly nulls `customConfig` doesn't nil-panic. `.Values.oidc`
itself has a kubebuilder default of `{}` so it stays reachable.
- `.Values.controlPlane.apiServer.extraArgs` in the collision guard
— wrapped with `| default list` so an explicit null renders as an
empty range rather than a nil deref. (`dig` on `.Values` itself
doesn't work because it's `chartutil.Values`, not
`map[string]interface{}`.)
- `base64 -w 0` in the OIDC kubeconfig patch — replaced with
`base64 | tr -d '\n'`. Both work on the current alpine/k8s image
(BusyBox 1.37 supports `-w`), but the pipe form is portable back
to older BusyBox releases and to macOS/BSD base64 (which does not
know `-w`).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Same class of upgrade regression: a customer who wired their own
structured AuthenticationConfiguration by hand via
`controlPlane.apiServer.extraArgs: [--authentication-config=...]`
would hit a duplicate-flag scenario the moment they toggle
`spec.oidc.mode` — kube-apiserver would receive two conflicting
--authentication-config= arguments and the last-wins behaviour is
undefined between apiserver versions.
Add a second render-time guard that fails when a legacy
--authentication-config= is present in the user-supplied extraArgs
AND spec.oidc.mode is not None, pointing at the CustomConfig
migration path (customConfig.config or customConfig.secretRef.name).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Reviewer flagged a real customer-upgrade regression: the previous
values.yaml docstring on `controlPlane.apiServer.extraArgs` documented
`--oidc-issuer-url`/`--oidc-client-id` as a valid usage. An existing
customer who wired OIDC by hand through that passthrough (following
the doc) would see their tenant kube-apiserver BOOT-FAIL the moment
they toggle `spec.oidc.mode` to `System` or `CustomConfig`:
kube-apiserver refuses to boot with both `--authentication-config`
(injected by the new spec.oidc) and any legacy `--oidc-*` flag —
`--authentication-config cannot be used with any of --oidc-*`.
Add a render-time guard: when `spec.oidc.mode != None`, scan the
user-supplied `controlPlane.apiServer.extraArgs` for any flag with
prefix `--oidc-` and fail with a clear message pointing at the
`spec.oidc.mode: CustomConfig` migration path (they can carry the
same issuer/audience inside `customConfig.config`).
When `spec.oidc.mode: None` the passthrough still works — customers
who manage OIDC entirely by hand keep working until they opt into
the new selector explicitly.
Also update the values.yaml docstring on `extraArgs` to steer OIDC
users to `spec.oidc.mode` and warn against the collision.
Two new helm-unittest cases: the collision fails, and mode=None with
legacy flags renders normally (backward compat).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Two follow-ups from the final adversarial review:
- The chart-generated OIDC kubeconfig exec block requested only
`--oidc-extra-scope=groups`. On a Keycloak realm where `email` is
not a default client scope, kubectl-oidc-login would fetch a token
that lacks the `email` claim — apiserver would then fail to
identify the user, causing a silent auth failure. Add
`--oidc-extra-scope=email` explicitly so the flow does not depend
on realm-level default-scope hygiene.
- `docs/oidc-tenant.md` still said the `User:` subject "must match
the `preferred_username` claim emitted by the issuer" — obsolete
after the claim switch. Rewrite to say `email`, and add a note
that the chart requests the `email` scope explicitly.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Real bug caught by an end-to-end OIDC login test on dev3: apiserver
accepts the JWT (auth succeeds) but the RBAC check fails with 403
because the CRB `User:` subject was `alice-oidc@test3.example` (from
`users[].email`) but the apiserver-visible identity was `alice-oidc`
(from `preferred_username`, which in the cozy realm is the Keycloak
*username*, not the email).
The design proposal example used `claim: preferred_username`, but
the operator-facing `users[].email` field explicitly asks for the
email address. In the cozy realm — verified — `preferred_username`
is the Keycloak username (`kvaps`, `i.okhotnikov`), so the two do
NOT coincide. Every user whose Keycloak username differs from their
email would silently 403 after a successful OIDC handshake.
Switch the AuthenticationConfiguration `claimMappings.username.claim`
to `email`, aligning the apiserver-visible identity with the
operator-declared `users[].email`. Every conformant OIDC provider
emits `email` — cozy realm does (verified), and any BYO
`CustomConfig` issuer that follows OIDC does too.
The switch is invisible to installers where `preferred_username`
already equals the email (a common Keycloak setup); for installers
where it doesn't, this fix is required for RBAC to work at all.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Caught by dev3 Job runtime: `kubectl patch --type=merge` fetches the
current object before computing the merged state, so the Role needs
`get` in addition to `patch` on the specific Secret name. Without it
the Job fails with `secrets "..." is forbidden: ... cannot get
resource "secrets"` and the OIDC kubeconfig stays on its placeholder.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Tighten the release Role — the `delete` verb on `keycloakclients` and
`keycloakclientscopes` now carries `resourceNames` naming exactly the
per-cluster client and its audience scope. Without the scope, the SA
could delete any Keycloak object in the tenant namespace.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Follow-up to the KCP cleanup extension: Kamaji's SSA field manager
takes ownership of `spec.apiServer.extraArgs` and
`spec.apiServer.extraVolumeMounts` too, not just
`spec.deployment.extraVolumes`. The previous normalize patch only
replaced extraVolumes; the two remaining fields still carried the
`--authentication-config=` flag and the corresponding mount even
after mode was toggled to None.
Extend the merge patch to REPLACE all three fields at once, sourced
from Helm values so user-supplied entries in
`controlPlane.apiServer.extra{Args,VolumeMounts,Volumes}` remain
intact — only the chart-owned OIDC-only entries drop off.
Verified on dev3: with the extended patch, mode=System → mode=None
leaves the tenant KCP with `extraArgs: []`,
`extraVolumeMounts: []`, and `deployment.extraVolumes` back to just
`talos-ca`, `talos-tls-cert` (or those + operator-supplied extras).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Follow-up to the KCP normalize commit: Helm-controller's SSA-based
apply also does not delete namespaced Secrets and cozy-realm
Keycloak resources when the new manifest omits them — the resources
stay as ghost owner-labeled-by-Helm objects. The dashboard picks
them up (via the cozyrds `secrets.include` list) and shows the
placeholder kubeconfig even for clusters where OIDC was explicitly
turned off.
Extend the `mode=None` normalize path to also `kubectl delete`:
- `<release>-oidc-kubeconfig` Secret
- `<release>-oidc-authn-config` Secret
- `<namespace>-<release>` KeycloakClient in the cozy realm
- `<namespace>-<release>-audience` KeycloakClientScope
The delete verbs are added to the release-owned Role scoped to the
specific resourceNames (Secrets) and the API group (Keycloak — the
EDP operator does not support resourceNames on cluster-scoped-ish
CRDs). No new privileges beyond what the chart itself would need to
render these objects.
Together with the previous KCP-normalize commit, `mode=System →
mode=None` now leaves ZERO chart-owned OIDC leftovers.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Kamaji's controller takes SSA field-ownership on
`KamajiControlPlane.spec.deployment.extraVolumes` when it syncs the
resource back from the TenantControlPlane, and Helm's subsequent
apply (with the same field-manager but a manifest that omits the
OIDC volume) then no longer removes it — the tenant apiserver Pod
keeps an unmounted-but-declared `authentication-config` volume
pointing at a Secret the chart already reaped. Harmless at runtime
(the volume is not mounted after the extraVolumeMounts fix), but the
CRD gets dirty and the operator wants zero leftovers.
Fix: the OIDC bootstrap Job is now rendered for ALL modes (not only
`System`/`CustomConfig`). When `mode=None` the Job takes the
normalize path — a single `kubectl patch --type=merge` that replaces
`spec.deployment.extraVolumes` with the chart baseline (talos-ca +
talos-tls-cert). We verified end-to-end that `kubectl patch
--type=merge` on this array REPLACES it deterministically (the
earlier Helm patch didn't).
For `mode=System`/`CustomConfig` the Job continues to apply
per-user ClusterRoleBindings inside the tenant cluster, prune stale
ones labelled by this release, and patch the pre-rendered
`<release>-oidc-kubeconfig` Secret with a real kubeconfig.
The Role gains `patch` on the specific `kamajicontrolplanes/<release>`
so the normalize path has just enough privilege; nothing else new.
The pre-delete cleanup Job stays conditional on `mode != None` — no
CRBs to clean up when OIDC was never enabled.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Two real bugs caught by the dev3 e2e test — both would have shipped
without live-cluster validation:
- `rm --force` in the bootstrap and cleanup Job scripts is rejected
by BusyBox `rm` (the alpine/k8s image uses BusyBox coreutils, not
GNU). The bootstrap Job on dev3 died at the prune step with
`rm: unrecognized option: force`. Switch to `rm -f` (works on
both GNU and BusyBox). Also swap
`grep --line-regexp --quiet --fixed-strings` for `grep -x -q -F`
for the same reason.
- Kamaji's `KamajiControlPlane` is a CRD, and Helm applies
JSON-merge-patch to CRDs. That patch does not remove array-typed
fields the new manifest simply omits — it leaves the live array
untouched. Toggling `oidc.mode: System` → `None` therefore left
the tenant kube-apiserver with a dangling `--authentication-config`
flag pointing at a Secret the chart had already reaped, and the
apiserver would refuse to start on the next kubelet restart.
Fix: always render `extraArgs` and `extraVolumeMounts` on the
KamajiControlPlane, emitting `[]` on the off-path so the patch
REPLACES the stale array. Verified end-to-end on dev3.
Adds two helm-unittest cases: the mode=None regression, and an
adjustment of the baseline default-values test.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Three items from the gemini-code-assist and coderabbitai passes that
survived the Phase-1 redesign:
- `oidc-rbac-job.yaml` (gemini HIGH): pre-delete cleanup uses
`kubectl --request-timeout=30s` in addition to `--timeout=30s`
on the delete verb, so an unreachable tenant apiserver cannot
stall the pre-delete hook on the HTTP round-trip itself. The
`|| echo WARNING` fallback still keeps a truly unreachable
cluster from blocking `helm uninstall`.
- `oidc-keycloak.yaml` + `_helpers.tpl` (gemini LOW): switch
`_cluster.{oidc-enabled,root-host}` reads to `dig` with an empty
map default so a missing `_cluster` key nil-safely emits an
empty string and hits the platform-OIDC guard instead of a
template panic on `.Values._cluster`.
- `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats`
(coderabbit MAJOR): add an existence-poll `timeout ... until
kubectl get secret ... ; do sleep ...; done` before the
`kubectl wait secret --for=jsonpath` line. Without it a runner
that reaches the wait before the HR has rendered the Secret
gets an immediate `NotFound` and the test flakes.
Adds one helm-unittest case covering the nil-safe `_cluster` path.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix caught by deploying to dev3: the EDP Keycloak operator
CRD v1 does not declare `standardFlow` (the canonical field is
`standardFlowEnabled`, default true), and `webUrl` is meaningful only
for clients with a real web UI (not for the kubectl oidc-login flow
which is keyed only by `redirectUris`).
Both lines violated the CRD's openAPIV3Schema and Helm refused the
install with `field not declared in schema`. Dropping the empty
`webUrl` and relying on `standardFlowEnabled`'s default (true) keeps
the manifest minimal and CRD-clean. helm-unittest did not catch this
because it does not validate templates against live CRD schemas.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix: the Role granted `["get", "patch"]` on the OIDC
kubeconfig Secret, but the bootstrap Job script never issues `kubectl
get` against the management apiserver — it only `kubectl patch`es the
pre-rendered Secret. The tenant-side `kubectl get clusterrolebindings`
uses the mounted admin kubeconfig, which is a separate trust path.
Tightens the SA's permission surface to exactly what the script uses.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix: the KeycloakClientScope rendered for the per-cluster
audience used `included.custom.audience` (raw string), but the
project convention — set in
`packages/system/keycloak-configure/templates/configure-kk.yaml` for
the legacy global `kubernetes-client` scope — is
`included.client.audience` (refers to a clientId in the same realm).
Both keys make Keycloak emit the audience claim with the same value
in our setup, but the `client.audience` form is the idiomatic one
when the audience IS a registered client in the realm.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix: the design proposal specifies
`name: authentication-config` and
`mountPath: /etc/kubernetes/authentication-config`, but the previous
revision had renamed both to `cozy-oidc-authn-config` and
`/etc/kubernetes/cozy-oidc` to dodge a collision with the volume
name used in `apiserver_authentication_test.yaml`. That test was
demonstrating the GENERIC user-supplied `extra*` passthrough — not
the OIDC use case — so rename the test fixture instead and restore
the design-spec names on the chart side.
The reservation guard against user-supplied volumes reusing
`authentication-config` only fires when `spec.oidc.mode` is not
None, so clusters with OIDC off retain the previous freedom to name
their own volumes anything.
Renames in `apiserver_authentication_test.yaml`: the user-supplied
volume name `authentication-config` becomes `tenant-byo-mount`; the
Secret name `tenant-authentication-config` and the apiserver flag
`--authentication-config=` are untouched.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fixes for four linked lifecycle issues:
- The post-install Job created `<release>-oidc-kubeconfig` via
`kubectl apply` from inside the script; the resulting Secret had
no Helm release labels and was orphaned on `helm uninstall`. Now
a release-owned placeholder Secret is rendered up front in
`oidc-kubeconfig-secret.yaml` and the Job patches its `data`
field at runtime (the tenant CA still has to come from the
Kamaji-issued admin kubeconfig, which appears asynchronously).
- The ServiceAccount/Role/RoleBinding for the bootstrap Job were
hook resources with `helm.sh/hook-delete-policy: before-hook-creation`,
so they leaked on `helm uninstall`. They are now regular release
resources (no hook annotations) and are reaped by `helm uninstall`
alongside everything else. Both the post-install bootstrap Job
and the pre-delete cleanup Job find a working SA.
- The previous design tried to add `hook-succeeded` to the SA, but
that would delete the SA right after the bootstrap hook
succeeded — leaving the pre-delete cleanup Job with no SA to
schedule. Release-owned resources sidestep this race entirely.
- The Role had `verbs: ["create"]` on `secrets` without
`resourceNames` (Kubernetes RBAC cannot scope `create` by name).
With the placeholder Secret pre-rendered the Job only needs
`get`/`patch` on the specific name — the broad `create` rule is
removed.
Also tightens the script: `set -euo pipefail` (was `set -eu`, which
silently swallowed `kubectl get` failures inside command
substitutions) and uses `kubectl patch --type=merge` against the
existing Secret instead of `kubectl create --dry-run | kubectl apply`.
Adds three helm-unittest cases for the placeholder Secret across all
three modes.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Add the operator-facing guide for the new selector — what the three
modes mean, what the chart provisions, how a user logs in, what the
failure modes are, and what is intentionally out of scope (per-tenant
realms, federation, cross-cluster SSO). The architectural rationale
points back to cozystack/community#24.
Update the chart NOTES.txt to print the OIDC kubeconfig retrieval
snippet, the per-cluster client/issuer values, and the listed
`users[]` roster — only when OIDC is on, otherwise the previous
admin-kubeconfig hint is unchanged.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Cover the three branches of the new selector API:
- `mode: None` (default) — every OIDC template is empty, the
KamajiControlPlane has no extra args and the standard two talos
volumes, the bootstrap Job is not rendered.
- `mode: System` — per-cluster `KeycloakClient` + audience scope
in the `cozy` realm, namespace-prefixed clientId, PKCE attribute
on the client, soft-render when the EDP Keycloak operator CRDs
are absent, the AuthenticationConfiguration Secret pinned to the
cozy issuer, the chart-owned `--authentication-config` flag and
`cozy-oidc-authn-config` mount injected into the KCP, the
bootstrap Job carrying `policy.cozystack.io/allow-to-apiserver`,
and one ClusterRoleBinding per `users[]` entry with the right
ClusterRole (`cluster-admin` vs `view`).
- `mode: CustomConfig` — inline config written verbatim; secretRef
path mounted on the KCP without a chart-owned Secret; no
Keycloak objects emitted; mutual-exclusion and missing-source
guards both hard-fail.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Add `kubernetes-<name>-oidc-kubeconfig` to the Kubernetes
ApplicationDefinition's `spec.secrets.include[].resourceNames` so
the dashboard surfaces the OIDC kubeconfig to tenant viewers
alongside the existing admin kubeconfig.
The Secret is produced by the post-install,post-upgrade bootstrap
Job in `apps/kubernetes` when `spec.oidc.mode: System` is set.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Add a post-install,post-upgrade Helm-hook Job that reconciles the
operator-supplied `users[]` list into the tenant cluster:
- For every entry, apply a ClusterRoleBinding labelled by the
release (managed-by=cozystack-oidc, instance=<release>).
`role: admin` maps to `cluster-admin`; `role: view` maps to
`view`. The subject is the literal `email` value, matched
against the issuer's `preferred_username` claim.
- Prune any orphan CRBs with the same labels — removing a user
from `users[]` revokes their kubectl access on the next loop.
For `mode: System` the same Job also writes
`<release>-oidc-kubeconfig` on the management cluster: a ready
kubeconfig with the per-cluster apiserver endpoint, the tenant CA
extracted from the Kamaji-issued admin kubeconfig, and a
`kubectl oidc-login` exec block keyed to the per-cluster client.
A pre-delete cleanup Job removes the labelled CRBs before the
control plane is torn down (best-effort: the CRBs disappear with
the tenant cluster anyway).
Pod is labelled `policy.cozystack.io/allow-to-apiserver: "true"`
so Cilium lets it reach the konnectivity tunnel.
ServiceAccount/Role/RoleBinding are minimum-privilege — only the
specific admin-kubeconfig and OIDC kubeconfig Secrets are listed.
Renders nothing for `mode: None`.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Render a per-release Secret carrying the structured kube-apiserver
`AuthenticationConfiguration` (apiserver.config.k8s.io/v1beta1) and
mount it into the KamajiControlPlane.
For `mode: System` the Secret is chart-owned: one JWT entry pointing
at the `cozy` realm issuer with the per-cluster audience, plus the
standard `preferred_username`/`groups` claim mappings. For `mode:
CustomConfig` the chart either writes the operator-supplied inline
config into a Secret, or — when `secretRef.name` is set — mounts an
existing operator-owned Secret directly (mutually exclusive; the
chart fails the render if neither or both are set).
The `--authentication-config=` flag, mount, and volume on the
KamajiControlPlane are kept orthogonal to the user-supplied
`controlPlane.apiServer.extra*` knobs: the OIDC additions are
appended, and the chart-managed volume name `cozy-oidc-authn-config`
is reserved (only when `mode` is not None) so user-supplied volumes
cannot collide.
Switches from the legacy `--oidc-*` flag form to the file form so
multiple issuers, inline private-CA PEM, and future Phase-2 issuers
extend the same Secret without rewiring the chart.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
For `oidc.mode: System`, emit a per-cluster public `KeycloakClient`
plus a per-cluster `KeycloakClientScope` inside the platform `cozy`
realm. The scope carries an `oidc-audience-mapper` that pins
`id_token.aud` to the per-cluster clientId, so a token minted for
cluster A is rejected by cluster B's apiserver.
The clientId is namespace-prefixed (`<ns>-<release>`) to stay unique
across the realm — two clusters with the same name in different
tenants would otherwise collide.
The client is `public: true`, redirects locked to localhost:8000 /
localhost:18000 (the kubelogin/oidc-login defaults), with PKCE
required via the standard Keycloak attribute. `defaultClientScopes`
includes both `groups` and the per-cluster audience scope.
A render-time guard fails fast when `oidc.mode: System` is set but
`_cluster.oidc-enabled` is not `"true"` — the Keycloak operator CRDs
are otherwise absent and the chart would silently no-op.
Nothing is rendered for `mode: CustomConfig` or `mode: None`.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Introduce a flat selector — `mode: System | CustomConfig | None` —
plus a `users[]` list and a `customConfig` block for tenant-supplied
AuthenticationConfiguration. Default `None` preserves today's
behavior.
The `System` mode trusts the platform `cozy` realm via a per-cluster
public client; `CustomConfig` accepts an inline `AuthenticationConfiguration`
or a Secret reference (mutually exclusive). The `users[]` list drives
per-user ClusterRoleBindings inside the tenant cluster, with two
roles: `admin` and `view`.
Regenerates deepcopy, JSON schema, README, and the kubernetes-rd
cozyrds embedded schema.
Implements Phase 1 API of the per-tenant OIDC design proposal
(cozystack/community#24).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Reverts #3006. Dropping persistent EFI/TPM was motivated by a live-migration
concern that rested on an outdated assumption — that an RWO Filesystem
backend-storage PVC pins the VM to its node. KubeVirt has migrated
RWO-Filesystem backend storage since v1.4 (kubevirt/kubevirt#12629): on
migration it creates a fresh target state PVC and copies the small state blob,
so persistent-EFI/TPM VMs on the default replicated storage live-migrate fine
(verified: the VM reports LiveMigratable=True and the copy-on-target PVC is
created on migration). Restore upstream's persistent firmware state for the
windows.* preferences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
## What this PR does
The Gateway (`cozystack-gateway-hostname-policy`) and Route
(`cozystack-route-hostname-policy`) admission policies constrain tenant
hostnames to the tenant apex on the Gateway API path. Gateway API is
opt-in and off by default, so in the default configuration tenant
applications publish through a legacy `networking.k8s.io/Ingress` on the
shared ingress-nginx, which had no equivalent policy.
This adds `cozystack-ingress-hostname-policy`, a fail-closed
`ValidatingAdmissionPolicy` + binding for `Ingress` in `tenant-*`
namespaces — the legacy-Ingress counterpart of the existing
Gateway/Route hostname policies, and a new Layer 8 in the gateway
security model. A hostname on `spec.rules[].host` or
`spec.tls[].hosts[]` is allowed when it is within the namespace's own
apex (`namespace.cozystack.io/host`), or when it is a concrete
(non-wildcard) name entirely outside the platform root apex
(`_cluster.root-host`). The second case keeps the `kubernetes` app's
Proxied `exposeMethod` working, since it routes a user-supplied external
domain to a nested cluster. A hostname under the platform root apex but
outside the namespace's own apex, a wildcard taken through the
outside-root path (which could otherwise match the platform apex
itself), a rule with no host (absent or empty — an unbounded catch-all),
and `spec.defaultBackend` are rejected.
The policy is fail-closed: a `tenant-*` namespace missing its own apex
label is denied, and the policy renders only when `_cluster.root-host`
is set, so it is never shipped in a state that would admit every
hostname. `cozy-*` namespaces are cluster-admin-managed and not gated.
This is defense-in-depth in the same family as the Gateway/Route
policies — by design tenants hold only read access to `Ingress`.
The change ships helm-unittest coverage (render, plus the root-host,
empty-host, and wildcard guards) and behavioral admission tests against
the live apiserver in `hack/e2e-apps/gateway.bats` (in-apex allow,
external-domain allow, under-root-outside-apex deny,
wildcard-matching-apex deny, hostless deny, empty-string-host deny,
defaultBackend deny), and documents the new layer in
`packages/extra/gateway/README.md`.
### Screenshots
N/A — no UI change.
### Release note
```release-note
fix(cozystack-basics): constrain legacy Ingress hostnames in tenant namespaces to the tenant apex
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added **fail-closed** defense-in-depth Layer 8
(`cozystack-ingress-hostname-policy`) to enforce allowed legacy Ingress
rule and TLS hostnames for tenant namespaces, while blocking unsafe
catch-all configurations (including missing/empty hosts and
`spec.defaultBackend`) when the platform root host is configured.
* **Tests**
* Added end-to-end and render/deny validation for permitted vs denied
hostname patterns, wildcard behavior, required namespace host labeling,
and correct deny/fail-closed rendering.
* **Documentation**
* Updated the gateway security model to document Layer 8 behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
The VPA admission-controller webhook TLS Secret is currently rendered by
an in-chart `genCA` + `genSignedCert` template. Those functions are not
idempotent across `helm template` invocations — every Flux HelmRelease
reconcile produces a fresh CA + cert. A `checksum/tls-secret`
podAnnotation on the admission-controller Deployment then flips its
sha, materialising a new ReplicaSet on every reconcile.
Under load — the E2E install phase where cert-manager / CDI / LINSTOR
cascade races repeatedly refire the VPA HelmRelease — this produces
10+ admission-controller ReplicaSets in minutes and blocks the whole
cozystack-platform HR: the admission-controller pod livelocks in
startup (waits on its own election lease which its previous
ReplicaSet is still holding, hits livenessProbe timeout,
CrashLoopBackOffs), VPA never becomes Ready, PackageSources stay
InProgress, and cozystack-platform aborts at its 15-minute install
timeout.
Fix: replace the source of the TLS Secret with a namespace-scoped
cert-manager `Issuer` + `Certificate`. cert-manager only rewrites the
Secret when the certificate is close to expiry (`renewBefore`), so
reconcile-driven cert churn stops. Pod rotation is delegated to a
runtime-lookup-based `checksum/tls-cert` podAnnotation that only
changes when the Secret's `tls.crt` content actually changes — not on
every render, not on every reconcile.
## What
-
`charts/vertical-pod-autoscaler/templates/admission-controller/certificate.yaml`
(new): namespace-scoped selfSigned Issuer + Certificate. The
Certificate mirrors the legacy `genSignedCert` shape (single DNS SAN
`vpa-webhook.<ns>.svc`, 365d validity, matching Secret name), so
cert-manager on the first upgrade adopts the same Secret name the
legacy path wrote — no rename cascade in the Deployment volume mount.
Rendered only when `admissionController.tls.certManagerEnabled=true`
AND no `existingSecret` is supplied.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/tls-secret.yaml`
(modified): wrapped in an extra `certManagerEnabled=false` guard so
the legacy genCA path remains available as an explicit opt-out.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/deployment.yaml`
(modified): the legacy `checksum/tls-secret` annotation is gated
behind `certManagerEnabled=false`. Under the cert-manager path the
Deployment carries a `checksum/tls-cert` annotation computed from
`lookup`-ing the runtime Secret's `tls.crt` — see the template
comment for the full timing analysis and the day-350 renewal
trade-off it documents.
- `charts/vertical-pod-autoscaler/values.yaml` (modified): new
`admissionController.tls.certManagerEnabled` value defaulting to
`true`. Values-block header documents the switch as effectively
one-way (cert-manager writes `type: kubernetes.io/tls`;
`Secret.type` is immutable, so falling back requires `kubectl
delete` first).
- `patches/cert-manager-admission-tls.diff` (new) +
`Makefile update` extended: the four subchart edits above live as a
single reviewable diff that Makefile re-applies over the freshly
vendored chart on every `make update`, matching the fluxcd-operator
package convention. Prevents a bump from silently reverting the
fix.
- `packages/core/platform/sources/vertical-pod-autoscaler.yaml`
(modified): VPA PackageSource now depends on `cozystack.cert-manager`.
Non-circular: cert-manager depends on `cozystack.networking` +
`cozystack.gateway-api-crds`, neither reaches VPA.
- `tests/vpa_test.yaml` (modified): +11 helm-unittest cases (15 total,
4 pre-existing) covering `certManagerEnabled={true,false}`,
`existingSecret` bypass, disabled admission-controller, and
the offline-render behaviour of the `checksum/tls-cert` lookup.
## Why (evidence)
Snapshot from CI run 28587656430 (this branch, pre-workaround-review):
```
admission-controller ReplicaSets after first-install churn window:
vertical-pod-autoscaler-admission-controller-55d888d6d4
vertical-pod-autoscaler-admission-controller-56f8d4c747
vertical-pod-autoscaler-admission-controller-5775dbc9b7
vertical-pod-autoscaler-admission-controller-585b7c56c9
vertical-pod-autoscaler-admission-controller-6fc44944fb
vertical-pod-autoscaler-admission-controller-74bccf4849
vertical-pod-autoscaler-admission-controller-84df98f98b
vertical-pod-autoscaler-admission-controller-8666dffb85
vertical-pod-autoscaler-admission-controller-86b469457c
vertical-pod-autoscaler-admission-controller-9b979f9d6
→ 10 ReplicaSets, one per Flux reconcile, each with a different
genCA-rooted TLS Secret sha in checksum/tls-secret.
```
## Migration for existing clusters
On upgrade the legacy Opaque Secret disappears from the release
manifest, Helm garbage-collects it, and cert-manager immediately
writes a fresh `kubernetes.io/tls` Secret of the same name via the
new Certificate. `Secret.type` is immutable, so in-place adoption is
impossible — the delete/create pass is the correct path, and the
seconds-long window it produces is safe: the `vpa-webhook-config`
MutatingWebhookConfiguration ships with `failurePolicy: Ignore`
(verified on cozy-dev3), so Pod admissions in that window skip VPA
mutation but never get rejected — no cluster-wide disruption.
The Deployment rolls in the same upgrade because the new
`checksum/tls-cert` annotation differs from anything the pod had
before, so the admission-controller re-registers `vpa-webhook-config`
with the fresh caBundle on the same pass.
## Test plan
- [x] `helm unittest packages/system/vertical-pod-autoscaler` — 15/15
pass.
- [x] `helm template` diff between main and this branch is scoped
exactly to the cert-manager path (new Certificate + Issuer,
new `checksum/tls-cert` annotation, guarded legacy Secret, new
values key, patch file + Makefile step, PackageSource
dependsOn). No unrelated lines change.
- [x] `make update`-style simulation (rm -rf charts && helm pull &&
patch) produces a byte-identical subchart to what git tracks —
the patch stays clean across future upstream bumps until it
genuinely conflicts.
- [ ] Live migration on cozy-dev3: `helm upgrade` VPA release from
pre-fix version → observe cert-manager write the new tls Secret
→ verify `checksum/tls-cert` on the new pod's podTemplate →
verify MutatingWebhookConfiguration `vpa-webhook-config`
caBundle updated in place → confirm no ReplicaSet churn on
subsequent Flux reconciles. Blocking the draft's removal.
- [ ] Full E2E on this branch — verify only 1 admission-controller
ReplicaSet after install, VPA HR reaches Ready inside
`cozystack-platform`'s 15m install timeout, and
`kubernetes-latest.bats` / `kubernetes-previous.bats` no longer
time out on VPA-attributable resource churn.
## Review pass — kvaps' CHANGES_REQUESTED addressed
- ✅ Vendored chart mods now carried as `patches/*.diff` re-applied by
`Makefile update` (matches fluxcd-operator convention).
- ✅ Retired the tls-migrator hook — kvaps correctly showed it was
a no-op: Helm's delete decision is driven by the stored release
manifest not by live labels (`ResourcePolicyAnno` is the only opt-
out), and `Secret.type` is immutable so cert-manager could not have
adopted the legacy Opaque Secret in place anyway. ~200 lines
removed.
- ✅ PR description rewritten to match the actual diff (no more
reference to the removed `tls-rollout.yaml` post-upgrade hook, real
test count, dev3 verification honestly listed as pending).
- ✅ `checksum/tls-cert` timing comment corrected: `lookup` runs on
Helm actions only, day-350 renewal trade-off spelled out.
- ✅ `certManagerEnabled=false` documented as effectively one-way
(Secret.type immutable).
- ✅ Dropped the `secretTemplate` labels — `app.kubernetes.io/managed-
by: Helm` on a cert-manager-owned Secret was misleading.
- ✅ Banner comment indentation restored in `vpa_test.yaml`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* VPA admission-controller webhook TLS now uses cert-manager by default,
with options to disable cert-manager or supply an existing TLS secret.
* Webhook pod restarts are now tied to cert-manager Secret contents via
a content-based checksum.
* **Bug Fixes**
* Improved webhook TLS provisioning reliability when TLS readiness is
delayed.
* Corrected conditional rendering/rollout behavior to avoid mismatched
TLS ownership.
* **Tests**
* Added rendering-invariant tests covering cert-manager, legacy TLS,
existing-secret bypass, and admission-controller disabled behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The Gateway (cozystack-gateway-hostname-policy) and Route
(cozystack-route-hostname-policy) admission policies constrain tenant
hostnames on the Gateway API path. Gateway API is opt-in and off by
default, so in the default configuration tenant applications publish
through a legacy networking.k8s.io/Ingress on the shared ingress-nginx,
which had no equivalent policy.
Add a fail-closed ValidatingAdmissionPolicy + Binding for Ingress in
tenant-* namespaces. A hostname on spec.rules[].host or spec.tls[].hosts[]
is allowed when it is within the namespace's own apex
(namespace.cozystack.io/host), or when it is a concrete (non-wildcard)
name entirely outside the platform root apex (_cluster.root-host); a
hostname under the platform root apex but outside the namespace's own
apex, and a wildcard taken through the outside-root path (which could
match the platform apex itself), are rejected. The outside-root branch
keeps the kubernetes app's Proxied exposeMethod working, since it routes
a user-supplied external domain to a nested cluster. A rule with no host
(absent or empty, an unbounded catch-all) and spec.defaultBackend are
also rejected.
The policy templates the platform root apex into its CEL and renders
only when _cluster.root-host is set, so it is never shipped in a state
that would admit every hostname. A tenant-* namespace missing its own
apex label is denied.
Cover the policy with helm-unittest (render, including the root-host,
empty-host, and wildcard guards) and behavioral admission tests against
the live apiserver (in-apex allow, external-domain allow,
own-apex-wildcard allow, external-tls allow, under-root-outside-apex
deny, prefix-adjacent-under-root deny, wildcard-matching-apex deny,
hostless deny, empty-string-host deny, defaultBackend deny), and document
it as an additional layer in the gateway security model.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
cainjector loads all ValidatingWebhookConfigurations, APIServices and CRDs
into its informer caches at startup. On a full cozystack install that working
set exceeds the 128Mi limit, so the single leader-elected cainjector Pod is
OOMKilled during cache population and CrashLoops before it injects any
caBundle. Webhook ValidatingWebhookConfigurations then keep an empty caBundle
and every admission call fails with "x509: certificate signed by unknown
authority" (failurePolicy: Fail), intermittently blocking Ingress creation
and stalling tenant installs.
Raise the cainjector memory limit to 512Mi (matching the controller) and the
request to 128Mi, and add a helm-unittest that pins the sizing so a blanket
resource change or a `make update` cannot silently lower it again.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
## What this PR does
`kamaji-controller-manager` runs **9 sub-controllers per
TenantControlPlane** (bootstraptoken, coredns, konnectivity, kubeadm*,
kubelet, kubeproxy, …) inside a single Go process, **and** serves the
admission webhooks for `tenantcontrolplanes` + `datastores` from the
same pod via `kamaji-webhook-service`. With the upstream default
`replicaCount=1`, every webhook round-trip from the apiserver shares CPU
with the reconciler goroutines. On a multi-tenant cluster this pushes
the apiserver-observed p99 of `mtenantcontrolplane.kb.io` and
`vtenantcontrolplane.kb.io` from the kamaji-side `~10ms` (per
`controller_runtime_webhook_latency_seconds_count/sum`) up to `85–95ms`
— purely due to CPU contention.
Three safe defaults are changed:
- **`kamaji.replicaCount: 2`** — HA-safe baseline. `--leader-elect` is
already enabled upstream, so only one replica reconciles tenants; the
follower's CPU is fully available for webhook handlers. On a single-node
cluster both pods still schedule (just colocated).
- **`kamaji.telemetry.disabled: true`** — drops the
`--disable-telemetry` handler **and** removes the
`telemetry.kamaji.clastix.io` entry from the
`ValidatingWebhookConfiguration`. The latter requires `patches/1.diff`
to wrap the telemetry block in `{{- if not .Values.telemetry.disabled }}
… {{- end }}` inside the vendored
`controller-gen/validating-webhook.yaml`. Without the webhook entry, the
apiserver no longer round-trips to a no-op handler on every
TenantControlPlane CREATE/UPDATE/DELETE.
- **`kamaji.affinity` (soft `podAntiAffinity`)** — spread the two
replicas across nodes when possible, while keeping single-node clusters
working (`preferredDuringScheduling`, not required).
The `patches/1.diff` approach (re-applied by `make update`) mirrors the
pattern already used in `packages/system/fluxcd-operator/Makefile` and
`packages/system/ingress-nginx/Makefile`, and survives the next
`clastix/kamaji` chart bump.
## Measured impact
Hikube prod, 22 TenantControlPlanes, before/after a manual rollout of
the same configuration:
| webhook | p99 before | p99 after | Δ |
| --- | --- | --- | --- |
| `mtenantcontrolplane.kb.io` | 85 ms | 25 ms | **-71%** |
| `vtenantcontrolplane.kb.io` | 92 ms | 25 ms | **-73%** |
| `telemetry.kamaji.clastix.io` | 94 ms | _webhook removed_ | **-100%**
|
Cumulative Kamaji admission **impact** (`p99 × calls/s`) dropped from
**830 → 140 (-83%)**. The Kamaji webhooks moved from the top three of
the `apiserver_admission_webhook_admission_duration_seconds_*` impact
ranking down to positions #3-#4.
## Release note
```release-note
feat(kamaji): default to 2 replicas with soft pod anti-affinity and disable telemetry (handler + webhook entry). Reduces apiserver admission p99 for tenantcontrolplane mutating/validating webhooks by ~70% on multi-tenant clusters.
```
## Test plan
- [x] `helm template packages/system/kamaji` renders `replicas: 2`,
`--disable-telemetry`, the soft anti-affinity, and **no**
`telemetry.kamaji.clastix.io` entry in the
`ValidatingWebhookConfiguration`.
- [x] `patches/1.diff` applies cleanly against the freshly extracted
upstream chart (`patch --dry-run -p1 < patches/1.diff`).
- [x] `make update` re-applies the patch automatically.
- [x] Live deploy on a 22-TCP cluster: 3 pods Running (when overriden to
`replicaCount: 3` via cozystack-values), zero pod restarts, leader
election visible, `kamaji-webhook-service` Endpoints serving N IPs.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Telemetry can be disabled via configuration.
* Controller configured for high availability with 2 replicas.
* Pod anti-affinity added to spread replicas across nodes for better
resilience.
* **Bug Fixes**
* Validating webhook is now omitted when telemetry is disabled to avoid
unintended behavior.
* **Chores**
* Update process improved to re-apply chart patches after extraction for
reproducible updates.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2671?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
- Exposes the CDI `cloneStrategyOverride` as a Helm value in
`packages/system/kubevirt-cdi` instead of hardcoding it in the CR
template.
- Default stays `csi-clone`, so **behavior is unchanged** for everyone.
- It can now be overridden via a cozystack `Package` (e.g.
`cloneStrategyOverride: copy`), the same way `uploadProxyURL` already is
— no fork needed.
### Why
The value was hardcoded, so the only way to change the clone strategy
was to fork the package. On some storage backends `csi-clone` is
problematic: with LINSTOR/DRBD a CSI clone is created co-located with
the source golden image and then relocated to optimal nodes
asynchronously; that async relocation (`migrate-disk`) can race the
post-clone resize CDI performs, occasionally leaving a volume whose DRBD
device is smaller than its PVC capacity. Being able to select `snapshot`
or `copy` per cluster, without maintaining a fork, is a useful escape
hatch.
This only adds a knob; it does not change the default or recommend a
strategy.
## Release note
```release-note
feat(kubevirt-cdi): make CDI cloneStrategyOverride configurable via values (default csi-clone)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* The KubeVirt CDI clone strategy is now configurable via Helm chart
values, allowing selection of `csi-clone`, `snapshot`, or `copy` for
DataVolume cloning.
* The default remains `csi-clone` to preserve existing behavior and
ensure backward compatibility.
* **Documentation**
* Added the `cloneStrategyOverride` Helm value description and allowed
options to the chart’s configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
- Adds `WEED_POSTGRES2_CONNECTION_MAX_IDLE=20`,
`WEED_POSTGRES2_CONNECTION_MAX_OPEN=40` and
`WEED_POSTGRES2_CONNECTION_MAX_LIFETIME_SECONDS=600` to the filer
`extraEnvironmentVars` in `packages/system/seaweedfs/values.yaml`.
The vendored seaweedfs chart only ships connection pool settings for the
**mysql** store (`WEED_MYSQL_CONNECTION_MAX_*`), which cozystack does
not use. The **postgres2** store, which cozystack enables, has no pool
settings at all. As a result the Go `database/sql` pool runs with
`SetMaxIdleConns(0)`: every filer metadata lookup opens a fresh
PostgreSQL connection (TCP + TLS + SCRAM, ~300 ms each). With ~5
path-component lookups per S3 operation, this adds roughly **2 seconds
of latency to every S3 request**, even on an idle cluster.
### Measurements (production cluster, 12 nodes, 2 filers, CNPG backend)
Before / after applying these three variables, same test (sequential ops
through the S3 gateway):
| Operation | Before | After |
|---|---|---|
| PUT 4 KiB (p50) | 10.0 s | 0.28 s |
| GET 4 KiB (p50) | 3.0 s | 0.14 s |
| HEAD (p50) | 2.6 s | 0.16 s |
| Bounded LIST, 100 keys (p50) | 3.8 s | 0.37 s |
`pg_stat_activity` confirmed the root cause: before the fix every
backend was younger than ~200 ms (permanent connection churn); after the
fix connections are reused for minutes. Real-traffic p95 of
`SeaweedFS_filerStore_request_seconds` dropped from 1.3–6.5 s to ~0.18
s.
`max_open` is capped at 40 so that the default two filer replicas stay
below the CNPG default `max_connections` of 100.
## Release note
```release-note
fix(seaweedfs): configure postgres2 connection pool for the filer, removing ~2s of per-request S3 latency caused by PostgreSQL connection churn
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Optimized PostgreSQL connection pool configuration in SeaweedFS to
reduce latency and improve performance during metadata operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
The upstream Stakater reloader chart ships with `resources: {}`, which
triggers a `no CPU limit` conformance warning on the `reloader`
Deployment.
Override `packages/system/reloader/values.yaml` at
`reloader.reloader.deployment.resources` with the exact values from the
chart's own commented example:
- `limits` : `cpu: 100m`, `memory: 512Mi`
- `requests` : `cpu: 10m`, `memory: 128Mi`
Reloader is a light controller that watches ConfigMap / Secret changes
and triggers workload restarts — small CPU envelope with enough memory
to track many objects in clusters with lots of namespaces.
## Test plan
- [ ] `helm template . --namespace cozy-reloader` from
`packages/system/reloader/` renders a `resources` block on the
`reloader-reloader` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `reloader-reloader has no
CPU limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on the reloader Deployment to clear
the "no CPU limit" conformance warning. Values follow the upstream
chart's commented example (limits 100m/512Mi, requests 10m/128Mi).
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated resource allocation configuration to define CPU and memory
parameters for system component deployments, enhancing infrastructure
stability and resource optimization.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2631)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
The upstream metrics-server chart ships with `limits` commented out
(only `requests cpu 100m / memory 200Mi` are set), which triggers a `no
CPU limit` conformance warning on the `metrics-server` Deployment.
Override `packages/system/metrics-server/values.yaml` to keep the
upstream-recommended baseline `requests` and add `limits`:
- `limits` : `cpu: 500m`, `memory: 500Mi`
- `requests` : `cpu: 100m`, `memory: 200Mi`
Comfortable headroom on top of the [official sizing
formula](https://github.com/kubernetes-sigs/metrics-server#configuration)
(`100m + 1m/node`, `200Mi + 2Mi/node`), covering clusters up to ~150
nodes without needing to revisit. Larger clusters can override these
values further.
## Test plan
- [ ] `helm template . --namespace cozy-monitoring` from
`packages/system/metrics-server/` renders a `resources` block on the
`metrics-server` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `metrics-server has no CPU
limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on the metrics-server Deployment to
clear the "no CPU limit" conformance warning. Values follow the upstream
sizing recommendation (requests 100m/200Mi) with a comfortable limits
cap (500m/500Mi).
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Configured explicit resource requests and limits for the
metrics-server service (CPU: 100m-500m, Memory: 200Mi-500Mi) to ensure
consistent resource allocation.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2630)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
The upstream external-dns chart ships with `resources: {}`, which
triggers a `no CPU limit` conformance warning on the `external-dns`
Deployment.
Override `packages/system/external-dns/values.yaml` to set:
- `limits` : `cpu: 100m`, `memory: 128Mi`
- `requests` : `cpu: 10m`, `memory: 64Mi`
Modest sizing for a controller that just watches Service / Ingress
objects and pushes DNS records to the provider.
## Test plan
- [ ] `helm template . --namespace cozy-external-dns` from
`packages/system/external-dns/` renders a `resources` block on the
`external-dns` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `external-dns has no CPU
limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on the external-dns Deployment to
clear the "no CPU limit" conformance warning.
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated ExternalDNS resource allocation with optimized CPU and memory
limits and requests for improved resource efficiency.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2629)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
The `matchbox` container in
`packages/extra/bootbox/templates/matchbox/deployment.yaml` had no
`resources` block, triggering a `no CPU limit` conformance warning on
the `bootbox-matchbox` Deployment (PXE/iPXE provisioning service).
Bootbox is a standalone chart (no vendored upstream), so the values are
set directly in the template — same pattern as #2614 (csi-resizer) and
#2624 (s3manager).
- `limits` : `cpu: 200m`, `memory: 256Mi`
- `requests` : `cpu: 50m`, `memory: 128Mi`
Modest sizing since matchbox is a lightweight HTTP server serving iPXE
profiles and boot assets from ConfigMaps.
## Test plan
- [ ] `helm template bootbox . --namespace tenant-root --show-only
templates/matchbox/deployment.yaml --set _namespace.host=example.com
--set _cluster.solver=letsencrypt` from `packages/extra/bootbox/`
renders the `resources` block on the `matchbox` container (verified
locally)
- [ ] Re-run the conformance/lint tool — the `matchbox has no CPU limit`
warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Add CPU and memory requests/limits to the matchbox container of the
bootbox Deployment to clear the "no CPU limit" conformance warning.
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added Kubernetes resource limits and requests configuration to
optimize container resource allocation and improve deployment stability.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2625)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
The `s3manager` container in
`packages/system/bucket/templates/deployment.yaml` had no `resources`
block, triggering a `no CPU limit` conformance warning on the bucket UI
Deployment.
The bucket package is a standalone chart (no vendored upstream), so the
values are set directly in the template — same pattern as the
csi-resizer fix in #2614 and the existing `capi-providers-cpprovider`
template.
- `limits` : `cpu: 500m`, `memory: 512Mi`
- `requests` : `cpu: 100m`, `memory: 128Mi`
Comfortable headroom for the lightweight S3 browser UI while keeping a
small baseline request.
## Test plan
- [ ] `helm template . --show-only templates/deployment.yaml` from
`packages/system/bucket/` renders the `resources` block on the
`s3manager` container (verified locally)
- [ ] Re-run the conformance/lint tool — the `s3manager has no CPU
limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Add CPU and memory requests/limits to the s3manager container of the
bucket UI Deployment to clear the "no CPU limit" conformance warning.
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Improved deployment configuration with explicit resource allocation
settings to ensure optimal service stability and resource management.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2624)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
The upstream Bloomberg goldpinger chart ships with `resources: {}`,
which triggers a `no CPU limit` conformance warning on the
`goldpinger-daemon` container running on every node.
Override `packages/system/goldpinger/values.yaml` to set:
- `limits` : `cpu: 200m`, `memory: 256Mi`
- `requests` : `cpu: 50m`, `memory: 64Mi`
Comfortable margin for the ping-mesh fan-out on larger clusters while
keeping a small baseline request so the DaemonSet schedules everywhere.
## Test plan
- [ ] `helm template . --namespace cozy-goldpinger` from
`packages/system/goldpinger/` renders the `resources` block on the
`goldpinger-daemon` container of the DaemonSet (verified locally)
- [ ] Re-run the conformance/lint tool — the `goldpinger has no CPU
limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on the goldpinger DaemonSet to clear
the "no CPU limit" conformance warning.
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated Kubernetes resource requirements for the goldpinger component,
defining CPU limits (200m), memory limits (256Mi), CPU requests (50m),
and memory requests (64Mi).
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2623)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
The upstream cert-manager chart ships with `resources: {}` for all four
workloads, which triggers `no CPU limit` (and `no memory limit`)
conformance warnings on every pod.
Override `packages/system/cert-manager/values.yaml` (previously empty)
to set sane requests/limits sized to each component's role:
| Component | requests cpu/mem | limits cpu/mem | Notes |
|---|---|---|---|
| controller | 50m / 128Mi | 500m / 512Mi | Main workload — issues and
renews certs |
| webhook | 10m / 32Mi | 200m / 128Mi | Admission webhook, short-lived
requests |
| cainjector | 10m / 32Mi | 200m / 128Mi | CA bundle injection, light
watch loop |
| startupapicheck | 10m / 16Mi | 100m / 64Mi | One-shot Job at install
time |
Requests for `cainjector` (`10m` cpu / `32Mi` mem) match the example in
the upstream values.yaml comment.
## Test plan
- [ ] `helm template . --namespace cert-manager` from
`packages/system/cert-manager/` renders a `resources` block on all four
pods (verified locally for controller, webhook, cainjector and the
startupapicheck Job)
- [ ] Re-run the conformance/lint tool that originally flagged the
warnings on `cert-manager-cainjector` — should be clean on the other
three pods too
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on cert-manager controller, webhook,
cainjector and startupapicheck pods to clear "no CPU limit" conformance
warnings.
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added resource allocation configuration for cert-manager deployment,
specifying CPU and memory requests and limits for improved system
stability and resource management across core components.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2616)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->