Commit Graph
1740 Commits
Author SHA1 Message Date
IvanHunters d30a2cfffb docs+test(apps/kubernetes): align nodeGroups default docs with actual zero-worker topology + regression-guard the replicas fix
Follow-ups from the self-review of #3233:

- `values.yaml` / helper docstring / regenerated README + schema +
  cozyrds openAPISchema + Go type doc: the previous "one
  ingress-nginx worker" contract was already misleading (helper
  default carried `minReplicas: 0` but the chart hardcoded
  `spec.replicas: 2`), and the replicas fix makes the docstring
  strictly false — an empty-nodeGroups install now provisions zero
  workers until either the cluster-autoscaler responds to an
  unschedulable Pod or the operator explicitly scales the group.
  Update the docstrings to describe the actual behaviour and to
  spell out the ingress-nginx-addon implication so operators are
  not surprised at addon-install time.

- `tests/nodegroups_default_test.yaml`: two regression cases pinning
  `MachineDeployment.spec.replicas == $group.minReplicas` for
  `minReplicas: 0` (the render-side OIDC bats path — zero worker
  DataVolumes provisioned) and for `minReplicas: 2` (the
  kubernetes-latest / kubernetes-previous bats path — byte-
  identical to pre-PR behaviour). Without these a future refactor
  can silently re-hardcode `replicas: 2` and turn no tests red.

  helm-unittest suite grows from 171 to 173, both new cases pass.

Ref: cozystack/cozystack#3231.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-07 15:51:01 +03:00
Timofei Larkin 7b65cf2afd feat(platform): migrate to etcd-operator v1alpha2 (#2859)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note

```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for the newer etcd snapshot workflow
(BackupClass/BackupJob/RestoreJob) with in-place restore.
* Updated etcd chart/operator packaging and migration flow for the
v1alpha2 operator release.
* **Bug Fixes**
  * Switched readiness/condition handling to **Available**.
* Improved waits to cover all etcd member pods during restore and
scenario steps.
* **Documentation**
* Refreshed etcd backup/restore docs and example scenarios to match the
v1alpha2 APIs and parameters.
* **Tests**
* Updated e2e coverage to validate v1alpha2 operator runtime contracts
and replaced legacy backup-block tests with a snapshot round-trip test.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-06 19:04:59 +04:00
Andrey Kolkov 011d6d1639 fix FUs
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
Andrey Kolkov ee12295d30 fix(etcd-operator): set seccompProfile RuntimeDefault on controller pod
Address review feedback from gemini-code-assist on packages/system/etcd-operator/templates/deployment.yaml: add an explicit seccompProfile (RuntimeDefault) to the controller-manager pod securityContext.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
Andrey Kolkov 087a8c4be9 chore(etcd-operator): bump to v0.5.2
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
Andrey Kolkov cc5d688e15 feat(platform): migrate to etcd-operator v1alpha2
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
Ivan Okhotnikov 7b8a330ad4 docs(apps/kubernetes): update users[].email docstring — email claim, not preferred_username
Gemini flagged that the `values.yaml` @field docstring on
`oidc.users[].email` still described the claim mapping as
`preferred_username`, misleading operators after the claim was
switched to `email` for real (see the earlier fix and its
end-to-end verification on dev3). Rewrite the description to
match the actual mechanic: `email` claim, requested via
`--oidc-extra-scope=email` in the chart-generated kubeconfig,
emitted by every conformant OIDC provider.

`make generate` re-propagates the update through README.md,
types.go and the cozyrds embedded schema.

Note on the other finding in the same review — Gemini also
flagged `audiences` as being incorrectly nested under `issuer`
in the AuthenticationConfiguration Secret. That is a
false positive: per the kube-apiserver v1beta1 spec
(apiserver.config.k8s.io/v1beta1, `JWTAuthenticator.Issuer`),
`audiences` IS a child of `issuer`, not a sibling of `jwt[i]`.
The current shape is correct and verified end-to-end on dev3
(alice/bob tokens carry `aud: [tenant-test-kubernetes-test3,
kubernetes]` and apiserver validates them successfully with
CRB-driven RBAC).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 9c73b6ef10 fix(apps/kubernetes): render-fail when spec.oidc.mode collides with legacy --oidc-* extraArgs
Reviewer flagged a real customer-upgrade regression: the previous
values.yaml docstring on `controlPlane.apiServer.extraArgs` documented
`--oidc-issuer-url`/`--oidc-client-id` as a valid usage. An existing
customer who wired OIDC by hand through that passthrough (following
the doc) would see their tenant kube-apiserver BOOT-FAIL the moment
they toggle `spec.oidc.mode` to `System` or `CustomConfig`:
kube-apiserver refuses to boot with both `--authentication-config`
(injected by the new spec.oidc) and any legacy `--oidc-*` flag —
`--authentication-config cannot be used with any of --oidc-*`.

Add a render-time guard: when `spec.oidc.mode != None`, scan the
user-supplied `controlPlane.apiServer.extraArgs` for any flag with
prefix `--oidc-` and fail with a clear message pointing at the
`spec.oidc.mode: CustomConfig` migration path (they can carry the
same issuer/audience inside `customConfig.config`).

When `spec.oidc.mode: None` the passthrough still works — customers
who manage OIDC entirely by hand keep working until they opt into
the new selector explicitly.

Also update the values.yaml docstring on `extraArgs` to steer OIDC
users to `spec.oidc.mode` and warn against the collision.

Two new helm-unittest cases: the collision fails, and mode=None with
legacy flags renders normally (backward compat).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 0d70d5ff3e feat(system/kubernetes-rd): expose OIDC kubeconfig Secret to the dashboard
Add `kubernetes-<name>-oidc-kubeconfig` to the Kubernetes
ApplicationDefinition's `spec.secrets.include[].resourceNames` so
the dashboard surfaces the OIDC kubeconfig to tenant viewers
alongside the existing admin kubeconfig.

The Secret is produced by the post-install,post-upgrade bootstrap
Job in `apps/kubernetes` when `spec.oidc.mode: System` is set.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Ivan Okhotnikov 5950450758 feat(apps/kubernetes): add Kubernetes.spec.oidc selector API
Introduce a flat selector — `mode: System | CustomConfig | None` —
plus a `users[]` list and a `customConfig` block for tenant-supplied
AuthenticationConfiguration. Default `None` preserves today's
behavior.

The `System` mode trusts the platform `cozy` realm via a per-cluster
public client; `CustomConfig` accepts an inline `AuthenticationConfiguration`
or a Secret reference (mutually exclusive). The `users[]` list drives
per-user ClusterRoleBindings inside the tenant cluster, with two
roles: `admin` and `view`.

Regenerates deepcopy, JSON schema, README, and the kubernetes-rd
cozyrds embedded schema.

Implements Phase 1 API of the per-tenant OIDC design proposal
(cozystack/community#24).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-06 09:25:07 +03:00
Andrei Kvapil 59f035ff53 fix(kubevirt-instancetypes): restore persistent EFI/TPM state
Reverts #3006. Dropping persistent EFI/TPM was motivated by a live-migration
concern that rested on an outdated assumption — that an RWO Filesystem
backend-storage PVC pins the VM to its node. KubeVirt has migrated
RWO-Filesystem backend storage since v1.4 (kubevirt/kubevirt#12629): on
migration it creates a fresh target state PVC and copies the small state blob,
so persistent-EFI/TPM VMs on the default replicated storage live-migrate fine
(verified: the VM reports LiveMigratable=True and the copy-on-target PVC is
created on migration). Restore upstream's persistent firmware state for the
windows.* preferences.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-06 09:49:50 +05:00
Aleksei Sviridkin a44d11d2b6 fix(cozystack-basics): validate legacy Ingress hostnames against the tenant apex (#3200)
## What this PR does

The Gateway (`cozystack-gateway-hostname-policy`) and Route
(`cozystack-route-hostname-policy`) admission policies constrain tenant
hostnames to the tenant apex on the Gateway API path. Gateway API is
opt-in and off by default, so in the default configuration tenant
applications publish through a legacy `networking.k8s.io/Ingress` on the
shared ingress-nginx, which had no equivalent policy.

This adds `cozystack-ingress-hostname-policy`, a fail-closed
`ValidatingAdmissionPolicy` + binding for `Ingress` in `tenant-*`
namespaces — the legacy-Ingress counterpart of the existing
Gateway/Route hostname policies, and a new Layer 8 in the gateway
security model. A hostname on `spec.rules[].host` or
`spec.tls[].hosts[]` is allowed when it is within the namespace's own
apex (`namespace.cozystack.io/host`), or when it is a concrete
(non-wildcard) name entirely outside the platform root apex
(`_cluster.root-host`). The second case keeps the `kubernetes` app's
Proxied `exposeMethod` working, since it routes a user-supplied external
domain to a nested cluster. A hostname under the platform root apex but
outside the namespace's own apex, a wildcard taken through the
outside-root path (which could otherwise match the platform apex
itself), a rule with no host (absent or empty — an unbounded catch-all),
and `spec.defaultBackend` are rejected.

The policy is fail-closed: a `tenant-*` namespace missing its own apex
label is denied, and the policy renders only when `_cluster.root-host`
is set, so it is never shipped in a state that would admit every
hostname. `cozy-*` namespaces are cluster-admin-managed and not gated.
This is defense-in-depth in the same family as the Gateway/Route
policies — by design tenants hold only read access to `Ingress`.

The change ships helm-unittest coverage (render, plus the root-host,
empty-host, and wildcard guards) and behavioral admission tests against
the live apiserver in `hack/e2e-apps/gateway.bats` (in-apex allow,
external-domain allow, under-root-outside-apex deny,
wildcard-matching-apex deny, hostless deny, empty-string-host deny,
defaultBackend deny), and documents the new layer in
`packages/extra/gateway/README.md`.

### Screenshots

N/A — no UI change.

### Release note

```release-note
fix(cozystack-basics): constrain legacy Ingress hostnames in tenant namespaces to the tenant apex
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added **fail-closed** defense-in-depth Layer 8
(`cozystack-ingress-hostname-policy`) to enforce allowed legacy Ingress
rule and TLS hostnames for tenant namespaces, while blocking unsafe
catch-all configurations (including missing/empty hosts and
`spec.defaultBackend`) when the platform root host is configured.
* **Tests**
* Added end-to-end and render/deny validation for permitted vs denied
hostname patterns, wildcard behavior, required namespace host labeling,
and correct deny/fail-closed rendering.
* **Documentation**
  * Updated the gateway security model to document Layer 8 behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-05 00:18:16 +03:00
IvanHunters a1c779854b fix(vertical-pod-autoscaler): migrate admission-controller TLS to cert-manager to stop reconcile-driven ReplicaSet churn (#3178)
## Summary

The VPA admission-controller webhook TLS Secret is currently rendered by
an in-chart `genCA` + `genSignedCert` template. Those functions are not
idempotent across `helm template` invocations — every Flux HelmRelease
reconcile produces a fresh CA + cert. A `checksum/tls-secret`
podAnnotation on the admission-controller Deployment then flips its
sha, materialising a new ReplicaSet on every reconcile.

Under load — the E2E install phase where cert-manager / CDI / LINSTOR
cascade races repeatedly refire the VPA HelmRelease — this produces
10+ admission-controller ReplicaSets in minutes and blocks the whole
cozystack-platform HR: the admission-controller pod livelocks in
startup (waits on its own election lease which its previous
ReplicaSet is still holding, hits livenessProbe timeout,
CrashLoopBackOffs), VPA never becomes Ready, PackageSources stay
InProgress, and cozystack-platform aborts at its 15-minute install
timeout.

Fix: replace the source of the TLS Secret with a namespace-scoped
cert-manager `Issuer` + `Certificate`. cert-manager only rewrites the
Secret when the certificate is close to expiry (`renewBefore`), so
reconcile-driven cert churn stops. Pod rotation is delegated to a
runtime-lookup-based `checksum/tls-cert` podAnnotation that only
changes when the Secret's `tls.crt` content actually changes — not on
every render, not on every reconcile.

## What

-
`charts/vertical-pod-autoscaler/templates/admission-controller/certificate.yaml`
  (new): namespace-scoped selfSigned Issuer + Certificate. The
  Certificate mirrors the legacy `genSignedCert` shape (single DNS SAN
  `vpa-webhook.<ns>.svc`, 365d validity, matching Secret name), so
  cert-manager on the first upgrade adopts the same Secret name the
  legacy path wrote — no rename cascade in the Deployment volume mount.
  Rendered only when `admissionController.tls.certManagerEnabled=true`
  AND no `existingSecret` is supplied.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/tls-secret.yaml`
  (modified): wrapped in an extra `certManagerEnabled=false` guard so
  the legacy genCA path remains available as an explicit opt-out.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/deployment.yaml`
  (modified): the legacy `checksum/tls-secret` annotation is gated
  behind `certManagerEnabled=false`. Under the cert-manager path the
  Deployment carries a `checksum/tls-cert` annotation computed from
  `lookup`-ing the runtime Secret's `tls.crt` — see the template
  comment for the full timing analysis and the day-350 renewal
  trade-off it documents.
- `charts/vertical-pod-autoscaler/values.yaml` (modified): new
  `admissionController.tls.certManagerEnabled` value defaulting to
  `true`. Values-block header documents the switch as effectively
  one-way (cert-manager writes `type: kubernetes.io/tls`;
  `Secret.type` is immutable, so falling back requires `kubectl
  delete` first).
- `patches/cert-manager-admission-tls.diff` (new) +
  `Makefile update` extended: the four subchart edits above live as a
  single reviewable diff that Makefile re-applies over the freshly
  vendored chart on every `make update`, matching the fluxcd-operator
  package convention. Prevents a bump from silently reverting the
  fix.
- `packages/core/platform/sources/vertical-pod-autoscaler.yaml`
  (modified): VPA PackageSource now depends on `cozystack.cert-manager`.
  Non-circular: cert-manager depends on `cozystack.networking` +
  `cozystack.gateway-api-crds`, neither reaches VPA.
- `tests/vpa_test.yaml` (modified): +11 helm-unittest cases (15 total,
  4 pre-existing) covering `certManagerEnabled={true,false}`,
  `existingSecret` bypass, disabled admission-controller, and
  the offline-render behaviour of the `checksum/tls-cert` lookup.

## Why (evidence)

Snapshot from CI run 28587656430 (this branch, pre-workaround-review):

```
admission-controller ReplicaSets after first-install churn window:
  vertical-pod-autoscaler-admission-controller-55d888d6d4
  vertical-pod-autoscaler-admission-controller-56f8d4c747
  vertical-pod-autoscaler-admission-controller-5775dbc9b7
  vertical-pod-autoscaler-admission-controller-585b7c56c9
  vertical-pod-autoscaler-admission-controller-6fc44944fb
  vertical-pod-autoscaler-admission-controller-74bccf4849
  vertical-pod-autoscaler-admission-controller-84df98f98b
  vertical-pod-autoscaler-admission-controller-8666dffb85
  vertical-pod-autoscaler-admission-controller-86b469457c
  vertical-pod-autoscaler-admission-controller-9b979f9d6
  → 10 ReplicaSets, one per Flux reconcile, each with a different
    genCA-rooted TLS Secret sha in checksum/tls-secret.
```

## Migration for existing clusters

On upgrade the legacy Opaque Secret disappears from the release
manifest, Helm garbage-collects it, and cert-manager immediately
writes a fresh `kubernetes.io/tls` Secret of the same name via the
new Certificate. `Secret.type` is immutable, so in-place adoption is
impossible — the delete/create pass is the correct path, and the
seconds-long window it produces is safe: the `vpa-webhook-config`
MutatingWebhookConfiguration ships with `failurePolicy: Ignore`
(verified on cozy-dev3), so Pod admissions in that window skip VPA
mutation but never get rejected — no cluster-wide disruption.

The Deployment rolls in the same upgrade because the new
`checksum/tls-cert` annotation differs from anything the pod had
before, so the admission-controller re-registers `vpa-webhook-config`
with the fresh caBundle on the same pass.

## Test plan

- [x] `helm unittest packages/system/vertical-pod-autoscaler` — 15/15
      pass.
- [x] `helm template` diff between main and this branch is scoped
      exactly to the cert-manager path (new Certificate + Issuer,
      new `checksum/tls-cert` annotation, guarded legacy Secret, new
      values key, patch file + Makefile step, PackageSource
      dependsOn). No unrelated lines change.
- [x] `make update`-style simulation (rm -rf charts && helm pull &&
      patch) produces a byte-identical subchart to what git tracks —
      the patch stays clean across future upstream bumps until it
      genuinely conflicts.
- [ ] Live migration on cozy-dev3: `helm upgrade` VPA release from
      pre-fix version → observe cert-manager write the new tls Secret
      → verify `checksum/tls-cert` on the new pod's podTemplate →
      verify MutatingWebhookConfiguration `vpa-webhook-config`
      caBundle updated in place → confirm no ReplicaSet churn on
      subsequent Flux reconciles. Blocking the draft's removal.
- [ ] Full E2E on this branch — verify only 1 admission-controller
      ReplicaSet after install, VPA HR reaches Ready inside
      `cozystack-platform`'s 15m install timeout, and
      `kubernetes-latest.bats` / `kubernetes-previous.bats` no longer
      time out on VPA-attributable resource churn.

## Review pass — kvaps' CHANGES_REQUESTED addressed

- ✅ Vendored chart mods now carried as `patches/*.diff` re-applied by
  `Makefile update` (matches fluxcd-operator convention).
- ✅ Retired the tls-migrator hook — kvaps correctly showed it was
  a no-op: Helm's delete decision is driven by the stored release
  manifest not by live labels (`ResourcePolicyAnno` is the only opt-
  out), and `Secret.type` is immutable so cert-manager could not have
  adopted the legacy Opaque Secret in place anyway. ~200 lines
  removed.
- ✅ PR description rewritten to match the actual diff (no more
  reference to the removed `tls-rollout.yaml` post-upgrade hook, real
  test count, dev3 verification honestly listed as pending).
- ✅ `checksum/tls-cert` timing comment corrected: `lookup` runs on
  Helm actions only, day-350 renewal trade-off spelled out.
- ✅ `certManagerEnabled=false` documented as effectively one-way
  (Secret.type immutable).
- ✅ Dropped the `secretTemplate` labels — `app.kubernetes.io/managed-
  by: Helm` on a cert-manager-owned Secret was misleading.
- ✅ Banner comment indentation restored in `vpa_test.yaml`.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* VPA admission-controller webhook TLS now uses cert-manager by default,
with options to disable cert-manager or supply an existing TLS secret.
* Webhook pod restarts are now tied to cert-manager Secret contents via
a content-based checksum.
* **Bug Fixes**
* Improved webhook TLS provisioning reliability when TLS readiness is
delayed.
* Corrected conditional rendering/rollout behavior to avoid mismatched
TLS ownership.
* **Tests**
* Added rendering-invariant tests covering cert-manager, legacy TLS,
existing-secret bypass, and admission-controller disabled behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-04 22:48:00 +03:00
Aleksei Sviridkin d89ad70698 fix(cozystack-basics): validate legacy Ingress hostnames against the tenant apex
The Gateway (cozystack-gateway-hostname-policy) and Route
(cozystack-route-hostname-policy) admission policies constrain tenant
hostnames on the Gateway API path. Gateway API is opt-in and off by
default, so in the default configuration tenant applications publish
through a legacy networking.k8s.io/Ingress on the shared ingress-nginx,
which had no equivalent policy.

Add a fail-closed ValidatingAdmissionPolicy + Binding for Ingress in
tenant-* namespaces. A hostname on spec.rules[].host or spec.tls[].hosts[]
is allowed when it is within the namespace's own apex
(namespace.cozystack.io/host), or when it is a concrete (non-wildcard)
name entirely outside the platform root apex (_cluster.root-host); a
hostname under the platform root apex but outside the namespace's own
apex, and a wildcard taken through the outside-root path (which could
match the platform apex itself), are rejected. The outside-root branch
keeps the kubernetes app's Proxied exposeMethod working, since it routes
a user-supplied external domain to a nested cluster. A rule with no host
(absent or empty, an unbounded catch-all) and spec.defaultBackend are
also rejected.

The policy templates the platform root apex into its CEL and renders
only when _cluster.root-host is set, so it is never shipped in a state
that would admit every hostname. A tenant-* namespace missing its own
apex label is denied.

Cover the policy with helm-unittest (render, including the root-host,
empty-host, and wildcard guards) and behavioral admission tests against
the live apiserver (in-apex allow, external-domain allow,
own-apex-wildcard allow, external-tls allow, under-root-outside-apex
deny, prefix-adjacent-under-root deny, wildcard-matching-apex deny,
hostless deny, empty-string-host deny, defaultBackend deny), and document
it as an additional layer in the gateway security model.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-04 22:38:39 +03:00
Ivan Okhotnikov 223c0ef4fb fix(cert-manager): raise cainjector memory limit to unblock caBundle injection
cainjector loads all ValidatingWebhookConfigurations, APIServices and CRDs
into its informer caches at startup. On a full cozystack install that working
set exceeds the 128Mi limit, so the single leader-elected cainjector Pod is
OOMKilled during cache population and CrashLoops before it injects any
caBundle. Webhook ValidatingWebhookConfigurations then keep an empty caBundle
and every admission call fails with "x509: certificate signed by unknown
authority" (failurePolicy: Fail), intermittently blocking Ingress creation
and stalling tenant installs.

Raise the cainjector memory limit to 512Mi (matching the controller) and the
request to 128Mi, and add a helm-unittest that pins the sizing so a blanket
resource change or a `make update` cannot silently lower it again.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
2026-07-04 15:57:49 +03:00
Aleksei Sviridkin 10f8e2de1f feat(kamaji): default to 2 replicas, drop telemetry webhook (#2671)
## What this PR does

`kamaji-controller-manager` runs **9 sub-controllers per
TenantControlPlane** (bootstraptoken, coredns, konnectivity, kubeadm*,
kubelet, kubeproxy, …) inside a single Go process, **and** serves the
admission webhooks for `tenantcontrolplanes` + `datastores` from the
same pod via `kamaji-webhook-service`. With the upstream default
`replicaCount=1`, every webhook round-trip from the apiserver shares CPU
with the reconciler goroutines. On a multi-tenant cluster this pushes
the apiserver-observed p99 of `mtenantcontrolplane.kb.io` and
`vtenantcontrolplane.kb.io` from the kamaji-side `~10ms` (per
`controller_runtime_webhook_latency_seconds_count/sum`) up to `85–95ms`
— purely due to CPU contention.

Three safe defaults are changed:

- **`kamaji.replicaCount: 2`** — HA-safe baseline. `--leader-elect` is
already enabled upstream, so only one replica reconciles tenants; the
follower's CPU is fully available for webhook handlers. On a single-node
cluster both pods still schedule (just colocated).
- **`kamaji.telemetry.disabled: true`** — drops the
`--disable-telemetry` handler **and** removes the
`telemetry.kamaji.clastix.io` entry from the
`ValidatingWebhookConfiguration`. The latter requires `patches/1.diff`
to wrap the telemetry block in `{{- if not .Values.telemetry.disabled }}
… {{- end }}` inside the vendored
`controller-gen/validating-webhook.yaml`. Without the webhook entry, the
apiserver no longer round-trips to a no-op handler on every
TenantControlPlane CREATE/UPDATE/DELETE.
- **`kamaji.affinity` (soft `podAntiAffinity`)** — spread the two
replicas across nodes when possible, while keeping single-node clusters
working (`preferredDuringScheduling`, not required).

The `patches/1.diff` approach (re-applied by `make update`) mirrors the
pattern already used in `packages/system/fluxcd-operator/Makefile` and
`packages/system/ingress-nginx/Makefile`, and survives the next
`clastix/kamaji` chart bump.

## Measured impact

Hikube prod, 22 TenantControlPlanes, before/after a manual rollout of
the same configuration:

| webhook | p99 before | p99 after | Δ |
| --- | --- | --- | --- |
| `mtenantcontrolplane.kb.io` | 85 ms | 25 ms | **-71%** |
| `vtenantcontrolplane.kb.io` | 92 ms | 25 ms | **-73%** |
| `telemetry.kamaji.clastix.io` | 94 ms | _webhook removed_ | **-100%**
|

Cumulative Kamaji admission **impact** (`p99 × calls/s`) dropped from
**830 → 140 (-83%)**. The Kamaji webhooks moved from the top three of
the `apiserver_admission_webhook_admission_duration_seconds_*` impact
ranking down to positions #3-#4.

## Release note

```release-note
feat(kamaji): default to 2 replicas with soft pod anti-affinity and disable telemetry (handler + webhook entry). Reduces apiserver admission p99 for tenantcontrolplane mutating/validating webhooks by ~70% on multi-tenant clusters.
```

## Test plan

- [x] `helm template packages/system/kamaji` renders `replicas: 2`,
`--disable-telemetry`, the soft anti-affinity, and **no**
`telemetry.kamaji.clastix.io` entry in the
`ValidatingWebhookConfiguration`.
- [x] `patches/1.diff` applies cleanly against the freshly extracted
upstream chart (`patch --dry-run -p1 < patches/1.diff`).
- [x] `make update` re-applies the patch automatically.
- [x] Live deploy on a 22-TCP cluster: 3 pods Running (when overriden to
`replicaCount: 3` via cozystack-values), zero pod restarts, leader
election visible, `kamaji-webhook-service` Endpoints serving N IPs.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Telemetry can be disabled via configuration.
  * Controller configured for high availability with 2 replicas.
* Pod anti-affinity added to spread replicas across nodes for better
resilience.

* **Bug Fixes**
* Validating webhook is now omitted when telemetry is disabled to avoid
unintended behavior.

* **Chores**
* Update process improved to re-apply chart patches after extraction for
reproducible updates.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2671?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-03 19:31:51 +03:00
Aleksei Sviridkin 809a04a220 feat(kubevirt-cdi): make cloneStrategyOverride configurable (#2926)
## What this PR does

- Exposes the CDI `cloneStrategyOverride` as a Helm value in
`packages/system/kubevirt-cdi` instead of hardcoding it in the CR
template.
- Default stays `csi-clone`, so **behavior is unchanged** for everyone.
- It can now be overridden via a cozystack `Package` (e.g.
`cloneStrategyOverride: copy`), the same way `uploadProxyURL` already is
— no fork needed.

### Why

The value was hardcoded, so the only way to change the clone strategy
was to fork the package. On some storage backends `csi-clone` is
problematic: with LINSTOR/DRBD a CSI clone is created co-located with
the source golden image and then relocated to optimal nodes
asynchronously; that async relocation (`migrate-disk`) can race the
post-clone resize CDI performs, occasionally leaving a volume whose DRBD
device is smaller than its PVC capacity. Being able to select `snapshot`
or `copy` per cluster, without maintaining a fork, is a useful escape
hatch.

This only adds a knob; it does not change the default or recommend a
strategy.

## Release note

```release-note
feat(kubevirt-cdi): make CDI cloneStrategyOverride configurable via values (default csi-clone)
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* The KubeVirt CDI clone strategy is now configurable via Helm chart
values, allowing selection of `csi-clone`, `snapshot`, or `copy` for
DataVolume cloning.
* The default remains `csi-clone` to preserve existing behavior and
ensure backward compatibility.
* **Documentation**
* Added the `cloneStrategyOverride` Helm value description and allowed
options to the chart’s configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-03 19:31:47 +03:00
Aleksei Sviridkin 901bb981ae fix(seaweedfs): configure postgres2 connection pool for the filer (#2906)
## What this PR does

- Adds `WEED_POSTGRES2_CONNECTION_MAX_IDLE=20`,
`WEED_POSTGRES2_CONNECTION_MAX_OPEN=40` and
`WEED_POSTGRES2_CONNECTION_MAX_LIFETIME_SECONDS=600` to the filer
`extraEnvironmentVars` in `packages/system/seaweedfs/values.yaml`.

The vendored seaweedfs chart only ships connection pool settings for the
**mysql** store (`WEED_MYSQL_CONNECTION_MAX_*`), which cozystack does
not use. The **postgres2** store, which cozystack enables, has no pool
settings at all. As a result the Go `database/sql` pool runs with
`SetMaxIdleConns(0)`: every filer metadata lookup opens a fresh
PostgreSQL connection (TCP + TLS + SCRAM, ~300 ms each). With ~5
path-component lookups per S3 operation, this adds roughly **2 seconds
of latency to every S3 request**, even on an idle cluster.

### Measurements (production cluster, 12 nodes, 2 filers, CNPG backend)

Before / after applying these three variables, same test (sequential ops
through the S3 gateway):

| Operation | Before | After |
|---|---|---|
| PUT 4 KiB (p50) | 10.0 s | 0.28 s |
| GET 4 KiB (p50) | 3.0 s | 0.14 s |
| HEAD (p50) | 2.6 s | 0.16 s |
| Bounded LIST, 100 keys (p50) | 3.8 s | 0.37 s |

`pg_stat_activity` confirmed the root cause: before the fix every
backend was younger than ~200 ms (permanent connection churn); after the
fix connections are reused for minutes. Real-traffic p95 of
`SeaweedFS_filerStore_request_seconds` dropped from 1.3–6.5 s to ~0.18
s.

`max_open` is capped at 40 so that the default two filer replicas stay
below the CNPG default `max_connections` of 100.

## Release note

```release-note
fix(seaweedfs): configure postgres2 connection pool for the filer, removing ~2s of per-request S3 latency caused by PostgreSQL connection churn
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Optimized PostgreSQL connection pool configuration in SeaweedFS to
reduce latency and improve performance during metadata operations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-03 19:31:44 +03:00
Aleksei Sviridkin ec9c29948b [reloader] Set resources requests/limits on reloader container (#2631)
## Summary

The upstream Stakater reloader chart ships with `resources: {}`, which
triggers a `no CPU limit` conformance warning on the `reloader`
Deployment.

Override `packages/system/reloader/values.yaml` at
`reloader.reloader.deployment.resources` with the exact values from the
chart's own commented example:

- `limits` : `cpu: 100m`, `memory: 512Mi`
- `requests` : `cpu: 10m`, `memory: 128Mi`

Reloader is a light controller that watches ConfigMap / Secret changes
and triggers workload restarts — small CPU envelope with enough memory
to track many objects in clusters with lots of namespaces.

## Test plan

- [ ] `helm template . --namespace cozy-reloader` from
`packages/system/reloader/` renders a `resources` block on the
`reloader-reloader` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `reloader-reloader has no
CPU limit` warning should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Set CPU/memory requests and limits on the reloader Deployment to clear
the "no CPU limit" conformance warning. Values follow the upstream
chart's commented example (limits 100m/512Mi, requests 10m/128Mi).
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated resource allocation configuration to define CPU and memory
parameters for system component deployments, enhancing infrastructure
stability and resource optimization.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2631)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-03 19:31:39 +03:00
Aleksei Sviridkin f2d66579f1 [metrics-server] Set resources requests/limits on metrics-server container (#2630)
## Summary

The upstream metrics-server chart ships with `limits` commented out
(only `requests cpu 100m / memory 200Mi` are set), which triggers a `no
CPU limit` conformance warning on the `metrics-server` Deployment.

Override `packages/system/metrics-server/values.yaml` to keep the
upstream-recommended baseline `requests` and add `limits`:

- `limits` : `cpu: 500m`, `memory: 500Mi`
- `requests` : `cpu: 100m`, `memory: 200Mi`

Comfortable headroom on top of the [official sizing
formula](https://github.com/kubernetes-sigs/metrics-server#configuration)
(`100m + 1m/node`, `200Mi + 2Mi/node`), covering clusters up to ~150
nodes without needing to revisit. Larger clusters can override these
values further.

## Test plan

- [ ] `helm template . --namespace cozy-monitoring` from
`packages/system/metrics-server/` renders a `resources` block on the
`metrics-server` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `metrics-server has no CPU
limit` warning should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Set CPU/memory requests and limits on the metrics-server Deployment to
clear the "no CPU limit" conformance warning. Values follow the upstream
sizing recommendation (requests 100m/200Mi) with a comfortable limits
cap (500m/500Mi).
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Configured explicit resource requests and limits for the
metrics-server service (CPU: 100m-500m, Memory: 200Mi-500Mi) to ensure
consistent resource allocation.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2630)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-03 19:31:36 +03:00
Aleksei Sviridkin 4952909e4d [external-dns] Set resources requests/limits on external-dns container (#2629)
## Summary

The upstream external-dns chart ships with `resources: {}`, which
triggers a `no CPU limit` conformance warning on the `external-dns`
Deployment.

Override `packages/system/external-dns/values.yaml` to set:

- `limits` : `cpu: 100m`, `memory: 128Mi`
- `requests` : `cpu: 10m`, `memory: 64Mi`

Modest sizing for a controller that just watches Service / Ingress
objects and pushes DNS records to the provider.

## Test plan

- [ ] `helm template . --namespace cozy-external-dns` from
`packages/system/external-dns/` renders a `resources` block on the
`external-dns` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `external-dns has no CPU
limit` warning should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Set CPU/memory requests and limits on the external-dns Deployment to
clear the "no CPU limit" conformance warning.
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated ExternalDNS resource allocation with optimized CPU and memory
limits and requests for improved resource efficiency.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2629)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-03 19:31:32 +03:00
Aleksei Sviridkin 04a05f623c [bucket] Add resources requests/limits to s3manager container (#2624)
## Summary

The `s3manager` container in
`packages/system/bucket/templates/deployment.yaml` had no `resources`
block, triggering a `no CPU limit` conformance warning on the bucket UI
Deployment.

The bucket package is a standalone chart (no vendored upstream), so the
values are set directly in the template — same pattern as the
csi-resizer fix in #2614 and the existing `capi-providers-cpprovider`
template.

- `limits` : `cpu: 500m`, `memory: 512Mi`
- `requests` : `cpu: 100m`, `memory: 128Mi`

Comfortable headroom for the lightweight S3 browser UI while keeping a
small baseline request.

## Test plan

- [ ] `helm template . --show-only templates/deployment.yaml` from
`packages/system/bucket/` renders the `resources` block on the
`s3manager` container (verified locally)
- [ ] Re-run the conformance/lint tool — the `s3manager has no CPU
limit` warning should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Add CPU and memory requests/limits to the s3manager container of the
bucket UI Deployment to clear the "no CPU limit" conformance warning.
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Improved deployment configuration with explicit resource allocation
settings to ensure optimal service stability and resource management.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2624)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-03 19:31:25 +03:00
Aleksei Sviridkin 6762d2d95c [goldpinger] Set resources requests/limits on goldpinger DaemonSet (#2623)
## Summary

The upstream Bloomberg goldpinger chart ships with `resources: {}`,
which triggers a `no CPU limit` conformance warning on the
`goldpinger-daemon` container running on every node.

Override `packages/system/goldpinger/values.yaml` to set:

- `limits` : `cpu: 200m`, `memory: 256Mi`
- `requests` : `cpu: 50m`, `memory: 64Mi`

Comfortable margin for the ping-mesh fan-out on larger clusters while
keeping a small baseline request so the DaemonSet schedules everywhere.

## Test plan

- [ ] `helm template . --namespace cozy-goldpinger` from
`packages/system/goldpinger/` renders the `resources` block on the
`goldpinger-daemon` container of the DaemonSet (verified locally)
- [ ] Re-run the conformance/lint tool — the `goldpinger has no CPU
limit` warning should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Set CPU/memory requests and limits on the goldpinger DaemonSet to clear
the "no CPU limit" conformance warning.
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated Kubernetes resource requirements for the goldpinger component,
defining CPU limits (200m), memory limits (256Mi), CPU requests (50m),
and memory requests (64Mi).

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2623)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-03 19:31:22 +03:00
mattia-eleuteri 8bc62cca1c feat(kamaji): default to 2 replicas, drop telemetry webhook
The kamaji-controller-manager runs ~9 sub-controllers per
TenantControlPlane (bootstraptoken, coredns, konnectivity, kubeadm*,
kubelet, kubeproxy, etc.) inside a single Go process, AND serves the
admission webhooks for tenantcontrolplanes/datastores via the
kamaji-webhook-service ClusterIP. With the upstream default
replicaCount=1, every webhook RT shares CPU with the reconciler
goroutines. On a multi-tenant cluster (~20 TCPs), this pushes
apiserver-observed p99 for mtenantcontrolplane.kb.io and
vtenantcontrolplane.kb.io from the kamaji-side ~10ms to 85-95ms.

Three changes, each safe on small clusters and high-impact on large ones:

* replicaCount=2 — HA-safe baseline. --leader-elect is already enabled
  upstream, so only one replica reconciles tenants; the follower's CPU
  is fully available for webhook handlers, cutting p99 ~70% on busy
  clusters. On a single-node cluster both pods schedule normally.

* telemetry.disabled=true — drops both the --disable-telemetry handler
  and the telemetry.kamaji.clastix.io admission webhook entry from the
  ValidatingWebhookConfiguration. The latter requires patches/1.diff to
  wrap the telemetry block in a {{ if not .Values.telemetry.disabled }}
  conditional inside the vendored controller-gen file. Without the
  webhook entry, the apiserver no longer issues a TCP round-trip to a
  no-op handler on every TenantControlPlane CREATE/UPDATE/DELETE.

* podAntiAffinity (preferred) — spread the two replicas across nodes
  when possible. preferredDuringScheduling so single-node clusters
  still schedule (both pods colocated, which is acceptable).

Measured on a 22-TCP production cluster (Hikube, before/after):
mtenantcontrolplane p99 85 → 25 ms (-71%), vtenantcontrolplane p99
92 → 25 ms (-73%), telemetry webhook impact 291 → 0. Total Kamaji
admission impact -83% (830 → 140).

The patches/1.diff approach (re-applied by `make update`) mirrors the
pattern already used in fluxcd-operator and ingress-nginx and survives
the next clastix/kamaji chart bump.

Signed-off-by: mattia-eleuteri <mattia@hidora.io>
2026-07-03 19:14:36 +03:00
mattia-eleuteriandClaude Opus 4.8 c47bb6edb0 feat(kubevirt-cdi): make cloneStrategyOverride configurable
The CDI cloneStrategyOverride was hardcoded to csi-clone in the CR
template, so operators could only change it by forking the package.
Expose it as a Helm value (default csi-clone, behavior unchanged) so it
can be set via a cozystack Package override, the same way uploadProxyURL
already is.

This is useful where the csi-clone path is problematic on a given
storage backend (e.g. LINSTOR/DRBD, where a clone is co-located with the
source and relocated afterwards; the async relocation can race the
post-clone resize), letting operators select snapshot or copy without
maintaining a fork.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
2026-07-03 19:14:33 +03:00
mattia-eleuteri 49cbea8c73 fix(seaweedfs): configure postgres2 connection pool for the filer
The vendored seaweedfs chart ships connection pool settings only for the
mysql store, which cozystack does not use. The postgres2 store, which is
enabled, has no pool settings, so the Go sql pool keeps zero idle
connections and every filer metadata lookup opens a fresh PostgreSQL
connection (TCP + TLS + SCRAM, ~300ms each). With ~5 path lookups per S3
operation this adds roughly 2 seconds of latency to every S3 request,
regardless of cluster load.

Measured on a production cluster (12 nodes, 2 filers): PUT of a 4KiB
object went from 10s (p50) to 0.28s, HEAD from 2.6s to 0.16s after
setting these values; pg_stat_activity confirmed connections were
previously recycled every ~200ms and are now reused.

max_open is capped at 40 so that two filer replicas stay below the CNPG
default max_connections of 100.

Signed-off-by: mattia-eleuteri <mattia@hidora.io>
2026-07-03 19:14:30 +03:00
Matthieu adb307cbae [goldpinger] Set resources requests/limits on goldpinger DaemonSet
The Bloomberg goldpinger chart ships with resources: {}, which triggers
a 'no CPU limit' conformance warning on the goldpinger-daemon container
deployed on every node. Override the package values.yaml to set
limits 200m/256Mi and requests 50m/64Mi — comfortable margin for the
ping-mesh fan-out on larger clusters while keeping a small baseline
request.

Signed-off-by: Matthieu <matthieu@hidora.com>
2026-07-03 19:14:27 +03:00
Matthieu bf54945c08 [reloader] Set resources requests/limits on reloader container
The upstream Stakater reloader chart ships with resources: {}, which
triggers a 'no CPU limit' conformance warning on the reloader
Deployment. Apply the exact values from the chart's own commented
example (limits 100m/512Mi, requests 10m/128Mi) — reloader is a light
controller that watches ConfigMap/Secret changes and triggers restarts,
so a small CPU envelope with enough memory to track many objects fits.

Signed-off-by: Matthieu <matthieu@hidora.com>
2026-07-03 19:14:21 +03:00
Matthieu 3c2133ce4b [metrics-server] Set resources requests/limits on metrics-server container
The upstream metrics-server chart ships with limits commented out
(requests cpu 100m / memory 200Mi only), which triggers a 'no CPU
limit' conformance warning. Override the package values.yaml to keep
the upstream-recommended baseline requests and add limits 500m/500Mi
— comfortable headroom on top of the official sizing formula
(100m + 1m/node, 200Mi + 2Mi/node), covering clusters up to ~150 nodes
without needing to revisit.

Signed-off-by: Matthieu <matthieu@hidora.com>
2026-07-03 19:14:18 +03:00
Matthieu 7f72e7bbe8 [external-dns] Set resources requests/limits on external-dns container
The upstream external-dns chart ships with resources: {}, which triggers
a 'no CPU limit' conformance warning on the external-dns Deployment.
Override the package values.yaml to set limits 100m/128Mi and requests
10m/64Mi — modest sizing for a controller that just watches Service /
Ingress objects and pushes DNS records to the provider.

Signed-off-by: Matthieu <matthieu@hidora.com>
2026-07-03 19:14:15 +03:00
Matthieu c8b180d91a [bucket] Add resources requests/limits to s3manager container
The s3manager Deployment in the bucket package had no resources block,
triggering a 'no CPU limit' conformance warning. Set limits 500m/512Mi
and requests 100m/128Mi — comfortable headroom for the lightweight S3
browser UI. Bucket is a standalone chart (no vendored upstream), so
the values are set directly in the template, matching the existing
pattern in csi-resizer and capi-providers.

Signed-off-by: Matthieu <matthieu@hidora.com>
2026-07-03 19:14:09 +03:00
Matthieu cd95040c58 [cert-manager] Set resources requests/limits on cert-manager pods
The upstream cert-manager chart ships with resources: {} for all four
workloads (controller, webhook, cainjector, startupapicheck), which
triggers 'no CPU limit' conformance warnings on every pod. Override the
package values.yaml to set sane requests and limits sized to each
component's role: controller is the most loaded, webhook and cainjector
handle short-lived admission/injection work, and startupapicheck is a
one-shot Job.

Signed-off-by: Matthieu <matthieu@hidora.com>
2026-07-03 19:14:05 +03:00
Ivan Okhotnikov 2ac269a04b docs(vertical-pod-autoscaler): fix stale ~350d in checksum/tls-cert ascii timeline
Second self-review pass caught a remaining reference to the old
rotation math in the checksum/tls-cert doc block's ascii timeline
(`renewal (~350d)` — from `renewBefore: 360h` when 365 - 15 = 350).
With the widened `renewBefore: 2160h` rotation now lands at day ~275,
matching the bullet-list narrative directly below. Update the number
in place and regenerate the vendored-chart patch.

No functional change; one-line doc fix.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-03 16:04:46 +03:00
Ivan Okhotnikov d6e57d67e7 docs(vertical-pod-autoscaler): sync deployment.yaml trade-off narrative with widened renewBefore
Self-review caught a stale day-math block in deployment.yaml that still
described the previous `renewBefore: 360h` behaviour ("cert-manager
renews at day ~350 and NO Helm action happens in the following ~15
days"). After bumping `renewBefore` to 2160h in the previous commit,
rotation now lands at day ~275 with a ~90-day window before expiry.

Update the numbers in place and regenerate the vendored-chart patch so
`make update` re-applies the corrected text instead of silently
reverting to the stale description on the next chart refresh.

No functional change — cert rotation math already lives in
`certificate.yaml`; this only fixes the parallel rationale block a
future reader would read alongside it.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-03 16:02:04 +03:00
Ivan Okhotnikov 79ce69a3b4 fix(vertical-pod-autoscaler): widen renewBefore + document selfSigned checksum, add existingSecret belt-and-suspenders assertion
Address the three non-blocking follow-ups on myasnikovdaniil's approval:

* renewBefore: 360h → 2160h (15d → 90d). With webhook `failurePolicy:
  Ignore`, an expired in-memory cert silently drops admission traffic
  until a Helm action rolls the ReplicaSet. helm-controller does not
  re-render on reconcile ticks after the release converges, only on
  install/upgrade/rollback — a 15-day window ties admission-webhook
  liveness to release cadence. 90 days survives realistic release
  schedules on customer clusters. Rotate-and-restart requirement is
  unchanged; the window is widened.
* Document on deployment.yaml why hashing only `tls.crt` is safe:
  the Issuer is `selfSigned`, so `ca.crt == tls.crt` and every rotation
  re-issues `tls.crt`. If a future refactor ever swaps the Issuer to a
  CA-backed one (external CA, ACME, Vault), the checksum MUST be
  extended to also cover `ca.crt` — otherwise a CA-only rotation would
  leave the self-registered MutatingWebhookConfiguration serving a stale
  caBundle until the next leaf rotation.
* Add a belt-and-suspenders helm-unittest assertion pinning that under
  `existingSecret` the admission-controller pod-template carries NEITHER
  `checksum/tls-secret` (legacy amplifier) NOR `checksum/tls-cert`
  (cert-manager runtime lookup). Templates are already structurally
  gated; this catches a future refactor that accidentally moves either
  checksum outside its `if` gate.

Regenerated patches/cert-manager-admission-tls.diff against upstream
via `make update` flow so the vendored chart edits stay in sync with
the patch file (avoids silent revert on the next update).

Verified: 16/16 helm-unittest cases pass, `helm lint` clean,
`helm template ... certManagerEnabled=true` renders `renewBefore: 2160h`,
and re-applying the regenerated patch onto a fresh upstream pull
produces a byte-identical chart tree.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-03 15:59:07 +03:00
Myasnikov Daniil 47638f8569 fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122)
Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes left the DataStore stuck Terminating (finalizer kamaji.clastix.io/TenantControlPlane), hanging the etcd helm uninstall and wedging the tenant namespace. Backports clastix/kamaji#1122 (fixes clastix/kamaji#1115) as a source patch on the pinned 26.3.6-edge build. Refs #3018.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-03 12:59:38 +05:00
scooby87 71ca2b1bae fix(mariadb): re-guard replication so single-replica passes the operator webhook (#3180)
## What this PR does

Fixes a regression where **single-replica MariaDB can no longer be
created**.

[PR #2279](https://github.com/cozystack/cozystack/pull/2279) made
`replication.enabled: true` unconditional in the MariaDB CR so the
operator would always create `<release>-primary` / `<release>-secondary`
Services (expected by the dashboard RBAC and the ApplicationDefinition).
But the vendored mariadb-operator's validating webhook
(`vmariadb-v1alpha1.kb.io`) **rejects `replication.enabled: true` when
`spec.replicas == 1`** — replication needs a primary + at least one
replica. As a result, single-replica installs never became Ready (the
HelmRelease failed at admission).

The webhook rule lives in the operator's Go code (not in the vendored
CRD — there is no CEL rule tying replication to replica count), so the
failing path is reproduced by an e2e test rather than a schema
assertion.

### Fix — re-guard replication, teach consumers about the bare service

The operator provisions a bare `<release>` Service in **both**
topologies; `-primary`/`-secondary` exist **only** when replication is
on (`replicas>1`). Instead of forcing replication for everyone, this PR
lets replication track the replica count and makes the service-name
consumers handle the single-replica bare service:

- **`mariadb.yaml`** — gate `replication` on `replicas>1`; restore
single-replica external access via `service.type: LoadBalancer`, and
keep `primaryService: LoadBalancer` for `replicas>1` only. (Effectively
reverts #2279's two hunks.)
- **`backup-cronjob.yaml`** — single-replica dumps target the bare
service (no `-secondary` exists without replication); `replicas>1` still
offloads to `-secondary`.
- **`dashboard-resourcemap.yaml`** — always grant RBAC on the bare
service, plus `-primary`/`-secondary` when `replicas>1`, matching what
the ApplicationDefinition lists.
- **`mariadb-rd`** — list the bare service in `services.include` so
single-replica installs are visible in the dashboard.
- **`hack/e2e-apps/mariadb.bats`** — new `Create single-replica MariaDB`
test that reproduces the webhook rejection and guards against
regression.

Verified: `helm template` at `replicas=1`/`2` renders the intended
CR/RBAC/backup host; `helm unittest` passes; `update-crd.sh` keeps the
RD stable; `select-e2e.sh` maps the change to the mariadb suite.

### Release note

```release-note
fix(mariadb): single-replica instances can be created again — replication is now enabled only for replicas>1 (matching the operator webhook), while the dashboard, RBAC, and backups correctly target the bare service for single-replica setups
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an end-to-end test covering MariaDB deployments with a single
replica, including validation of the main service, endpoints,
StatefulSet replica count, and metrics components.

* **Bug Fixes**
* Updated MariaDB chart behavior to correctly handle single-replica vs
multi-replica configurations for external access, replication settings,
and backup host selection.
* Refined dashboard RBAC and secret inclusion so access applies to the
main MariaDB service in addition to replica-specific services.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 22:54:14 +03:00
Ivan Okhotnikov b40703503c docs(vertical-pod-autoscaler): correct first-upgrade rollout attribution in certificate.yaml
Kvaps' approval comment flagged that certificate.yaml attributed
the migration-upgrade rollout to the checksum/tls-cert value
flipping. That is technically inaccurate: at first-upgrade render
time, \`lookup\` runs BEFORE Helm deletes the legacy Opaque
Secret, so the annotation's value at that moment is the sha of the
old Secret content, not the yet-to-be-written cert-manager one.

The rollout still happens on the first upgrade — but because the
pod-template itself carries a structural change: the pre-migration
chart did not emit \`checksum/tls-cert\` at all, and the new
chart does. That new-annotation-present is what Helm sees as a
podTemplate diff and rolls the ReplicaSet.

The annotation's VALUE only catches up on the NEXT Helm action
(when lookup re-fires and sees the cert-manager-written Secret).
That is fine on first upgrade because the structural pod-template
change already turns the ReplicaSet over, and the fresh pod re-
registers the MutatingWebhookConfiguration with the new caBundle
after cert-manager has written the new Secret. Subsequent renewal-
driven rotations (or manual \`flux reconcile hr\`) are driven by
the annotation VALUE flip, as before.

Comment rewritten to lay this out explicitly.
Patch file regenerated to include the comment change.
15/15 helm-unittest cases still pass.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 22:04:26 +03:00
Andrei Kvapil 41fd08e2ba feat(keycloak): enable UPDATE_EMAIL required action in cozy realm (#3175)
## What this PR does

The portal user-settings page lets a user change their email via a
Keycloak `kc_action=UPDATE_EMAIL` redirect. That flow only works when
the **UPDATE_EMAIL required action is enabled at the realm level** — and
on a fresh realm it is registered but **disabled by default**.

`update-email` is a supported, default-on feature since Keycloak 26.4
(keycloak/keycloak#39722), so the provider is already loaded on the 26.x
server this repo ships — **no `KC_FEATURES` change and no Keycloak
restart are needed**. The `edp-keycloak-operator` has no CRD field for
required actions, so this adds a small one-shot, idempotent
`post-install`/`post-upgrade` Job to `keycloak-configure` that enables
it via `kcadm.sh`, waiting for the operator to reconcile the realm
first.

Verified against a live Keycloak 26.5.2: the `UPDATE_EMAIL` action goes
from `enabled=false` to `enabled=true`, and re-running the Job is a
no-op.

### Release note

```release-note
feat(keycloak): enable the UPDATE_EMAIL required action in the cozy realm so users can change their email from the portal
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Keycloak now automatically enables the email update action during
install and upgrade, helping users change their email address in the
`cozy` realm.
* The configuration now includes the required Keycloak container image
for this setup task.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 21:02:18 +02:00
scooby87 6bf836db30 fix(dashboard): show LoadBalancer external IP on the app Services tab (#3172)
## What this PR does

For an application exposed with `external: true` (which creates a `type:
LoadBalancer` Service), the app detail → **Services** tab showed the
service Type and Cluster IP but not the external IP assigned by MetalLB
— so there was no way to learn the connection address from the per-app
view. The pre-1.4.0 dashboard showed this, so it read as a regression
(reported in #2789).

Root cause: `ServicesTab.tsx` queried Services with a spec-only local
type (`{ type, clusterIP, ports }`) and never read
`status.loadBalancer.ingress[]`. The data was already reachable —
`ExternalIpsPage.tsx` renders it — so this was a wiring gap in the
per-app tab, not missing data.

Changes:
- Add a `ServiceStatus` type and extend the `useK8sList` query to
`K8sResource<ServiceSpec, ServiceStatus>`.
- Add an **External IP** column: `ip ?? hostname ?? "Pending"` for
`LoadBalancer` services, `—` otherwise — mirroring `ExternalIpsPage`.
- Add a co-located regression test (`ServicesTab.test.tsx`) covering the
IP, hostname→Pending fallback, and the ClusterIP no-external case.

Verified: `pnpm typecheck` clean, `pnpm test` green (320 tests, incl. 3
new), `eslint` clean on the changed files.

Fixes #3111

### Screenshots
<img width="1220" height="572" alt="Снимок экрана 2026-07-02 в 14 12 11"
src="https://github.com/user-attachments/assets/82cd1da8-ff81-4b95-a6bf-18729b6c85c7"
/>


### Release note

```release-note
fix(dashboard): show the external LoadBalancer IP on the per-app Services tab
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Added an **External IP** column to the service details view.
* Load balancer services now show the external IP or hostname when
available.

* **Bug Fixes**
* Fixed the external IP display for load balancer services so it no
longer misses valid IPs.
* Improved fallback behavior by showing **Pending** when no external
endpoint is assigned.
* Non-load balancer services now show an em dash instead of misleading
external IP text.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 20:06:26 +03:00
Ivan Okhotnikov 73c73df9ee docs(vertical-pod-autoscaler): clean up dangling references to the retired migrator/rollout hooks
Self-review after the tls-migrator removal caught a stale comment in
certificate.yaml still promising that a pre-upgrade hook would strip
Helm/Flux ownership so cert-manager could adopt the legacy Secret in
place. That hook no longer exists and — as kvaps' review pointed out
— could not have done what it claimed anyway: \`Secret.type\` is
immutable, so cert-manager cannot rewrite an Opaque Secret in place
into a \`kubernetes.io/tls\` one.

Rewrite the certificate.yaml secretName comment to describe the
actual migration path: Helm garbage-collects the legacy Opaque
Secret from the release manifest, cert-manager immediately writes a
fresh \`kubernetes.io/tls\` Secret of the same name via this
Certificate, and the same-upgrade rollout — driven by the fresh
\`checksum/tls-cert\` annotation — brings up a pod that re-registers
\`vpa-webhook-config\` with the new caBundle before the delete/
create window on the MWC matters (given failurePolicy: Ignore).

Also refresh patches/cert-manager-admission-tls.diff to reflect the
comment edit and re-verify end-to-end that
\`rm -rf charts && helm pull && patch\` produces a byte-identical
tree to what git tracks. 15/15 helm-unittest cases still pass.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 19:00:49 +03:00
Ivan Okhotnikov 5e65a3e792 docs(vertical-pod-autoscaler): tighten cert-manager comments per review
Kvaps' review on #3178 called out three imprecisions in the
supporting comments. Fix them without touching runtime behaviour:

  1. certificate.yaml previously emitted a secretTemplate whose
     labels include \`app.kubernetes.io/managed-by: Helm\`. Helm
     does NOT own the Secret written by cert-manager — the Certificate
     resource does — and stamping that label on it misleads operators
     reading the object with \`kubectl get secret -oyaml\`. Drop
     secretTemplate; cert-manager labels its outputs unambiguously
     (\`cert-manager.io/*\`, \`controller.cert-manager.io/*\`).

  2. deployment.yaml comment on checksum/tls-cert claimed the lookup
     \"resolves at every reconcile\". Not true. \`lookup\` runs only
     on Helm actions (install / upgrade / rollback); helm-controller
     does not re-render a deployed release on interval ticks. Under
     cozystack's release cadence this is fine — any real chart bump,
     values change, or manual \`flux reconcile hr\` triggers a
     re-template within days — but at day ~350 renewal the pod
     continues serving the certificate it loaded at startup until the
     next Helm action. If no Helm action lands in the ~15 days
     between renewal and expiry the webhook silently stops mutating.
     The comment now spells that trade-off out so a future reader can
     find it, and points at a small post-renewal Job as the follow-up
     without re-introducing the ReplicaSet-churn amplifier the whole
     PR retires.

  3. values.yaml comment on certManagerEnabled did not warn that the
     switch is effectively one-way. Once cert-manager has written the
     Secret with \`type: kubernetes.io/tls\`, flipping the flag back
     to false and rolling back to the legacy Opaque path fails —
     \`Secret.type\` is immutable, so the legacy Secret template
     cannot overwrite the cert-manager-owned one in place. The doc
     now says operators must \`kubectl delete\` the Secret first if
     they need to roll back.

Also refreshed patches/cert-manager-admission-tls.diff to reflect
these comment-only edits and re-verified end-to-end that
\`make update\`-style pull + patch produces a byte-identical tree
to what git tracks now.

15/15 helm-unittest cases still pass.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 18:57:01 +03:00
Ivan Okhotnikov 039bb7e180 fix(vertical-pod-autoscaler): carry cert-manager mods as a patches/*.diff on the vendored chart
Kvaps' review on #3178 flagged that this package's \`make update\`
target does \`rm -rf charts && helm pull cowboysysop/vertical-pod-
autoscaler --version 11.1.1\` with no re-apply step. Any local edits
to the vendored subchart were therefore one \`make update\` away
from vanishing without warning — exactly what the fluxcd-operator
package pre-empts via a \`patches/*.diff\` re-apply loop after the
helm pull.

Adopt the same convention here:

  * New file \`patches/cert-manager-admission-tls.diff\` capturing
    all four modifications the cert-manager path makes to the
    vendored subchart — the new \`certificate.yaml\` template, the
    checksum/tls-secret + new checksum/tls-cert annotation gating on
    the admission-controller Deployment, the certManagerEnabled
    guard on the legacy \`tls-secret.yaml\`, and the new
    \`certManagerEnabled: true\` default in the subchart values.

  * Makefile \`update\` target now runs
    \`patch --no-backup-if-mismatch -p1 < patches/*.diff\` after the
    helm pull, so a future upstream bump (or a routine
    \`make update\` today) re-applies the cozystack modifications
    over the freshly-fetched vendored files and leaves the chart in
    the same state git tracks.

Verified end-to-end: with the on-disk chart intact I ran
  cp -r charts/vertical-pod-autoscaler /tmp/reference
  rm -rf charts
  helm pull cowboysysop/vertical-pod-autoscaler --untar --untardir charts --version 11.1.1
  patch --no-backup-if-mismatch -p1 < patches/cert-manager-admission-tls.diff
  diff -r charts/vertical-pod-autoscaler /tmp/reference
The diff is empty — \`make update\` now produces a byte-identical
tree to what git has right now, and the 15 helm-unittest cases still
pass unchanged.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 18:57:01 +03:00
Ivan Okhotnikov 0c4525afe2 fix(vertical-pod-autoscaler): drop tls-migrator hook — it was a no-op
Kvaps' review on #3178 correctly identified that the pre-upgrade
tls-migrator Job cannot do what its comments claim, for two
independent reasons:

  1. Helm decides what to delete on upgrade from the STORED release
     manifest, not from live labels on the object. The only live-
     object opt-out is the \`helm.sh/resource-policy: keep\`
     annotation — see the \`original.Difference(target)\` loop in
     helm's pkg/kube/client.go, which only checks
     \`ResourcePolicyAnno\`. Stripping \`meta.helm.sh/*\` and
     \`app.kubernetes.io/managed-by\` does not prevent the deletion;
     those markers only matter for adoption checks on resources
     entering a manifest, not on resources leaving it.

     Likewise, helm-controller has no independent label-based prune
     for Helm-managed resources — the \`helm.toolkit.fluxcd.io/*\`
     labels are tracking metadata only, not a deletion gate.

  2. Even if the Secret survived the Helm/Flux delete pass,
     cert-manager still could not adopt it in place: the legacy
     Secret has \`type: Opaque\`, cert-manager writes
     \`type: kubernetes.io/tls\`, and Secret.type is immutable in
     Kubernetes. The Certificate reconcile would fail with
     \"field is immutable\" and never become Ready.

The migration converges anyway because the Job is a no-op: Helm
deletes the old Opaque Secret, cert-manager mints a fresh one of
the right type, and the Deployment rolls in the same upgrade
(its podTemplate carries the new \`checksum/tls-cert\` annotation,
which is the sha of the runtime Secret and therefore differs the
first time cert-manager writes new content). The webhook's
\`failurePolicy: Ignore\` makes the seconds-long window benign.

Delete the migrator entirely: template file, RBAC (ServiceAccount +
Role + RoleBinding), the hardcoded clastix/kubectl image reference,
and the four helm-unittest cases pinning its hook wiring / RBAC
scope. ~200 lines removed, one less pinned image, and the
documented migration flow now matches what actually happens.

Existing helm-unittest suite: 15/15 pass (was 19 with the migrator
tests; removed 4 that referenced the retired template).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 18:57:01 +03:00
Ivan Okhotnikov 78e199a75e fix(vertical-pod-autoscaler): rotate admission-controller pods via checksum/tls-cert on Secret content, not via kubectl rollout hook
Retiring the tls-rollout Job. Every Flux HelmRelease reconcile
re-fires the post-install/post-upgrade hooks, and the hook stamped a
fresh `kubectl.kubernetes.io/restartedAt` timestamp into the
Deployment podTemplate at each firing. That mutation spawns a new
ReplicaSet on every reconcile — the exact ReplicaSet-churn pattern
this whole change was meant to end, resurrected from a different
angle. CI run 28584337959 showed 10 admission-controller
ReplicaSets accumulated in ~20 minutes with no Secret content
change; the cert-manager Certificate was Ready and stable, the pod
churn was entirely driven by the hook itself.

Replace it with a declarative `checksum/tls-cert` podAnnotation on
the admission-controller Deployment. The annotation hashes the
runtime Secret's `tls.crt` via `lookup`, so it mutates only when
the Secret content actually mutates:

  * fresh install:       Secret not yet present at first render;
                         annotation absent; pod comes up on whichever
                         Secret cert-manager writes next.
  * steady state:        Secret content stable → sha stable → no
                         rotation on any Flux reconcile.
  * cert-manager adopt:  cert-manager rewrites Secret once on the
                         first upgrade (issuer change) → sha flips
                         → pod restarts ONCE and picks up the new
                         CA + cert.
  * renewal (~350d):     cert-manager rotates the Secret → sha flips
                         → pod restarts ONCE.

`lookup` returns nil during offline `helm template --dry-run=client`
(CI, unit tests) — the annotation just does not appear in that mode,
which is fine because the pod does not need to rotate in that mode
either. Flux HelmController runs the Helm SDK against the live
apiserver, so `lookup` resolves at every reconcile there.

The tls-migrator pre-upgrade hook is retained. It performs a
one-shot ownership transfer (strip Helm/Flux labels so cert-manager
can adopt the existing Secret without a delete/create race). It
does not stamp any mutable timestamps into the Deployment, so it
does not amplify. It is idempotent (no-op on fresh install and
after the first successful migration).

helm-unittest updated:
  * dropped the four assertions that pinned the tls-rollout Job's
    hook wiring and RBAC (template no longer exists);
  * added an assertion that `checksum/tls-cert` is absent when the
    tests run offline (lookup returns nil, exactly as intended).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 18:57:01 +03:00
Ivan Okhotnikov 997224a553 fix(vertical-pod-autoscaler): tls-rollout waits on the Certificate's real name
certificate.yaml renders the Certificate with name <fullname>-tls (its
secretName matches the legacy chart-owned Secret exactly), but the
rollout hook was passing plain <fullname> to `kubectl wait` — so
every post-install/post-upgrade cycle sat 3 minutes on NotFound,
failed, and burned through the Job's backoffLimit. cozystack-platform
then aborted its HR reconcile because the VPA release's post-upgrade
hook was Failed.

Snapshot from the first CI run of #3178 confirms:
  - Certificate object: vertical-pod-autoscaler-admission-controller-tls
    (namespace cozy-vertical-pod-autoscaler)
  - Rollout Job pods: 10+ BackOff / BackoffLimitExceeded
  - HR condition: post-upgrade hooks failed
  - Admission-controller Deployment itself: 1 ReplicaSet, 1 Ready pod
    (the actual chart change works — churn is gone; only the rollout
    trigger was broken)

Fix propagates the -tls suffix through both places that reference the
Certificate:
  * The rollout container's `cert` variable used in `kubectl wait`.
  * The Role's resourceNames on cert-manager.io/certificates so RBAC
    still admits the get/watch/list against the actual object.

Regression test in vpa_test.yaml asserts rules[0].resourceNames[0]
carries the -tls suffix so this exact off-by-suffix bug cannot come
back through a future rename.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 18:57:01 +03:00
Ivan Okhotnikov 522fb534a7 test(vertical-pod-autoscaler): helm-unittest coverage for cert-manager migration paths
Locks in the four configurations the chart must support:

  1. certManagerEnabled=true (default): Certificate + Issuer render
     with the same DNS shape (`vpa-webhook.<ns>.svc`) and validity
     (365d) the legacy genSignedCert produced, so cert-manager
     adoption of an existing Secret rotates content once and stays
     stable. Legacy Secret is suppressed. The checksum/tls-secret
     podAnnotation — the ReplicaSet-churn amplifier of the legacy
     path — is absent. A future refactor that reintroduces that
     annotation on the cert-manager branch would silently regress
     to the same infinite rotation loop from a different angle;
     this test blocks that.

  2. certManagerEnabled=false: opt-out for operators without cert-
     manager. Legacy Secret must be back AND the
     checksum/tls-secret annotation must be back — either one alone
     leaves admission-controller pods failing to mount.

  3. existingSecret set: operator supplies TLS out-of-band. Chart
     mints nothing (no Certificate, no Issuer, no legacy Secret, no
     migration or rollout hook). Preserves the pre-existing chart
     contract.

  4. admissionController.enabled=false (nested vpa-for-vpa release):
     no admission-controller means no TLS Secret is needed. Every
     admission-controller template must render zero documents,
     otherwise the nested release would apply stray
     Certificate/Issuer/Job manifests into cozy-vpa-for-vpa for no
     reason.

Also locks the hook wiring for tls-migrator (pre-upgrade, weight -10)
and tls-rollout (post-install,post-upgrade, weight 10) and asserts
the RBAC scoping-by-resourceName so a future edit cannot accidentally
grant broader Secret/Deployment access than the Job requires.

All 22 tests pass locally under helm-unittest.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 18:57:01 +03:00
Ivan Okhotnikov 3f52a330ed feat(vertical-pod-autoscaler): post-upgrade Job rolls admission-controller after cert-manager writes the Secret
The admission-controller reads TLS material once at process startup
(--tls-cert-file / --tls-private-key) — it does not watch the Secret.
The MutatingWebhookConfiguration vpa-webhook-config is (re-)registered
by the pod itself via --register-webhook=true at that same startup,
so its caBundle lives only as long as the pod that wrote it.

Under the new cert-manager path, nothing in the chart triggers a
Deployment rollout when cert-manager rewrites the Secret — the old
checksum/tls-secret podAnnotation is intentionally gone (it was the
ReplicaSet-churn amplifier we removed in the first commit). Left on
its own, an in-place upgrade leaves the cluster in a mixed-generation
state: cert-manager wrote a new CA + cert (issuer changed from the
ephemeral in-chart CA to the selfSigned Issuer, so the adopted Secret
content gets regenerated), but the running admission-controller pod
still holds the old cert in memory and MWC.caBundle still points at
the old CA. That's internally consistent (old pod ↔ old MWC), but
the tenant apiserver is trusting a caBundle that no longer
corresponds to the Secret on disk — the next node reboot, next
preemption, next manual rollout, or the next renewal-driven Secret
rewrite would flip half of the chain at random.

The post-install,post-upgrade Job resolves the transition
deterministically:
  1. wait for cert-manager to make the Certificate Ready
     (3-minute budget)
  2. rollout restart the admission-controller Deployment
  3. wait for rollout to complete (3-minute budget)

When the new pod comes up, it reads the new Secret and re-registers
MWC.caBundle with the new CA — old pod, old cert, and old MWC
caBundle all leave the cluster together in a single rolling update.

Downtime analysis:
  * replicaCount=1 (chart default), strategy=RollingUpdate default →
    ≤60s window with zero admission-controller pods.
  * MWC failurePolicy is Ignore (verified on cozy-dev3), so during
    that window the apiserver's webhook call times out and pod
    admission proceeds without VPA mutation. No cluster-wide
    request rejection.

RBAC is scoped by resourceName to the single Certificate and the
single Deployment; the Job's ServiceAccount cannot inspect any other
resource in the namespace. Same kubectl image
(docker.io/clastix/kubectl:v1.32) as the tls-migrator hook so
cozystack ships one image across hook Jobs.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 18:57:01 +03:00
Ivan Okhotnikov 26d1a1bfc1 feat(vertical-pod-autoscaler): pre-upgrade Job hands Secret ownership from Helm+Flux to cert-manager
Existing clusters carry an admission-controller TLS Secret that was
rendered by the legacy in-chart genCA template. It is owned by both
Helm (via app.kubernetes.io/managed-by=Helm and meta.helm.sh/*
annotations) and Flux HelmController (via helm.toolkit.fluxcd.io/*
labels). On upgrade to a chart version that no longer renders the
Secret directly — the certManagerEnabled=true path — Helm's release
diff shows the Secret as removed from the manifest and garbage-
collects it; Flux prune does the same, independently. If cert-manager
has not yet reconciled its new Certificate at that instant, the
admission-controller pod loses its TLS mount and CrashLoops until
cert-manager catches up.

Add a pre-upgrade Job with hook-weight -10 that strips both Helm and
Flux ownership from the existing Secret before Helm computes the
release diff. cert-manager then adopts the orphaned Secret via the
new Certificate (see previous commit) and rewrites its content in
place — no delete/create window, no dangling mount.

The Job is idempotent:
  * No secret → fresh install, no-op.
  * Secret already carries controller.cert-manager.io/fao=true → the
    migration already ran, no-op.
  * Otherwise → strip ownership labels/annotations. The kubectl
    label/annotate -- form is itself idempotent.

RBAC is scoped to the single Secret by resourceName; the Job's
ServiceAccount cannot list or modify any other Secret in the namespace.
kubectl image tracks docker.io/clastix/kubectl:v1.32 to match the hook
Job in packages/system/keycloak-configure/templates/delete.yaml — one
shared kubectl image across cozystack hook Jobs.

If the Job fails (RBAC misconfig, apiserver blip), Helm aborts the
upgrade. The release stays on the previous chart version; no partial
migration state.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 18:57:01 +03:00
Ivan Okhotnikov 8541a95cb7 feat(vertical-pod-autoscaler): cert-manager TLS for admission-controller webhook
The legacy path used genCA/genSignedCert inside a Helm template. Those
functions are not idempotent across renders — every Flux HelmRelease
reconcile produces a fresh CA + cert. The deployment's
checksum/tls-secret podAnnotation is a sha256 of that ever-changing
template, so each reconcile flips the PodTemplate and materialises a
new ReplicaSet. Under load — e.g. the E2E install phase where
cert-manager / CDI / LINSTOR cascade races cause repeated VPA HR
reconciles — the churn produces 10+ admission-controller ReplicaSets
in minutes and blocks the entire cozystack-platform HR because VPA
never converges to Ready.

Replace the source of the TLS Secret with a cert-manager Certificate
plus a namespace-scoped selfSigned Issuer, gated on a new
`admissionController.tls.certManagerEnabled` value that defaults to
`true`. cert-manager only writes the Secret when the certificate is
close to expiry (renewBefore), so reconcile-driven churn stops.

  * New charts/vertical-pod-autoscaler/templates/admission-controller/
    certificate.yaml renders the Issuer and the Certificate. The
    Certificate spec exactly mirrors the legacy genSignedCert shape —
    single DNS SAN `vpa-webhook.<ns>.svc`, 365d validity — so the
    admission-controller sees the same certificate topology it always
    saw. The Secret name resolves via the existing
    `admissionController.tls.secretName` helper, which means
    cert-manager writes into the same Secret the legacy template used
    to own (`<release>-admission-controller-tls`); a separate
    pre-upgrade hook (next commit) strips Helm/Flux ownership from
    that existing Secret on upgrade so cert-manager can adopt it
    without racing Helm's garbage collector.

  * The legacy Secret template stays in the chart, wrapped in an
    additional `{{- if not .certManagerEnabled }}` guard, so
    operators who explicitly opt out keep the old behaviour.

  * The deployment's `checksum/tls-secret` podAnnotation is gated
    behind the same flag. Under cert-manager it would either be
    misleading (Secret content is not driven by that template
    anymore) or trigger a pointless rollout on every unrelated chart
    change; both worse than not emitting it.

The `existingSecret` path is untouched — operators supplying an
external TLS Secret continue to bypass both cert-manager and the
legacy Secret template.

Verified by `helm template`:
  - certManagerEnabled=true (default): Certificate+Issuer rendered,
    no legacy Secret, no checksum/tls-secret annotation.
  - certManagerEnabled=false: legacy Secret rendered, no Certificate,
    checksum/tls-secret annotation restored.
  - existingSecret=<name>: neither path renders, deployment mounts
    the external Secret.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 18:57:01 +03:00