mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 12:43:32 +09:00
d30a2cfffbebd196d58ef0d92b73cf14893506ff
1740
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d30a2cfffb |
docs+test(apps/kubernetes): align nodeGroups default docs with actual zero-worker topology + regression-guard the replicas fix
Follow-ups from the self-review of #3233: - `values.yaml` / helper docstring / regenerated README + schema + cozyrds openAPISchema + Go type doc: the previous "one ingress-nginx worker" contract was already misleading (helper default carried `minReplicas: 0` but the chart hardcoded `spec.replicas: 2`), and the replicas fix makes the docstring strictly false — an empty-nodeGroups install now provisions zero workers until either the cluster-autoscaler responds to an unschedulable Pod or the operator explicitly scales the group. Update the docstrings to describe the actual behaviour and to spell out the ingress-nginx-addon implication so operators are not surprised at addon-install time. - `tests/nodegroups_default_test.yaml`: two regression cases pinning `MachineDeployment.spec.replicas == $group.minReplicas` for `minReplicas: 0` (the render-side OIDC bats path — zero worker DataVolumes provisioned) and for `minReplicas: 2` (the kubernetes-latest / kubernetes-previous bats path — byte- identical to pre-PR behaviour). Without these a future refactor can silently re-hardcode `replicas: 2` and turn no tests red. helm-unittest suite grows from 171 to 173, both new cases pass. Ref: cozystack/cozystack#3231. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
7b65cf2afd |
feat(platform): migrate to etcd-operator v1alpha2 (#2859)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for the newer etcd snapshot workflow (BackupClass/BackupJob/RestoreJob) with in-place restore. * Updated etcd chart/operator packaging and migration flow for the v1alpha2 operator release. * **Bug Fixes** * Switched readiness/condition handling to **Available**. * Improved waits to cover all etcd member pods during restore and scenario steps. * **Documentation** * Refreshed etcd backup/restore docs and example scenarios to match the v1alpha2 APIs and parameters. * **Tests** * Updated e2e coverage to validate v1alpha2 operator runtime contracts and replaced legacy backup-block tests with a snapshot round-trip test. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
011d6d1639 |
fix FUs
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
ee12295d30 |
fix(etcd-operator): set seccompProfile RuntimeDefault on controller pod
Address review feedback from gemini-code-assist on packages/system/etcd-operator/templates/deployment.yaml: add an explicit seccompProfile (RuntimeDefault) to the controller-manager pod securityContext. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
087a8c4be9 |
chore(etcd-operator): bump to v0.5.2
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
cc5d688e15 |
feat(platform): migrate to etcd-operator v1alpha2
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
7b8a330ad4 |
docs(apps/kubernetes): update users[].email docstring — email claim, not preferred_username
Gemini flagged that the `values.yaml` @field docstring on `oidc.users[].email` still described the claim mapping as `preferred_username`, misleading operators after the claim was switched to `email` for real (see the earlier fix and its end-to-end verification on dev3). Rewrite the description to match the actual mechanic: `email` claim, requested via `--oidc-extra-scope=email` in the chart-generated kubeconfig, emitted by every conformant OIDC provider. `make generate` re-propagates the update through README.md, types.go and the cozyrds embedded schema. Note on the other finding in the same review — Gemini also flagged `audiences` as being incorrectly nested under `issuer` in the AuthenticationConfiguration Secret. That is a false positive: per the kube-apiserver v1beta1 spec (apiserver.config.k8s.io/v1beta1, `JWTAuthenticator.Issuer`), `audiences` IS a child of `issuer`, not a sibling of `jwt[i]`. The current shape is correct and verified end-to-end on dev3 (alice/bob tokens carry `aud: [tenant-test-kubernetes-test3, kubernetes]` and apiserver validates them successfully with CRB-driven RBAC). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
9c73b6ef10 |
fix(apps/kubernetes): render-fail when spec.oidc.mode collides with legacy --oidc-* extraArgs
Reviewer flagged a real customer-upgrade regression: the previous values.yaml docstring on `controlPlane.apiServer.extraArgs` documented `--oidc-issuer-url`/`--oidc-client-id` as a valid usage. An existing customer who wired OIDC by hand through that passthrough (following the doc) would see their tenant kube-apiserver BOOT-FAIL the moment they toggle `spec.oidc.mode` to `System` or `CustomConfig`: kube-apiserver refuses to boot with both `--authentication-config` (injected by the new spec.oidc) and any legacy `--oidc-*` flag — `--authentication-config cannot be used with any of --oidc-*`. Add a render-time guard: when `spec.oidc.mode != None`, scan the user-supplied `controlPlane.apiServer.extraArgs` for any flag with prefix `--oidc-` and fail with a clear message pointing at the `spec.oidc.mode: CustomConfig` migration path (they can carry the same issuer/audience inside `customConfig.config`). When `spec.oidc.mode: None` the passthrough still works — customers who manage OIDC entirely by hand keep working until they opt into the new selector explicitly. Also update the values.yaml docstring on `extraArgs` to steer OIDC users to `spec.oidc.mode` and warn against the collision. Two new helm-unittest cases: the collision fails, and mode=None with legacy flags renders normally (backward compat). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
0d70d5ff3e |
feat(system/kubernetes-rd): expose OIDC kubeconfig Secret to the dashboard
Add `kubernetes-<name>-oidc-kubeconfig` to the Kubernetes ApplicationDefinition's `spec.secrets.include[].resourceNames` so the dashboard surfaces the OIDC kubeconfig to tenant viewers alongside the existing admin kubeconfig. The Secret is produced by the post-install,post-upgrade bootstrap Job in `apps/kubernetes` when `spec.oidc.mode: System` is set. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
5950450758 |
feat(apps/kubernetes): add Kubernetes.spec.oidc selector API
Introduce a flat selector — `mode: System | CustomConfig | None` — plus a `users[]` list and a `customConfig` block for tenant-supplied AuthenticationConfiguration. Default `None` preserves today's behavior. The `System` mode trusts the platform `cozy` realm via a per-cluster public client; `CustomConfig` accepts an inline `AuthenticationConfiguration` or a Secret reference (mutually exclusive). The `users[]` list drives per-user ClusterRoleBindings inside the tenant cluster, with two roles: `admin` and `view`. Regenerates deepcopy, JSON schema, README, and the kubernetes-rd cozyrds embedded schema. Implements Phase 1 API of the per-tenant OIDC design proposal (cozystack/community#24). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
59f035ff53 |
fix(kubevirt-instancetypes): restore persistent EFI/TPM state
Reverts #3006. Dropping persistent EFI/TPM was motivated by a live-migration concern that rested on an outdated assumption — that an RWO Filesystem backend-storage PVC pins the VM to its node. KubeVirt has migrated RWO-Filesystem backend storage since v1.4 (kubevirt/kubevirt#12629): on migration it creates a fresh target state PVC and copies the small state blob, so persistent-EFI/TPM VMs on the default replicated storage live-migrate fine (verified: the VM reports LiveMigratable=True and the copy-on-target PVC is created on migration). Restore upstream's persistent firmware state for the windows.* preferences. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
a44d11d2b6 |
fix(cozystack-basics): validate legacy Ingress hostnames against the tenant apex (#3200)
## What this PR does The Gateway (`cozystack-gateway-hostname-policy`) and Route (`cozystack-route-hostname-policy`) admission policies constrain tenant hostnames to the tenant apex on the Gateway API path. Gateway API is opt-in and off by default, so in the default configuration tenant applications publish through a legacy `networking.k8s.io/Ingress` on the shared ingress-nginx, which had no equivalent policy. This adds `cozystack-ingress-hostname-policy`, a fail-closed `ValidatingAdmissionPolicy` + binding for `Ingress` in `tenant-*` namespaces — the legacy-Ingress counterpart of the existing Gateway/Route hostname policies, and a new Layer 8 in the gateway security model. A hostname on `spec.rules[].host` or `spec.tls[].hosts[]` is allowed when it is within the namespace's own apex (`namespace.cozystack.io/host`), or when it is a concrete (non-wildcard) name entirely outside the platform root apex (`_cluster.root-host`). The second case keeps the `kubernetes` app's Proxied `exposeMethod` working, since it routes a user-supplied external domain to a nested cluster. A hostname under the platform root apex but outside the namespace's own apex, a wildcard taken through the outside-root path (which could otherwise match the platform apex itself), a rule with no host (absent or empty — an unbounded catch-all), and `spec.defaultBackend` are rejected. The policy is fail-closed: a `tenant-*` namespace missing its own apex label is denied, and the policy renders only when `_cluster.root-host` is set, so it is never shipped in a state that would admit every hostname. `cozy-*` namespaces are cluster-admin-managed and not gated. This is defense-in-depth in the same family as the Gateway/Route policies — by design tenants hold only read access to `Ingress`. The change ships helm-unittest coverage (render, plus the root-host, empty-host, and wildcard guards) and behavioral admission tests against the live apiserver in `hack/e2e-apps/gateway.bats` (in-apex allow, external-domain allow, under-root-outside-apex deny, wildcard-matching-apex deny, hostless deny, empty-string-host deny, defaultBackend deny), and documents the new layer in `packages/extra/gateway/README.md`. ### Screenshots N/A — no UI change. ### Release note ```release-note fix(cozystack-basics): constrain legacy Ingress hostnames in tenant namespaces to the tenant apex ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added **fail-closed** defense-in-depth Layer 8 (`cozystack-ingress-hostname-policy`) to enforce allowed legacy Ingress rule and TLS hostnames for tenant namespaces, while blocking unsafe catch-all configurations (including missing/empty hosts and `spec.defaultBackend`) when the platform root host is configured. * **Tests** * Added end-to-end and render/deny validation for permitted vs denied hostname patterns, wildcard behavior, required namespace host labeling, and correct deny/fail-closed rendering. * **Documentation** * Updated the gateway security model to document Layer 8 behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a1c779854b |
fix(vertical-pod-autoscaler): migrate admission-controller TLS to cert-manager to stop reconcile-driven ReplicaSet churn (#3178)
## Summary
The VPA admission-controller webhook TLS Secret is currently rendered by
an in-chart `genCA` + `genSignedCert` template. Those functions are not
idempotent across `helm template` invocations — every Flux HelmRelease
reconcile produces a fresh CA + cert. A `checksum/tls-secret`
podAnnotation on the admission-controller Deployment then flips its
sha, materialising a new ReplicaSet on every reconcile.
Under load — the E2E install phase where cert-manager / CDI / LINSTOR
cascade races repeatedly refire the VPA HelmRelease — this produces
10+ admission-controller ReplicaSets in minutes and blocks the whole
cozystack-platform HR: the admission-controller pod livelocks in
startup (waits on its own election lease which its previous
ReplicaSet is still holding, hits livenessProbe timeout,
CrashLoopBackOffs), VPA never becomes Ready, PackageSources stay
InProgress, and cozystack-platform aborts at its 15-minute install
timeout.
Fix: replace the source of the TLS Secret with a namespace-scoped
cert-manager `Issuer` + `Certificate`. cert-manager only rewrites the
Secret when the certificate is close to expiry (`renewBefore`), so
reconcile-driven cert churn stops. Pod rotation is delegated to a
runtime-lookup-based `checksum/tls-cert` podAnnotation that only
changes when the Secret's `tls.crt` content actually changes — not on
every render, not on every reconcile.
## What
-
`charts/vertical-pod-autoscaler/templates/admission-controller/certificate.yaml`
(new): namespace-scoped selfSigned Issuer + Certificate. The
Certificate mirrors the legacy `genSignedCert` shape (single DNS SAN
`vpa-webhook.<ns>.svc`, 365d validity, matching Secret name), so
cert-manager on the first upgrade adopts the same Secret name the
legacy path wrote — no rename cascade in the Deployment volume mount.
Rendered only when `admissionController.tls.certManagerEnabled=true`
AND no `existingSecret` is supplied.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/tls-secret.yaml`
(modified): wrapped in an extra `certManagerEnabled=false` guard so
the legacy genCA path remains available as an explicit opt-out.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/deployment.yaml`
(modified): the legacy `checksum/tls-secret` annotation is gated
behind `certManagerEnabled=false`. Under the cert-manager path the
Deployment carries a `checksum/tls-cert` annotation computed from
`lookup`-ing the runtime Secret's `tls.crt` — see the template
comment for the full timing analysis and the day-350 renewal
trade-off it documents.
- `charts/vertical-pod-autoscaler/values.yaml` (modified): new
`admissionController.tls.certManagerEnabled` value defaulting to
`true`. Values-block header documents the switch as effectively
one-way (cert-manager writes `type: kubernetes.io/tls`;
`Secret.type` is immutable, so falling back requires `kubectl
delete` first).
- `patches/cert-manager-admission-tls.diff` (new) +
`Makefile update` extended: the four subchart edits above live as a
single reviewable diff that Makefile re-applies over the freshly
vendored chart on every `make update`, matching the fluxcd-operator
package convention. Prevents a bump from silently reverting the
fix.
- `packages/core/platform/sources/vertical-pod-autoscaler.yaml`
(modified): VPA PackageSource now depends on `cozystack.cert-manager`.
Non-circular: cert-manager depends on `cozystack.networking` +
`cozystack.gateway-api-crds`, neither reaches VPA.
- `tests/vpa_test.yaml` (modified): +11 helm-unittest cases (15 total,
4 pre-existing) covering `certManagerEnabled={true,false}`,
`existingSecret` bypass, disabled admission-controller, and
the offline-render behaviour of the `checksum/tls-cert` lookup.
## Why (evidence)
Snapshot from CI run 28587656430 (this branch, pre-workaround-review):
```
admission-controller ReplicaSets after first-install churn window:
vertical-pod-autoscaler-admission-controller-55d888d6d4
vertical-pod-autoscaler-admission-controller-56f8d4c747
vertical-pod-autoscaler-admission-controller-5775dbc9b7
vertical-pod-autoscaler-admission-controller-585b7c56c9
vertical-pod-autoscaler-admission-controller-6fc44944fb
vertical-pod-autoscaler-admission-controller-74bccf4849
vertical-pod-autoscaler-admission-controller-84df98f98b
vertical-pod-autoscaler-admission-controller-8666dffb85
vertical-pod-autoscaler-admission-controller-86b469457c
vertical-pod-autoscaler-admission-controller-9b979f9d6
→ 10 ReplicaSets, one per Flux reconcile, each with a different
genCA-rooted TLS Secret sha in checksum/tls-secret.
```
## Migration for existing clusters
On upgrade the legacy Opaque Secret disappears from the release
manifest, Helm garbage-collects it, and cert-manager immediately
writes a fresh `kubernetes.io/tls` Secret of the same name via the
new Certificate. `Secret.type` is immutable, so in-place adoption is
impossible — the delete/create pass is the correct path, and the
seconds-long window it produces is safe: the `vpa-webhook-config`
MutatingWebhookConfiguration ships with `failurePolicy: Ignore`
(verified on cozy-dev3), so Pod admissions in that window skip VPA
mutation but never get rejected — no cluster-wide disruption.
The Deployment rolls in the same upgrade because the new
`checksum/tls-cert` annotation differs from anything the pod had
before, so the admission-controller re-registers `vpa-webhook-config`
with the fresh caBundle on the same pass.
## Test plan
- [x] `helm unittest packages/system/vertical-pod-autoscaler` — 15/15
pass.
- [x] `helm template` diff between main and this branch is scoped
exactly to the cert-manager path (new Certificate + Issuer,
new `checksum/tls-cert` annotation, guarded legacy Secret, new
values key, patch file + Makefile step, PackageSource
dependsOn). No unrelated lines change.
- [x] `make update`-style simulation (rm -rf charts && helm pull &&
patch) produces a byte-identical subchart to what git tracks —
the patch stays clean across future upstream bumps until it
genuinely conflicts.
- [ ] Live migration on cozy-dev3: `helm upgrade` VPA release from
pre-fix version → observe cert-manager write the new tls Secret
→ verify `checksum/tls-cert` on the new pod's podTemplate →
verify MutatingWebhookConfiguration `vpa-webhook-config`
caBundle updated in place → confirm no ReplicaSet churn on
subsequent Flux reconciles. Blocking the draft's removal.
- [ ] Full E2E on this branch — verify only 1 admission-controller
ReplicaSet after install, VPA HR reaches Ready inside
`cozystack-platform`'s 15m install timeout, and
`kubernetes-latest.bats` / `kubernetes-previous.bats` no longer
time out on VPA-attributable resource churn.
## Review pass — kvaps' CHANGES_REQUESTED addressed
- ✅ Vendored chart mods now carried as `patches/*.diff` re-applied by
`Makefile update` (matches fluxcd-operator convention).
- ✅ Retired the tls-migrator hook — kvaps correctly showed it was
a no-op: Helm's delete decision is driven by the stored release
manifest not by live labels (`ResourcePolicyAnno` is the only opt-
out), and `Secret.type` is immutable so cert-manager could not have
adopted the legacy Opaque Secret in place anyway. ~200 lines
removed.
- ✅ PR description rewritten to match the actual diff (no more
reference to the removed `tls-rollout.yaml` post-upgrade hook, real
test count, dev3 verification honestly listed as pending).
- ✅ `checksum/tls-cert` timing comment corrected: `lookup` runs on
Helm actions only, day-350 renewal trade-off spelled out.
- ✅ `certManagerEnabled=false` documented as effectively one-way
(Secret.type immutable).
- ✅ Dropped the `secretTemplate` labels — `app.kubernetes.io/managed-
by: Helm` on a cert-manager-owned Secret was misleading.
- ✅ Banner comment indentation restored in `vpa_test.yaml`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* VPA admission-controller webhook TLS now uses cert-manager by default,
with options to disable cert-manager or supply an existing TLS secret.
* Webhook pod restarts are now tied to cert-manager Secret contents via
a content-based checksum.
* **Bug Fixes**
* Improved webhook TLS provisioning reliability when TLS readiness is
delayed.
* Corrected conditional rendering/rollout behavior to avoid mismatched
TLS ownership.
* **Tests**
* Added rendering-invariant tests covering cert-manager, legacy TLS,
existing-secret bypass, and admission-controller disabled behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
d89ad70698 |
fix(cozystack-basics): validate legacy Ingress hostnames against the tenant apex
The Gateway (cozystack-gateway-hostname-policy) and Route (cozystack-route-hostname-policy) admission policies constrain tenant hostnames on the Gateway API path. Gateway API is opt-in and off by default, so in the default configuration tenant applications publish through a legacy networking.k8s.io/Ingress on the shared ingress-nginx, which had no equivalent policy. Add a fail-closed ValidatingAdmissionPolicy + Binding for Ingress in tenant-* namespaces. A hostname on spec.rules[].host or spec.tls[].hosts[] is allowed when it is within the namespace's own apex (namespace.cozystack.io/host), or when it is a concrete (non-wildcard) name entirely outside the platform root apex (_cluster.root-host); a hostname under the platform root apex but outside the namespace's own apex, and a wildcard taken through the outside-root path (which could match the platform apex itself), are rejected. The outside-root branch keeps the kubernetes app's Proxied exposeMethod working, since it routes a user-supplied external domain to a nested cluster. A rule with no host (absent or empty, an unbounded catch-all) and spec.defaultBackend are also rejected. The policy templates the platform root apex into its CEL and renders only when _cluster.root-host is set, so it is never shipped in a state that would admit every hostname. A tenant-* namespace missing its own apex label is denied. Cover the policy with helm-unittest (render, including the root-host, empty-host, and wildcard guards) and behavioral admission tests against the live apiserver (in-apex allow, external-domain allow, own-apex-wildcard allow, external-tls allow, under-root-outside-apex deny, prefix-adjacent-under-root deny, wildcard-matching-apex deny, hostless deny, empty-string-host deny, defaultBackend deny), and document it as an additional layer in the gateway security model. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
223c0ef4fb |
fix(cert-manager): raise cainjector memory limit to unblock caBundle injection
cainjector loads all ValidatingWebhookConfigurations, APIServices and CRDs into its informer caches at startup. On a full cozystack install that working set exceeds the 128Mi limit, so the single leader-elected cainjector Pod is OOMKilled during cache population and CrashLoops before it injects any caBundle. Webhook ValidatingWebhookConfigurations then keep an empty caBundle and every admission call fails with "x509: certificate signed by unknown authority" (failurePolicy: Fail), intermittently blocking Ingress creation and stalling tenant installs. Raise the cainjector memory limit to 512Mi (matching the controller) and the request to 128Mi, and add a helm-unittest that pins the sizing so a blanket resource change or a `make update` cannot silently lower it again. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
|
10f8e2de1f |
feat(kamaji): default to 2 replicas, drop telemetry webhook (#2671)
## What this PR does
`kamaji-controller-manager` runs **9 sub-controllers per
TenantControlPlane** (bootstraptoken, coredns, konnectivity, kubeadm*,
kubelet, kubeproxy, …) inside a single Go process, **and** serves the
admission webhooks for `tenantcontrolplanes` + `datastores` from the
same pod via `kamaji-webhook-service`. With the upstream default
`replicaCount=1`, every webhook round-trip from the apiserver shares CPU
with the reconciler goroutines. On a multi-tenant cluster this pushes
the apiserver-observed p99 of `mtenantcontrolplane.kb.io` and
`vtenantcontrolplane.kb.io` from the kamaji-side `~10ms` (per
`controller_runtime_webhook_latency_seconds_count/sum`) up to `85–95ms`
— purely due to CPU contention.
Three safe defaults are changed:
- **`kamaji.replicaCount: 2`** — HA-safe baseline. `--leader-elect` is
already enabled upstream, so only one replica reconciles tenants; the
follower's CPU is fully available for webhook handlers. On a single-node
cluster both pods still schedule (just colocated).
- **`kamaji.telemetry.disabled: true`** — drops the
`--disable-telemetry` handler **and** removes the
`telemetry.kamaji.clastix.io` entry from the
`ValidatingWebhookConfiguration`. The latter requires `patches/1.diff`
to wrap the telemetry block in `{{- if not .Values.telemetry.disabled }}
… {{- end }}` inside the vendored
`controller-gen/validating-webhook.yaml`. Without the webhook entry, the
apiserver no longer round-trips to a no-op handler on every
TenantControlPlane CREATE/UPDATE/DELETE.
- **`kamaji.affinity` (soft `podAntiAffinity`)** — spread the two
replicas across nodes when possible, while keeping single-node clusters
working (`preferredDuringScheduling`, not required).
The `patches/1.diff` approach (re-applied by `make update`) mirrors the
pattern already used in `packages/system/fluxcd-operator/Makefile` and
`packages/system/ingress-nginx/Makefile`, and survives the next
`clastix/kamaji` chart bump.
## Measured impact
Hikube prod, 22 TenantControlPlanes, before/after a manual rollout of
the same configuration:
| webhook | p99 before | p99 after | Δ |
| --- | --- | --- | --- |
| `mtenantcontrolplane.kb.io` | 85 ms | 25 ms | **-71%** |
| `vtenantcontrolplane.kb.io` | 92 ms | 25 ms | **-73%** |
| `telemetry.kamaji.clastix.io` | 94 ms | _webhook removed_ | **-100%**
|
Cumulative Kamaji admission **impact** (`p99 × calls/s`) dropped from
**830 → 140 (-83%)**. The Kamaji webhooks moved from the top three of
the `apiserver_admission_webhook_admission_duration_seconds_*` impact
ranking down to positions #3-#4.
## Release note
```release-note
feat(kamaji): default to 2 replicas with soft pod anti-affinity and disable telemetry (handler + webhook entry). Reduces apiserver admission p99 for tenantcontrolplane mutating/validating webhooks by ~70% on multi-tenant clusters.
```
## Test plan
- [x] `helm template packages/system/kamaji` renders `replicas: 2`,
`--disable-telemetry`, the soft anti-affinity, and **no**
`telemetry.kamaji.clastix.io` entry in the
`ValidatingWebhookConfiguration`.
- [x] `patches/1.diff` applies cleanly against the freshly extracted
upstream chart (`patch --dry-run -p1 < patches/1.diff`).
- [x] `make update` re-applies the patch automatically.
- [x] Live deploy on a 22-TCP cluster: 3 pods Running (when overriden to
`replicaCount: 3` via cozystack-values), zero pod restarts, leader
election visible, `kamaji-webhook-service` Endpoints serving N IPs.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Telemetry can be disabled via configuration.
* Controller configured for high availability with 2 replicas.
* Pod anti-affinity added to spread replicas across nodes for better
resilience.
* **Bug Fixes**
* Validating webhook is now omitted when telemetry is disabled to avoid
unintended behavior.
* **Chores**
* Update process improved to re-apply chart patches after extraction for
reproducible updates.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2671?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
809a04a220 |
feat(kubevirt-cdi): make cloneStrategyOverride configurable (#2926)
## What this PR does - Exposes the CDI `cloneStrategyOverride` as a Helm value in `packages/system/kubevirt-cdi` instead of hardcoding it in the CR template. - Default stays `csi-clone`, so **behavior is unchanged** for everyone. - It can now be overridden via a cozystack `Package` (e.g. `cloneStrategyOverride: copy`), the same way `uploadProxyURL` already is — no fork needed. ### Why The value was hardcoded, so the only way to change the clone strategy was to fork the package. On some storage backends `csi-clone` is problematic: with LINSTOR/DRBD a CSI clone is created co-located with the source golden image and then relocated to optimal nodes asynchronously; that async relocation (`migrate-disk`) can race the post-clone resize CDI performs, occasionally leaving a volume whose DRBD device is smaller than its PVC capacity. Being able to select `snapshot` or `copy` per cluster, without maintaining a fork, is a useful escape hatch. This only adds a knob; it does not change the default or recommend a strategy. ## Release note ```release-note feat(kubevirt-cdi): make CDI cloneStrategyOverride configurable via values (default csi-clone) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * The KubeVirt CDI clone strategy is now configurable via Helm chart values, allowing selection of `csi-clone`, `snapshot`, or `copy` for DataVolume cloning. * The default remains `csi-clone` to preserve existing behavior and ensure backward compatibility. * **Documentation** * Added the `cloneStrategyOverride` Helm value description and allowed options to the chart’s configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
901bb981ae |
fix(seaweedfs): configure postgres2 connection pool for the filer (#2906)
## What this PR does - Adds `WEED_POSTGRES2_CONNECTION_MAX_IDLE=20`, `WEED_POSTGRES2_CONNECTION_MAX_OPEN=40` and `WEED_POSTGRES2_CONNECTION_MAX_LIFETIME_SECONDS=600` to the filer `extraEnvironmentVars` in `packages/system/seaweedfs/values.yaml`. The vendored seaweedfs chart only ships connection pool settings for the **mysql** store (`WEED_MYSQL_CONNECTION_MAX_*`), which cozystack does not use. The **postgres2** store, which cozystack enables, has no pool settings at all. As a result the Go `database/sql` pool runs with `SetMaxIdleConns(0)`: every filer metadata lookup opens a fresh PostgreSQL connection (TCP + TLS + SCRAM, ~300 ms each). With ~5 path-component lookups per S3 operation, this adds roughly **2 seconds of latency to every S3 request**, even on an idle cluster. ### Measurements (production cluster, 12 nodes, 2 filers, CNPG backend) Before / after applying these three variables, same test (sequential ops through the S3 gateway): | Operation | Before | After | |---|---|---| | PUT 4 KiB (p50) | 10.0 s | 0.28 s | | GET 4 KiB (p50) | 3.0 s | 0.14 s | | HEAD (p50) | 2.6 s | 0.16 s | | Bounded LIST, 100 keys (p50) | 3.8 s | 0.37 s | `pg_stat_activity` confirmed the root cause: before the fix every backend was younger than ~200 ms (permanent connection churn); after the fix connections are reused for minutes. Real-traffic p95 of `SeaweedFS_filerStore_request_seconds` dropped from 1.3–6.5 s to ~0.18 s. `max_open` is capped at 40 so that the default two filer replicas stay below the CNPG default `max_connections` of 100. ## Release note ```release-note fix(seaweedfs): configure postgres2 connection pool for the filer, removing ~2s of per-request S3 latency caused by PostgreSQL connection churn ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Optimized PostgreSQL connection pool configuration in SeaweedFS to reduce latency and improve performance during metadata operations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ec9c29948b |
[reloader] Set resources requests/limits on reloader container (#2631)
## Summary
The upstream Stakater reloader chart ships with `resources: {}`, which
triggers a `no CPU limit` conformance warning on the `reloader`
Deployment.
Override `packages/system/reloader/values.yaml` at
`reloader.reloader.deployment.resources` with the exact values from the
chart's own commented example:
- `limits` : `cpu: 100m`, `memory: 512Mi`
- `requests` : `cpu: 10m`, `memory: 128Mi`
Reloader is a light controller that watches ConfigMap / Secret changes
and triggers workload restarts — small CPU envelope with enough memory
to track many objects in clusters with lots of namespaces.
## Test plan
- [ ] `helm template . --namespace cozy-reloader` from
`packages/system/reloader/` renders a `resources` block on the
`reloader-reloader` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `reloader-reloader has no
CPU limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on the reloader Deployment to clear
the "no CPU limit" conformance warning. Values follow the upstream
chart's commented example (limits 100m/512Mi, requests 10m/128Mi).
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated resource allocation configuration to define CPU and memory
parameters for system component deployments, enhancing infrastructure
stability and resource optimization.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2631)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
f2d66579f1 |
[metrics-server] Set resources requests/limits on metrics-server container (#2630)
## Summary The upstream metrics-server chart ships with `limits` commented out (only `requests cpu 100m / memory 200Mi` are set), which triggers a `no CPU limit` conformance warning on the `metrics-server` Deployment. Override `packages/system/metrics-server/values.yaml` to keep the upstream-recommended baseline `requests` and add `limits`: - `limits` : `cpu: 500m`, `memory: 500Mi` - `requests` : `cpu: 100m`, `memory: 200Mi` Comfortable headroom on top of the [official sizing formula](https://github.com/kubernetes-sigs/metrics-server#configuration) (`100m + 1m/node`, `200Mi + 2Mi/node`), covering clusters up to ~150 nodes without needing to revisit. Larger clusters can override these values further. ## Test plan - [ ] `helm template . --namespace cozy-monitoring` from `packages/system/metrics-server/` renders a `resources` block on the `metrics-server` container of the Deployment (verified locally) - [ ] Re-run the conformance/lint tool — the `metrics-server has no CPU limit` warning should be gone - [ ] CI passes ### Release note \`\`\`release-note Set CPU/memory requests and limits on the metrics-server Deployment to clear the "no CPU limit" conformance warning. Values follow the upstream sizing recommendation (requests 100m/200Mi) with a comfortable limits cap (500m/500Mi). \`\`\` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Configured explicit resource requests and limits for the metrics-server service (CPU: 100m-500m, Memory: 200Mi-500Mi) to ensure consistent resource allocation. [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2630) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
4952909e4d |
[external-dns] Set resources requests/limits on external-dns container (#2629)
## Summary
The upstream external-dns chart ships with `resources: {}`, which
triggers a `no CPU limit` conformance warning on the `external-dns`
Deployment.
Override `packages/system/external-dns/values.yaml` to set:
- `limits` : `cpu: 100m`, `memory: 128Mi`
- `requests` : `cpu: 10m`, `memory: 64Mi`
Modest sizing for a controller that just watches Service / Ingress
objects and pushes DNS records to the provider.
## Test plan
- [ ] `helm template . --namespace cozy-external-dns` from
`packages/system/external-dns/` renders a `resources` block on the
`external-dns` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `external-dns has no CPU
limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on the external-dns Deployment to
clear the "no CPU limit" conformance warning.
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated ExternalDNS resource allocation with optimized CPU and memory
limits and requests for improved resource efficiency.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2629)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
04a05f623c |
[bucket] Add resources requests/limits to s3manager container (#2624)
## Summary The `s3manager` container in `packages/system/bucket/templates/deployment.yaml` had no `resources` block, triggering a `no CPU limit` conformance warning on the bucket UI Deployment. The bucket package is a standalone chart (no vendored upstream), so the values are set directly in the template — same pattern as the csi-resizer fix in #2614 and the existing `capi-providers-cpprovider` template. - `limits` : `cpu: 500m`, `memory: 512Mi` - `requests` : `cpu: 100m`, `memory: 128Mi` Comfortable headroom for the lightweight S3 browser UI while keeping a small baseline request. ## Test plan - [ ] `helm template . --show-only templates/deployment.yaml` from `packages/system/bucket/` renders the `resources` block on the `s3manager` container (verified locally) - [ ] Re-run the conformance/lint tool — the `s3manager has no CPU limit` warning should be gone - [ ] CI passes ### Release note \`\`\`release-note Add CPU and memory requests/limits to the s3manager container of the bucket UI Deployment to clear the "no CPU limit" conformance warning. \`\`\` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Improved deployment configuration with explicit resource allocation settings to ensure optimal service stability and resource management. [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2624) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
6762d2d95c |
[goldpinger] Set resources requests/limits on goldpinger DaemonSet (#2623)
## Summary
The upstream Bloomberg goldpinger chart ships with `resources: {}`,
which triggers a `no CPU limit` conformance warning on the
`goldpinger-daemon` container running on every node.
Override `packages/system/goldpinger/values.yaml` to set:
- `limits` : `cpu: 200m`, `memory: 256Mi`
- `requests` : `cpu: 50m`, `memory: 64Mi`
Comfortable margin for the ping-mesh fan-out on larger clusters while
keeping a small baseline request so the DaemonSet schedules everywhere.
## Test plan
- [ ] `helm template . --namespace cozy-goldpinger` from
`packages/system/goldpinger/` renders the `resources` block on the
`goldpinger-daemon` container of the DaemonSet (verified locally)
- [ ] Re-run the conformance/lint tool — the `goldpinger has no CPU
limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on the goldpinger DaemonSet to clear
the "no CPU limit" conformance warning.
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated Kubernetes resource requirements for the goldpinger component,
defining CPU limits (200m), memory limits (256Mi), CPU requests (50m),
and memory requests (64Mi).
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2623)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
8bc62cca1c |
feat(kamaji): default to 2 replicas, drop telemetry webhook
The kamaji-controller-manager runs ~9 sub-controllers per
TenantControlPlane (bootstraptoken, coredns, konnectivity, kubeadm*,
kubelet, kubeproxy, etc.) inside a single Go process, AND serves the
admission webhooks for tenantcontrolplanes/datastores via the
kamaji-webhook-service ClusterIP. With the upstream default
replicaCount=1, every webhook RT shares CPU with the reconciler
goroutines. On a multi-tenant cluster (~20 TCPs), this pushes
apiserver-observed p99 for mtenantcontrolplane.kb.io and
vtenantcontrolplane.kb.io from the kamaji-side ~10ms to 85-95ms.
Three changes, each safe on small clusters and high-impact on large ones:
* replicaCount=2 — HA-safe baseline. --leader-elect is already enabled
upstream, so only one replica reconciles tenants; the follower's CPU
is fully available for webhook handlers, cutting p99 ~70% on busy
clusters. On a single-node cluster both pods schedule normally.
* telemetry.disabled=true — drops both the --disable-telemetry handler
and the telemetry.kamaji.clastix.io admission webhook entry from the
ValidatingWebhookConfiguration. The latter requires patches/1.diff to
wrap the telemetry block in a {{ if not .Values.telemetry.disabled }}
conditional inside the vendored controller-gen file. Without the
webhook entry, the apiserver no longer issues a TCP round-trip to a
no-op handler on every TenantControlPlane CREATE/UPDATE/DELETE.
* podAntiAffinity (preferred) — spread the two replicas across nodes
when possible. preferredDuringScheduling so single-node clusters
still schedule (both pods colocated, which is acceptable).
Measured on a 22-TCP production cluster (Hikube, before/after):
mtenantcontrolplane p99 85 → 25 ms (-71%), vtenantcontrolplane p99
92 → 25 ms (-73%), telemetry webhook impact 291 → 0. Total Kamaji
admission impact -83% (830 → 140).
The patches/1.diff approach (re-applied by `make update`) mirrors the
pattern already used in fluxcd-operator and ingress-nginx and survives
the next clastix/kamaji chart bump.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
|
||
|
|
c47bb6edb0 |
feat(kubevirt-cdi): make cloneStrategyOverride configurable
The CDI cloneStrategyOverride was hardcoded to csi-clone in the CR template, so operators could only change it by forking the package. Expose it as a Helm value (default csi-clone, behavior unchanged) so it can be set via a cozystack Package override, the same way uploadProxyURL already is. This is useful where the csi-clone path is problematic on a given storage backend (e.g. LINSTOR/DRBD, where a clone is co-located with the source and relocated afterwards; the async relocation can race the post-clone resize), letting operators select snapshot or copy without maintaining a fork. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: mattia-eleuteri <mattia@hidora.io> |
||
|
|
49cbea8c73 |
fix(seaweedfs): configure postgres2 connection pool for the filer
The vendored seaweedfs chart ships connection pool settings only for the mysql store, which cozystack does not use. The postgres2 store, which is enabled, has no pool settings, so the Go sql pool keeps zero idle connections and every filer metadata lookup opens a fresh PostgreSQL connection (TCP + TLS + SCRAM, ~300ms each). With ~5 path lookups per S3 operation this adds roughly 2 seconds of latency to every S3 request, regardless of cluster load. Measured on a production cluster (12 nodes, 2 filers): PUT of a 4KiB object went from 10s (p50) to 0.28s, HEAD from 2.6s to 0.16s after setting these values; pg_stat_activity confirmed connections were previously recycled every ~200ms and are now reused. max_open is capped at 40 so that two filer replicas stay below the CNPG default max_connections of 100. Signed-off-by: mattia-eleuteri <mattia@hidora.io> |
||
|
|
adb307cbae |
[goldpinger] Set resources requests/limits on goldpinger DaemonSet
The Bloomberg goldpinger chart ships with resources: {}, which triggers
a 'no CPU limit' conformance warning on the goldpinger-daemon container
deployed on every node. Override the package values.yaml to set
limits 200m/256Mi and requests 50m/64Mi — comfortable margin for the
ping-mesh fan-out on larger clusters while keeping a small baseline
request.
Signed-off-by: Matthieu <matthieu@hidora.com>
|
||
|
|
bf54945c08 |
[reloader] Set resources requests/limits on reloader container
The upstream Stakater reloader chart ships with resources: {}, which
triggers a 'no CPU limit' conformance warning on the reloader
Deployment. Apply the exact values from the chart's own commented
example (limits 100m/512Mi, requests 10m/128Mi) — reloader is a light
controller that watches ConfigMap/Secret changes and triggers restarts,
so a small CPU envelope with enough memory to track many objects fits.
Signed-off-by: Matthieu <matthieu@hidora.com>
|
||
|
|
3c2133ce4b |
[metrics-server] Set resources requests/limits on metrics-server container
The upstream metrics-server chart ships with limits commented out (requests cpu 100m / memory 200Mi only), which triggers a 'no CPU limit' conformance warning. Override the package values.yaml to keep the upstream-recommended baseline requests and add limits 500m/500Mi — comfortable headroom on top of the official sizing formula (100m + 1m/node, 200Mi + 2Mi/node), covering clusters up to ~150 nodes without needing to revisit. Signed-off-by: Matthieu <matthieu@hidora.com> |
||
|
|
7f72e7bbe8 |
[external-dns] Set resources requests/limits on external-dns container
The upstream external-dns chart ships with resources: {}, which triggers
a 'no CPU limit' conformance warning on the external-dns Deployment.
Override the package values.yaml to set limits 100m/128Mi and requests
10m/64Mi — modest sizing for a controller that just watches Service /
Ingress objects and pushes DNS records to the provider.
Signed-off-by: Matthieu <matthieu@hidora.com>
|
||
|
|
c8b180d91a |
[bucket] Add resources requests/limits to s3manager container
The s3manager Deployment in the bucket package had no resources block, triggering a 'no CPU limit' conformance warning. Set limits 500m/512Mi and requests 100m/128Mi — comfortable headroom for the lightweight S3 browser UI. Bucket is a standalone chart (no vendored upstream), so the values are set directly in the template, matching the existing pattern in csi-resizer and capi-providers. Signed-off-by: Matthieu <matthieu@hidora.com> |
||
|
|
cd95040c58 |
[cert-manager] Set resources requests/limits on cert-manager pods
The upstream cert-manager chart ships with resources: {} for all four
workloads (controller, webhook, cainjector, startupapicheck), which
triggers 'no CPU limit' conformance warnings on every pod. Override the
package values.yaml to set sane requests and limits sized to each
component's role: controller is the most loaded, webhook and cainjector
handle short-lived admission/injection work, and startupapicheck is a
one-shot Job.
Signed-off-by: Matthieu <matthieu@hidora.com>
|
||
|
|
2ac269a04b |
docs(vertical-pod-autoscaler): fix stale ~350d in checksum/tls-cert ascii timeline
Second self-review pass caught a remaining reference to the old rotation math in the checksum/tls-cert doc block's ascii timeline (`renewal (~350d)` — from `renewBefore: 360h` when 365 - 15 = 350). With the widened `renewBefore: 2160h` rotation now lands at day ~275, matching the bullet-list narrative directly below. Update the number in place and regenerate the vendored-chart patch. No functional change; one-line doc fix. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
d6e57d67e7 |
docs(vertical-pod-autoscaler): sync deployment.yaml trade-off narrative with widened renewBefore
Self-review caught a stale day-math block in deployment.yaml that still
described the previous `renewBefore: 360h` behaviour ("cert-manager
renews at day ~350 and NO Helm action happens in the following ~15
days"). After bumping `renewBefore` to 2160h in the previous commit,
rotation now lands at day ~275 with a ~90-day window before expiry.
Update the numbers in place and regenerate the vendored-chart patch so
`make update` re-applies the corrected text instead of silently
reverting to the stale description on the next chart refresh.
No functional change — cert rotation math already lives in
`certificate.yaml`; this only fixes the parallel rationale block a
future reader would read alongside it.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
|
79ce69a3b4 |
fix(vertical-pod-autoscaler): widen renewBefore + document selfSigned checksum, add existingSecret belt-and-suspenders assertion
Address the three non-blocking follow-ups on myasnikovdaniil's approval: * renewBefore: 360h → 2160h (15d → 90d). With webhook `failurePolicy: Ignore`, an expired in-memory cert silently drops admission traffic until a Helm action rolls the ReplicaSet. helm-controller does not re-render on reconcile ticks after the release converges, only on install/upgrade/rollback — a 15-day window ties admission-webhook liveness to release cadence. 90 days survives realistic release schedules on customer clusters. Rotate-and-restart requirement is unchanged; the window is widened. * Document on deployment.yaml why hashing only `tls.crt` is safe: the Issuer is `selfSigned`, so `ca.crt == tls.crt` and every rotation re-issues `tls.crt`. If a future refactor ever swaps the Issuer to a CA-backed one (external CA, ACME, Vault), the checksum MUST be extended to also cover `ca.crt` — otherwise a CA-only rotation would leave the self-registered MutatingWebhookConfiguration serving a stale caBundle until the next leaf rotation. * Add a belt-and-suspenders helm-unittest assertion pinning that under `existingSecret` the admission-controller pod-template carries NEITHER `checksum/tls-secret` (legacy amplifier) NOR `checksum/tls-cert` (cert-manager runtime lookup). Templates are already structurally gated; this catches a future refactor that accidentally moves either checksum outside its `if` gate. Regenerated patches/cert-manager-admission-tls.diff against upstream via `make update` flow so the vendored chart edits stay in sync with the patch file (avoids silent revert on the next update). Verified: 16/16 helm-unittest cases pass, `helm lint` clean, `helm template ... certManagerEnabled=true` renders `renewBefore: 2160h`, and re-applying the regenerated patch onto a fresh upstream pull produces a byte-identical chart tree. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
47638f8569 |
fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122)
Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes left the DataStore stuck Terminating (finalizer kamaji.clastix.io/TenantControlPlane), hanging the etcd helm uninstall and wedging the tenant namespace. Backports clastix/kamaji#1122 (fixes clastix/kamaji#1115) as a source patch on the pinned 26.3.6-edge build. Refs #3018. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
71ca2b1bae |
fix(mariadb): re-guard replication so single-replica passes the operator webhook (#3180)
## What this PR does Fixes a regression where **single-replica MariaDB can no longer be created**. [PR #2279](https://github.com/cozystack/cozystack/pull/2279) made `replication.enabled: true` unconditional in the MariaDB CR so the operator would always create `<release>-primary` / `<release>-secondary` Services (expected by the dashboard RBAC and the ApplicationDefinition). But the vendored mariadb-operator's validating webhook (`vmariadb-v1alpha1.kb.io`) **rejects `replication.enabled: true` when `spec.replicas == 1`** — replication needs a primary + at least one replica. As a result, single-replica installs never became Ready (the HelmRelease failed at admission). The webhook rule lives in the operator's Go code (not in the vendored CRD — there is no CEL rule tying replication to replica count), so the failing path is reproduced by an e2e test rather than a schema assertion. ### Fix — re-guard replication, teach consumers about the bare service The operator provisions a bare `<release>` Service in **both** topologies; `-primary`/`-secondary` exist **only** when replication is on (`replicas>1`). Instead of forcing replication for everyone, this PR lets replication track the replica count and makes the service-name consumers handle the single-replica bare service: - **`mariadb.yaml`** — gate `replication` on `replicas>1`; restore single-replica external access via `service.type: LoadBalancer`, and keep `primaryService: LoadBalancer` for `replicas>1` only. (Effectively reverts #2279's two hunks.) - **`backup-cronjob.yaml`** — single-replica dumps target the bare service (no `-secondary` exists without replication); `replicas>1` still offloads to `-secondary`. - **`dashboard-resourcemap.yaml`** — always grant RBAC on the bare service, plus `-primary`/`-secondary` when `replicas>1`, matching what the ApplicationDefinition lists. - **`mariadb-rd`** — list the bare service in `services.include` so single-replica installs are visible in the dashboard. - **`hack/e2e-apps/mariadb.bats`** — new `Create single-replica MariaDB` test that reproduces the webhook rejection and guards against regression. Verified: `helm template` at `replicas=1`/`2` renders the intended CR/RBAC/backup host; `helm unittest` passes; `update-crd.sh` keeps the RD stable; `select-e2e.sh` maps the change to the mariadb suite. ### Release note ```release-note fix(mariadb): single-replica instances can be created again — replication is now enabled only for replicas>1 (matching the operator webhook), while the dashboard, RBAC, and backups correctly target the bare service for single-replica setups ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an end-to-end test covering MariaDB deployments with a single replica, including validation of the main service, endpoints, StatefulSet replica count, and metrics components. * **Bug Fixes** * Updated MariaDB chart behavior to correctly handle single-replica vs multi-replica configurations for external access, replication settings, and backup host selection. * Refined dashboard RBAC and secret inclusion so access applies to the main MariaDB service in addition to replica-specific services. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b40703503c |
docs(vertical-pod-autoscaler): correct first-upgrade rollout attribution in certificate.yaml
Kvaps' approval comment flagged that certificate.yaml attributed the migration-upgrade rollout to the checksum/tls-cert value flipping. That is technically inaccurate: at first-upgrade render time, \`lookup\` runs BEFORE Helm deletes the legacy Opaque Secret, so the annotation's value at that moment is the sha of the old Secret content, not the yet-to-be-written cert-manager one. The rollout still happens on the first upgrade — but because the pod-template itself carries a structural change: the pre-migration chart did not emit \`checksum/tls-cert\` at all, and the new chart does. That new-annotation-present is what Helm sees as a podTemplate diff and rolls the ReplicaSet. The annotation's VALUE only catches up on the NEXT Helm action (when lookup re-fires and sees the cert-manager-written Secret). That is fine on first upgrade because the structural pod-template change already turns the ReplicaSet over, and the fresh pod re- registers the MutatingWebhookConfiguration with the new caBundle after cert-manager has written the new Secret. Subsequent renewal- driven rotations (or manual \`flux reconcile hr\`) are driven by the annotation VALUE flip, as before. Comment rewritten to lay this out explicitly. Patch file regenerated to include the comment change. 15/15 helm-unittest cases still pass. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
41fd08e2ba |
feat(keycloak): enable UPDATE_EMAIL required action in cozy realm (#3175)
## What this PR does The portal user-settings page lets a user change their email via a Keycloak `kc_action=UPDATE_EMAIL` redirect. That flow only works when the **UPDATE_EMAIL required action is enabled at the realm level** — and on a fresh realm it is registered but **disabled by default**. `update-email` is a supported, default-on feature since Keycloak 26.4 (keycloak/keycloak#39722), so the provider is already loaded on the 26.x server this repo ships — **no `KC_FEATURES` change and no Keycloak restart are needed**. The `edp-keycloak-operator` has no CRD field for required actions, so this adds a small one-shot, idempotent `post-install`/`post-upgrade` Job to `keycloak-configure` that enables it via `kcadm.sh`, waiting for the operator to reconcile the realm first. Verified against a live Keycloak 26.5.2: the `UPDATE_EMAIL` action goes from `enabled=false` to `enabled=true`, and re-running the Job is a no-op. ### Release note ```release-note feat(keycloak): enable the UPDATE_EMAIL required action in the cozy realm so users can change their email from the portal ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Keycloak now automatically enables the email update action during install and upgrade, helping users change their email address in the `cozy` realm. * The configuration now includes the required Keycloak container image for this setup task. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
6bf836db30 |
fix(dashboard): show LoadBalancer external IP on the app Services tab (#3172)
## What this PR does For an application exposed with `external: true` (which creates a `type: LoadBalancer` Service), the app detail → **Services** tab showed the service Type and Cluster IP but not the external IP assigned by MetalLB — so there was no way to learn the connection address from the per-app view. The pre-1.4.0 dashboard showed this, so it read as a regression (reported in #2789). Root cause: `ServicesTab.tsx` queried Services with a spec-only local type (`{ type, clusterIP, ports }`) and never read `status.loadBalancer.ingress[]`. The data was already reachable — `ExternalIpsPage.tsx` renders it — so this was a wiring gap in the per-app tab, not missing data. Changes: - Add a `ServiceStatus` type and extend the `useK8sList` query to `K8sResource<ServiceSpec, ServiceStatus>`. - Add an **External IP** column: `ip ?? hostname ?? "Pending"` for `LoadBalancer` services, `—` otherwise — mirroring `ExternalIpsPage`. - Add a co-located regression test (`ServicesTab.test.tsx`) covering the IP, hostname→Pending fallback, and the ClusterIP no-external case. Verified: `pnpm typecheck` clean, `pnpm test` green (320 tests, incl. 3 new), `eslint` clean on the changed files. Fixes #3111 ### Screenshots <img width="1220" height="572" alt="Снимок экрана 2026-07-02 в 14 12 11" src="https://github.com/user-attachments/assets/82cd1da8-ff81-4b95-a6bf-18729b6c85c7" /> ### Release note ```release-note fix(dashboard): show the external LoadBalancer IP on the per-app Services tab ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an **External IP** column to the service details view. * Load balancer services now show the external IP or hostname when available. * **Bug Fixes** * Fixed the external IP display for load balancer services so it no longer misses valid IPs. * Improved fallback behavior by showing **Pending** when no external endpoint is assigned. * Non-load balancer services now show an em dash instead of misleading external IP text. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
73c73df9ee |
docs(vertical-pod-autoscaler): clean up dangling references to the retired migrator/rollout hooks
Self-review after the tls-migrator removal caught a stale comment in certificate.yaml still promising that a pre-upgrade hook would strip Helm/Flux ownership so cert-manager could adopt the legacy Secret in place. That hook no longer exists and — as kvaps' review pointed out — could not have done what it claimed anyway: \`Secret.type\` is immutable, so cert-manager cannot rewrite an Opaque Secret in place into a \`kubernetes.io/tls\` one. Rewrite the certificate.yaml secretName comment to describe the actual migration path: Helm garbage-collects the legacy Opaque Secret from the release manifest, cert-manager immediately writes a fresh \`kubernetes.io/tls\` Secret of the same name via this Certificate, and the same-upgrade rollout — driven by the fresh \`checksum/tls-cert\` annotation — brings up a pod that re-registers \`vpa-webhook-config\` with the new caBundle before the delete/ create window on the MWC matters (given failurePolicy: Ignore). Also refresh patches/cert-manager-admission-tls.diff to reflect the comment edit and re-verify end-to-end that \`rm -rf charts && helm pull && patch\` produces a byte-identical tree to what git tracks. 15/15 helm-unittest cases still pass. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
5e65a3e792 |
docs(vertical-pod-autoscaler): tighten cert-manager comments per review
Kvaps' review on #3178 called out three imprecisions in the supporting comments. Fix them without touching runtime behaviour: 1. certificate.yaml previously emitted a secretTemplate whose labels include \`app.kubernetes.io/managed-by: Helm\`. Helm does NOT own the Secret written by cert-manager — the Certificate resource does — and stamping that label on it misleads operators reading the object with \`kubectl get secret -oyaml\`. Drop secretTemplate; cert-manager labels its outputs unambiguously (\`cert-manager.io/*\`, \`controller.cert-manager.io/*\`). 2. deployment.yaml comment on checksum/tls-cert claimed the lookup \"resolves at every reconcile\". Not true. \`lookup\` runs only on Helm actions (install / upgrade / rollback); helm-controller does not re-render a deployed release on interval ticks. Under cozystack's release cadence this is fine — any real chart bump, values change, or manual \`flux reconcile hr\` triggers a re-template within days — but at day ~350 renewal the pod continues serving the certificate it loaded at startup until the next Helm action. If no Helm action lands in the ~15 days between renewal and expiry the webhook silently stops mutating. The comment now spells that trade-off out so a future reader can find it, and points at a small post-renewal Job as the follow-up without re-introducing the ReplicaSet-churn amplifier the whole PR retires. 3. values.yaml comment on certManagerEnabled did not warn that the switch is effectively one-way. Once cert-manager has written the Secret with \`type: kubernetes.io/tls\`, flipping the flag back to false and rolling back to the legacy Opaque path fails — \`Secret.type\` is immutable, so the legacy Secret template cannot overwrite the cert-manager-owned one in place. The doc now says operators must \`kubectl delete\` the Secret first if they need to roll back. Also refreshed patches/cert-manager-admission-tls.diff to reflect these comment-only edits and re-verified end-to-end that \`make update\`-style pull + patch produces a byte-identical tree to what git tracks now. 15/15 helm-unittest cases still pass. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
039bb7e180 |
fix(vertical-pod-autoscaler): carry cert-manager mods as a patches/*.diff on the vendored chart
Kvaps' review on #3178 flagged that this package's \`make update\` target does \`rm -rf charts && helm pull cowboysysop/vertical-pod- autoscaler --version 11.1.1\` with no re-apply step. Any local edits to the vendored subchart were therefore one \`make update\` away from vanishing without warning — exactly what the fluxcd-operator package pre-empts via a \`patches/*.diff\` re-apply loop after the helm pull. Adopt the same convention here: * New file \`patches/cert-manager-admission-tls.diff\` capturing all four modifications the cert-manager path makes to the vendored subchart — the new \`certificate.yaml\` template, the checksum/tls-secret + new checksum/tls-cert annotation gating on the admission-controller Deployment, the certManagerEnabled guard on the legacy \`tls-secret.yaml\`, and the new \`certManagerEnabled: true\` default in the subchart values. * Makefile \`update\` target now runs \`patch --no-backup-if-mismatch -p1 < patches/*.diff\` after the helm pull, so a future upstream bump (or a routine \`make update\` today) re-applies the cozystack modifications over the freshly-fetched vendored files and leaves the chart in the same state git tracks. Verified end-to-end: with the on-disk chart intact I ran cp -r charts/vertical-pod-autoscaler /tmp/reference rm -rf charts helm pull cowboysysop/vertical-pod-autoscaler --untar --untardir charts --version 11.1.1 patch --no-backup-if-mismatch -p1 < patches/cert-manager-admission-tls.diff diff -r charts/vertical-pod-autoscaler /tmp/reference The diff is empty — \`make update\` now produces a byte-identical tree to what git has right now, and the 15 helm-unittest cases still pass unchanged. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
0c4525afe2 |
fix(vertical-pod-autoscaler): drop tls-migrator hook — it was a no-op
Kvaps' review on #3178 correctly identified that the pre-upgrade tls-migrator Job cannot do what its comments claim, for two independent reasons: 1. Helm decides what to delete on upgrade from the STORED release manifest, not from live labels on the object. The only live- object opt-out is the \`helm.sh/resource-policy: keep\` annotation — see the \`original.Difference(target)\` loop in helm's pkg/kube/client.go, which only checks \`ResourcePolicyAnno\`. Stripping \`meta.helm.sh/*\` and \`app.kubernetes.io/managed-by\` does not prevent the deletion; those markers only matter for adoption checks on resources entering a manifest, not on resources leaving it. Likewise, helm-controller has no independent label-based prune for Helm-managed resources — the \`helm.toolkit.fluxcd.io/*\` labels are tracking metadata only, not a deletion gate. 2. Even if the Secret survived the Helm/Flux delete pass, cert-manager still could not adopt it in place: the legacy Secret has \`type: Opaque\`, cert-manager writes \`type: kubernetes.io/tls\`, and Secret.type is immutable in Kubernetes. The Certificate reconcile would fail with \"field is immutable\" and never become Ready. The migration converges anyway because the Job is a no-op: Helm deletes the old Opaque Secret, cert-manager mints a fresh one of the right type, and the Deployment rolls in the same upgrade (its podTemplate carries the new \`checksum/tls-cert\` annotation, which is the sha of the runtime Secret and therefore differs the first time cert-manager writes new content). The webhook's \`failurePolicy: Ignore\` makes the seconds-long window benign. Delete the migrator entirely: template file, RBAC (ServiceAccount + Role + RoleBinding), the hardcoded clastix/kubectl image reference, and the four helm-unittest cases pinning its hook wiring / RBAC scope. ~200 lines removed, one less pinned image, and the documented migration flow now matches what actually happens. Existing helm-unittest suite: 15/15 pass (was 19 with the migrator tests; removed 4 that referenced the retired template). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
78e199a75e |
fix(vertical-pod-autoscaler): rotate admission-controller pods via checksum/tls-cert on Secret content, not via kubectl rollout hook
Retiring the tls-rollout Job. Every Flux HelmRelease reconcile
re-fires the post-install/post-upgrade hooks, and the hook stamped a
fresh `kubectl.kubernetes.io/restartedAt` timestamp into the
Deployment podTemplate at each firing. That mutation spawns a new
ReplicaSet on every reconcile — the exact ReplicaSet-churn pattern
this whole change was meant to end, resurrected from a different
angle. CI run 28584337959 showed 10 admission-controller
ReplicaSets accumulated in ~20 minutes with no Secret content
change; the cert-manager Certificate was Ready and stable, the pod
churn was entirely driven by the hook itself.
Replace it with a declarative `checksum/tls-cert` podAnnotation on
the admission-controller Deployment. The annotation hashes the
runtime Secret's `tls.crt` via `lookup`, so it mutates only when
the Secret content actually mutates:
* fresh install: Secret not yet present at first render;
annotation absent; pod comes up on whichever
Secret cert-manager writes next.
* steady state: Secret content stable → sha stable → no
rotation on any Flux reconcile.
* cert-manager adopt: cert-manager rewrites Secret once on the
first upgrade (issuer change) → sha flips
→ pod restarts ONCE and picks up the new
CA + cert.
* renewal (~350d): cert-manager rotates the Secret → sha flips
→ pod restarts ONCE.
`lookup` returns nil during offline `helm template --dry-run=client`
(CI, unit tests) — the annotation just does not appear in that mode,
which is fine because the pod does not need to rotate in that mode
either. Flux HelmController runs the Helm SDK against the live
apiserver, so `lookup` resolves at every reconcile there.
The tls-migrator pre-upgrade hook is retained. It performs a
one-shot ownership transfer (strip Helm/Flux labels so cert-manager
can adopt the existing Secret without a delete/create race). It
does not stamp any mutable timestamps into the Deployment, so it
does not amplify. It is idempotent (no-op on fresh install and
after the first successful migration).
helm-unittest updated:
* dropped the four assertions that pinned the tls-rollout Job's
hook wiring and RBAC (template no longer exists);
* added an assertion that `checksum/tls-cert` is absent when the
tests run offline (lookup returns nil, exactly as intended).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
|
997224a553 |
fix(vertical-pod-autoscaler): tls-rollout waits on the Certificate's real name
certificate.yaml renders the Certificate with name <fullname>-tls (its secretName matches the legacy chart-owned Secret exactly), but the rollout hook was passing plain <fullname> to `kubectl wait` — so every post-install/post-upgrade cycle sat 3 minutes on NotFound, failed, and burned through the Job's backoffLimit. cozystack-platform then aborted its HR reconcile because the VPA release's post-upgrade hook was Failed. Snapshot from the first CI run of #3178 confirms: - Certificate object: vertical-pod-autoscaler-admission-controller-tls (namespace cozy-vertical-pod-autoscaler) - Rollout Job pods: 10+ BackOff / BackoffLimitExceeded - HR condition: post-upgrade hooks failed - Admission-controller Deployment itself: 1 ReplicaSet, 1 Ready pod (the actual chart change works — churn is gone; only the rollout trigger was broken) Fix propagates the -tls suffix through both places that reference the Certificate: * The rollout container's `cert` variable used in `kubectl wait`. * The Role's resourceNames on cert-manager.io/certificates so RBAC still admits the get/watch/list against the actual object. Regression test in vpa_test.yaml asserts rules[0].resourceNames[0] carries the -tls suffix so this exact off-by-suffix bug cannot come back through a future rename. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
522fb534a7 |
test(vertical-pod-autoscaler): helm-unittest coverage for cert-manager migration paths
Locks in the four configurations the chart must support:
1. certManagerEnabled=true (default): Certificate + Issuer render
with the same DNS shape (`vpa-webhook.<ns>.svc`) and validity
(365d) the legacy genSignedCert produced, so cert-manager
adoption of an existing Secret rotates content once and stays
stable. Legacy Secret is suppressed. The checksum/tls-secret
podAnnotation — the ReplicaSet-churn amplifier of the legacy
path — is absent. A future refactor that reintroduces that
annotation on the cert-manager branch would silently regress
to the same infinite rotation loop from a different angle;
this test blocks that.
2. certManagerEnabled=false: opt-out for operators without cert-
manager. Legacy Secret must be back AND the
checksum/tls-secret annotation must be back — either one alone
leaves admission-controller pods failing to mount.
3. existingSecret set: operator supplies TLS out-of-band. Chart
mints nothing (no Certificate, no Issuer, no legacy Secret, no
migration or rollout hook). Preserves the pre-existing chart
contract.
4. admissionController.enabled=false (nested vpa-for-vpa release):
no admission-controller means no TLS Secret is needed. Every
admission-controller template must render zero documents,
otherwise the nested release would apply stray
Certificate/Issuer/Job manifests into cozy-vpa-for-vpa for no
reason.
Also locks the hook wiring for tls-migrator (pre-upgrade, weight -10)
and tls-rollout (post-install,post-upgrade, weight 10) and asserts
the RBAC scoping-by-resourceName so a future edit cannot accidentally
grant broader Secret/Deployment access than the Job requires.
All 22 tests pass locally under helm-unittest.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
|
3f52a330ed |
feat(vertical-pod-autoscaler): post-upgrade Job rolls admission-controller after cert-manager writes the Secret
The admission-controller reads TLS material once at process startup
(--tls-cert-file / --tls-private-key) — it does not watch the Secret.
The MutatingWebhookConfiguration vpa-webhook-config is (re-)registered
by the pod itself via --register-webhook=true at that same startup,
so its caBundle lives only as long as the pod that wrote it.
Under the new cert-manager path, nothing in the chart triggers a
Deployment rollout when cert-manager rewrites the Secret — the old
checksum/tls-secret podAnnotation is intentionally gone (it was the
ReplicaSet-churn amplifier we removed in the first commit). Left on
its own, an in-place upgrade leaves the cluster in a mixed-generation
state: cert-manager wrote a new CA + cert (issuer changed from the
ephemeral in-chart CA to the selfSigned Issuer, so the adopted Secret
content gets regenerated), but the running admission-controller pod
still holds the old cert in memory and MWC.caBundle still points at
the old CA. That's internally consistent (old pod ↔ old MWC), but
the tenant apiserver is trusting a caBundle that no longer
corresponds to the Secret on disk — the next node reboot, next
preemption, next manual rollout, or the next renewal-driven Secret
rewrite would flip half of the chain at random.
The post-install,post-upgrade Job resolves the transition
deterministically:
1. wait for cert-manager to make the Certificate Ready
(3-minute budget)
2. rollout restart the admission-controller Deployment
3. wait for rollout to complete (3-minute budget)
When the new pod comes up, it reads the new Secret and re-registers
MWC.caBundle with the new CA — old pod, old cert, and old MWC
caBundle all leave the cluster together in a single rolling update.
Downtime analysis:
* replicaCount=1 (chart default), strategy=RollingUpdate default →
≤60s window with zero admission-controller pods.
* MWC failurePolicy is Ignore (verified on cozy-dev3), so during
that window the apiserver's webhook call times out and pod
admission proceeds without VPA mutation. No cluster-wide
request rejection.
RBAC is scoped by resourceName to the single Certificate and the
single Deployment; the Job's ServiceAccount cannot inspect any other
resource in the namespace. Same kubectl image
(docker.io/clastix/kubectl:v1.32) as the tls-migrator hook so
cozystack ships one image across hook Jobs.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
|
26d1a1bfc1 |
feat(vertical-pod-autoscaler): pre-upgrade Job hands Secret ownership from Helm+Flux to cert-manager
Existing clusters carry an admission-controller TLS Secret that was
rendered by the legacy in-chart genCA template. It is owned by both
Helm (via app.kubernetes.io/managed-by=Helm and meta.helm.sh/*
annotations) and Flux HelmController (via helm.toolkit.fluxcd.io/*
labels). On upgrade to a chart version that no longer renders the
Secret directly — the certManagerEnabled=true path — Helm's release
diff shows the Secret as removed from the manifest and garbage-
collects it; Flux prune does the same, independently. If cert-manager
has not yet reconciled its new Certificate at that instant, the
admission-controller pod loses its TLS mount and CrashLoops until
cert-manager catches up.
Add a pre-upgrade Job with hook-weight -10 that strips both Helm and
Flux ownership from the existing Secret before Helm computes the
release diff. cert-manager then adopts the orphaned Secret via the
new Certificate (see previous commit) and rewrites its content in
place — no delete/create window, no dangling mount.
The Job is idempotent:
* No secret → fresh install, no-op.
* Secret already carries controller.cert-manager.io/fao=true → the
migration already ran, no-op.
* Otherwise → strip ownership labels/annotations. The kubectl
label/annotate -- form is itself idempotent.
RBAC is scoped to the single Secret by resourceName; the Job's
ServiceAccount cannot list or modify any other Secret in the namespace.
kubectl image tracks docker.io/clastix/kubectl:v1.32 to match the hook
Job in packages/system/keycloak-configure/templates/delete.yaml — one
shared kubectl image across cozystack hook Jobs.
If the Job fails (RBAC misconfig, apiserver blip), Helm aborts the
upgrade. The release stays on the previous chart version; no partial
migration state.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
|
8541a95cb7 |
feat(vertical-pod-autoscaler): cert-manager TLS for admission-controller webhook
The legacy path used genCA/genSignedCert inside a Helm template. Those
functions are not idempotent across renders — every Flux HelmRelease
reconcile produces a fresh CA + cert. The deployment's
checksum/tls-secret podAnnotation is a sha256 of that ever-changing
template, so each reconcile flips the PodTemplate and materialises a
new ReplicaSet. Under load — e.g. the E2E install phase where
cert-manager / CDI / LINSTOR cascade races cause repeated VPA HR
reconciles — the churn produces 10+ admission-controller ReplicaSets
in minutes and blocks the entire cozystack-platform HR because VPA
never converges to Ready.
Replace the source of the TLS Secret with a cert-manager Certificate
plus a namespace-scoped selfSigned Issuer, gated on a new
`admissionController.tls.certManagerEnabled` value that defaults to
`true`. cert-manager only writes the Secret when the certificate is
close to expiry (renewBefore), so reconcile-driven churn stops.
* New charts/vertical-pod-autoscaler/templates/admission-controller/
certificate.yaml renders the Issuer and the Certificate. The
Certificate spec exactly mirrors the legacy genSignedCert shape —
single DNS SAN `vpa-webhook.<ns>.svc`, 365d validity — so the
admission-controller sees the same certificate topology it always
saw. The Secret name resolves via the existing
`admissionController.tls.secretName` helper, which means
cert-manager writes into the same Secret the legacy template used
to own (`<release>-admission-controller-tls`); a separate
pre-upgrade hook (next commit) strips Helm/Flux ownership from
that existing Secret on upgrade so cert-manager can adopt it
without racing Helm's garbage collector.
* The legacy Secret template stays in the chart, wrapped in an
additional `{{- if not .certManagerEnabled }}` guard, so
operators who explicitly opt out keep the old behaviour.
* The deployment's `checksum/tls-secret` podAnnotation is gated
behind the same flag. Under cert-manager it would either be
misleading (Secret content is not driven by that template
anymore) or trigger a pointless rollout on every unrelated chart
change; both worse than not emitting it.
The `existingSecret` path is untouched — operators supplying an
external TLS Secret continue to bypass both cert-manager and the
legacy Secret template.
Verified by `helm template`:
- certManagerEnabled=true (default): Certificate+Issuer rendered,
no legacy Secret, no checksum/tls-secret annotation.
- certManagerEnabled=false: legacy Secret rendered, no Certificate,
checksum/tls-secret annotation restored.
- existingSecret=<name>: neither path renders, deployment mounts
the external Secret.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|