mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
8f4c19daaeb00a2aeccf761cd96879867a4d86ce
5128
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ivan Okhotnikov
|
9dc64b9278 |
feat(platform): VPA depends on cert-manager after admission-controller TLS migration
The VPA admission-controller webhook TLS is now provisioned via a cert-manager Certificate + Issuer instead of the in-chart genCA Secret. On a fresh install the VPA package must not reconcile until cert-manager is Ready; otherwise the Certificate resource sits Pending forever, the admission-controller Secret is never written, and the pod CrashLoops on missing TLS mount — cascading up through the cozystack-platform PackageSource in exactly the way the old genCA path did, only for a different reason. Add cozystack.cert-manager to the variant's dependsOn list. Verified non-circular against packages/core/platform/sources/cert-manager.yaml (cert-manager depends on cozystack.networking + cozystack.gateway-api-crds, neither of which reaches VPA). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
3f52a330ed |
feat(vertical-pod-autoscaler): post-upgrade Job rolls admission-controller after cert-manager writes the Secret
The admission-controller reads TLS material once at process startup
(--tls-cert-file / --tls-private-key) — it does not watch the Secret.
The MutatingWebhookConfiguration vpa-webhook-config is (re-)registered
by the pod itself via --register-webhook=true at that same startup,
so its caBundle lives only as long as the pod that wrote it.
Under the new cert-manager path, nothing in the chart triggers a
Deployment rollout when cert-manager rewrites the Secret — the old
checksum/tls-secret podAnnotation is intentionally gone (it was the
ReplicaSet-churn amplifier we removed in the first commit). Left on
its own, an in-place upgrade leaves the cluster in a mixed-generation
state: cert-manager wrote a new CA + cert (issuer changed from the
ephemeral in-chart CA to the selfSigned Issuer, so the adopted Secret
content gets regenerated), but the running admission-controller pod
still holds the old cert in memory and MWC.caBundle still points at
the old CA. That's internally consistent (old pod ↔ old MWC), but
the tenant apiserver is trusting a caBundle that no longer
corresponds to the Secret on disk — the next node reboot, next
preemption, next manual rollout, or the next renewal-driven Secret
rewrite would flip half of the chain at random.
The post-install,post-upgrade Job resolves the transition
deterministically:
1. wait for cert-manager to make the Certificate Ready
(3-minute budget)
2. rollout restart the admission-controller Deployment
3. wait for rollout to complete (3-minute budget)
When the new pod comes up, it reads the new Secret and re-registers
MWC.caBundle with the new CA — old pod, old cert, and old MWC
caBundle all leave the cluster together in a single rolling update.
Downtime analysis:
* replicaCount=1 (chart default), strategy=RollingUpdate default →
≤60s window with zero admission-controller pods.
* MWC failurePolicy is Ignore (verified on cozy-dev3), so during
that window the apiserver's webhook call times out and pod
admission proceeds without VPA mutation. No cluster-wide
request rejection.
RBAC is scoped by resourceName to the single Certificate and the
single Deployment; the Job's ServiceAccount cannot inspect any other
resource in the namespace. Same kubectl image
(docker.io/clastix/kubectl:v1.32) as the tls-migrator hook so
cozystack ships one image across hook Jobs.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
Ivan Okhotnikov
|
26d1a1bfc1 |
feat(vertical-pod-autoscaler): pre-upgrade Job hands Secret ownership from Helm+Flux to cert-manager
Existing clusters carry an admission-controller TLS Secret that was
rendered by the legacy in-chart genCA template. It is owned by both
Helm (via app.kubernetes.io/managed-by=Helm and meta.helm.sh/*
annotations) and Flux HelmController (via helm.toolkit.fluxcd.io/*
labels). On upgrade to a chart version that no longer renders the
Secret directly — the certManagerEnabled=true path — Helm's release
diff shows the Secret as removed from the manifest and garbage-
collects it; Flux prune does the same, independently. If cert-manager
has not yet reconciled its new Certificate at that instant, the
admission-controller pod loses its TLS mount and CrashLoops until
cert-manager catches up.
Add a pre-upgrade Job with hook-weight -10 that strips both Helm and
Flux ownership from the existing Secret before Helm computes the
release diff. cert-manager then adopts the orphaned Secret via the
new Certificate (see previous commit) and rewrites its content in
place — no delete/create window, no dangling mount.
The Job is idempotent:
* No secret → fresh install, no-op.
* Secret already carries controller.cert-manager.io/fao=true → the
migration already ran, no-op.
* Otherwise → strip ownership labels/annotations. The kubectl
label/annotate -- form is itself idempotent.
RBAC is scoped to the single Secret by resourceName; the Job's
ServiceAccount cannot list or modify any other Secret in the namespace.
kubectl image tracks docker.io/clastix/kubectl:v1.32 to match the hook
Job in packages/system/keycloak-configure/templates/delete.yaml — one
shared kubectl image across cozystack hook Jobs.
If the Job fails (RBAC misconfig, apiserver blip), Helm aborts the
upgrade. The release stays on the previous chart version; no partial
migration state.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
Ivan Okhotnikov
|
8541a95cb7 |
feat(vertical-pod-autoscaler): cert-manager TLS for admission-controller webhook
The legacy path used genCA/genSignedCert inside a Helm template. Those
functions are not idempotent across renders — every Flux HelmRelease
reconcile produces a fresh CA + cert. The deployment's
checksum/tls-secret podAnnotation is a sha256 of that ever-changing
template, so each reconcile flips the PodTemplate and materialises a
new ReplicaSet. Under load — e.g. the E2E install phase where
cert-manager / CDI / LINSTOR cascade races cause repeated VPA HR
reconciles — the churn produces 10+ admission-controller ReplicaSets
in minutes and blocks the entire cozystack-platform HR because VPA
never converges to Ready.
Replace the source of the TLS Secret with a cert-manager Certificate
plus a namespace-scoped selfSigned Issuer, gated on a new
`admissionController.tls.certManagerEnabled` value that defaults to
`true`. cert-manager only writes the Secret when the certificate is
close to expiry (renewBefore), so reconcile-driven churn stops.
* New charts/vertical-pod-autoscaler/templates/admission-controller/
certificate.yaml renders the Issuer and the Certificate. The
Certificate spec exactly mirrors the legacy genSignedCert shape —
single DNS SAN `vpa-webhook.<ns>.svc`, 365d validity — so the
admission-controller sees the same certificate topology it always
saw. The Secret name resolves via the existing
`admissionController.tls.secretName` helper, which means
cert-manager writes into the same Secret the legacy template used
to own (`<release>-admission-controller-tls`); a separate
pre-upgrade hook (next commit) strips Helm/Flux ownership from
that existing Secret on upgrade so cert-manager can adopt it
without racing Helm's garbage collector.
* The legacy Secret template stays in the chart, wrapped in an
additional `{{- if not .certManagerEnabled }}` guard, so
operators who explicitly opt out keep the old behaviour.
* The deployment's `checksum/tls-secret` podAnnotation is gated
behind the same flag. Under cert-manager it would either be
misleading (Secret content is not driven by that template
anymore) or trigger a pointless rollout on every unrelated chart
change; both worse than not emitting it.
The `existingSecret` path is untouched — operators supplying an
external TLS Secret continue to bypass both cert-manager and the
legacy Secret template.
Verified by `helm template`:
- certManagerEnabled=true (default): Certificate+Issuer rendered,
no legacy Secret, no checksum/tls-secret annotation.
- certManagerEnabled=false: legacy Secret rendered, no Certificate,
checksum/tls-secret annotation restored.
- existingSecret=<name>: neither path renders, deployment mounts
the external Secret.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
scooby87
|
a16268152d |
fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant module delete (#3090) (#3170)
## What this PR does When the `etcd` tenant module is disabled (`etcd: false`), the etcd HelmRelease is pruned but two classes of resources leak into the tenant namespace (reported in #3090): - **Data PVCs** — the etcd-operator StatefulSet `volumeClaimTemplate` PVCs `data-etcd-<ordinal>` are never garbage-collected on `EtcdCluster` deletion. - **TLS secrets** — the five `etcd-*-tls` secrets survive the uninstall: - `etcd-ca-tls` / `etcd-client-tls` are created by this chart as `pre-install` hooks with `helm.sh/resource-policy: keep`, so Helm deliberately keeps them; - `etcd-peer-ca-tls` / `etcd-server-tls` / `etcd-peer-tls` are populated by cert-manager from `Certificate` resources and carry no owner reference back to the chart. This adds a `post-delete` cleanup hook (`packages/extra/etcd/templates/hooks/cleanup.yaml`), modeled on the accepted seaweedfs (#3092) and monitoring (#3091) sibling fixes, that: - deletes the data PVCs by name, one per replica (`data-etcd-0 … data-etcd-<replicas-1>`); - deletes the five `etcd-*-tls` secrets by name. `check-release-name` pins the release name to `etcd`, so every target name is deterministic and no sibling workload shares the `data-etcd-*` prefix in a tenant namespace — deletion by name is safe. cert-manager TLS secrets carry only the cluster-wide `controller.cert-manager.io/fao` label, so the RBAC grants `delete` by explicit `resourceNames` rather than an unsafe label sweep. The cleanup Job is hardened: non-root, read-only root fs, dropped capabilities, `seccompProfile: RuntimeDefault`, resource requests/limits, `activeDeadlineSeconds`. Deletes use `--ignore-not-found` and failures are logged but never block teardown. A Helm unit-test suite (`tests/cleanup_test.yaml`) asserts the hook metadata/RBAC/hardening and that the rendered commands target exactly the reported PVCs and secrets, including scaling the PVC list to the configured replica count. > [!NOTE] > The **stuck HelmRelease finalizer** noted in #3090 is the platform-wide sharded-helm-controller side-bug (tracked separately); it affects every `chartRef`-based tenant module identically, not just etcd. This PR fixes the PVC and secret orphans the etcd chart itself owns, matching the scope of the merged seaweedfs/monitoring fixes. > [!WARNING] > **Disabling the etcd module now permanently deletes its data PVCs and the etcd data they hold.** This is intentional and fixes the storage leak. Back up before disabling if the data matters. Fixes #3090. ### Release note ```release-note fix(etcd): garbage-collect the orphaned etcd data PVCs (data-etcd-*) and the five etcd-*-tls secrets left behind when a tenant's etcd module is disabled. WARNING: disabling etcd now permanently removes its data PVCs and the data they hold. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a Helm post-delete cleanup hook for etcd that removes per-replica etcd PVCs and the related TLS secrets when a release is deleted. * Cleanup permissions and targets now scale with replica count, including safe handling when replicas are set to zero. * **Tests** * Added Helm unit tests validating hook annotations/order, scoped RBAC, secure job hardening, and the exact deletion behavior for PVCs/secrets (including higher-replica scenarios). * **Chores** * Marked the `test` make target as phony for the etcd package. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Алексей Артамонов
|
f8e4222b7d |
test(mariadb): fix operator MariaDB CR name in single-replica failure dump
The failure-path dump queried mariadbs.k8s.mariadb.com with $name, but the
operator CR is named after the Helm release ({{ .Release.Name }} = mariadb-$name),
so the dump would hit NotFound instead of showing the rejected CR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
|
||
|
Алексей Артамонов
|
a76c5929fd |
fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry
The post-delete cleanup Jobs in the harbor and mariadb charts hardcoded `docker.io/clastix/kubectl:v1.32` — a moving tag pulled directly from Docker Hub. This broke build reproducibility (the tag can be re-pushed), was a supply-chain risk, and could not be satisfied on air-gapped / mirrored clusters, where it failed at the worst moment: during uninstall, when the post-delete hook runs and stalls release teardown. Add a canonical `cozy-lib.image` helper that prefixes a registry-relative image path with `cozy-lib.images-registry` when set and returns it unchanged (no leading "/") when empty, so standard installs resolve from the default registry and mirrored installs resolve internally. Consume it from both cleanup hooks with the image digest-pinned to the immutable multi-arch index sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c (the same digest already pinned for postgres-operator's webhook hook). Keep the pin fresh with a custom Renovate manager: the repo disables the helm-values manager (enabledManagers), and the ref is built at render time by the helper rather than being a Dockerfile/go.mod dependency, so the built-in managers cannot see it. A regex manager over the hook templates tracks clastix/kubectl by tag+digest — mirroring the existing extra/etcd manager — so the supply-chain pin does not silently age. Also harden `cozy-lib.images-registry` against a nil .Values._cluster so the helper never crashes the render — relevant precisely because the hook renders during teardown. Helm unit tests assert the rendered image (routed, digest-pinned, no leading slash) by pattern rather than an exact digest, so a Renovate bump keeps them green. Closes #3088 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Andrei Kvapil
|
caa6607f4d |
feat(keycloak): add optional KMS-encrypting database proxy (#3003)
## What this PR does Adds an optional, flag-gated **column-level PII encryption** feature to the `keycloak` system package. When `encryption.enabled=true`, a `keycloak-kms-proxy` Deployment + Service is placed on the wire between the Keycloak StatefulSet and its CloudNativePG database, and Keycloak's `KC_DB_URL_HOST`/`KC_DB_URL_PORT` are repointed at the proxy. The proxy transparently encrypts Keycloak PII columns (username, email, first/last name, `pii-*` attributes, credential secrets) before they reach PostgreSQL and decrypts them on reads; the data-encryption key is wrapped by a KMS and never lives inside the database. The proxy lives in its own repository: https://github.com/cozystack/keycloak-kms-proxy (image `ghcr.io/cozystack/keycloak-kms-proxy:v0.1.0`). **KMS backend** is selectable via `encryption.kms.backend`: - `static` (default): a self-generated 32-byte KEK kept in a Secret — works on any cluster, suitable for evaluation. - `vault-transit`: the KEK lives in HashiCorp Vault (recommended for production; supports KEK rotation). The feature is **off by default**, so existing installs are unaffected — `KC_DB_URL_HOST` keeps sourcing from the CNPG `keycloak-db-app` secret. ### Changes - `values.yaml`: new `encryption` block. - `templates/proxy.yaml` (new): gated Deployment + Service (+ self-generated KEK Secret for the static backend), hardened pod (nonroot, read-only rootfs, dropped caps, seccomp). - `templates/sts.yaml`: conditional DB-endpoint override when encryption is enabled. - `tests/encryption_test.yaml` (new): helm-unittest covering on/off and both KMS backends. `helm unittest .` → 11/11 pass. ### Follow-ups (not in this draft) - Backend TLS re-origination to the CNPG endpoint + CA validation. - Liquibase bootstrap-window handling (`KKP_LENIENT`). - Offline backfill Job for migrating existing data. - HA via a shared DEK set (currently single replica). ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(keycloak): add optional KMS-encrypting database proxy for column-level PII encryption (off by default; static KEK or Vault Transit) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional transparent encryption for Keycloak database sensitive fields * Users can configure encryption scope and select key management backend options * **Tests** * Added comprehensive test coverage validating encryption feature across multiple configuration scenarios <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Алексей АртамоновandClaude Opus 4.8
|
9472bb3412 |
fix(etcd): clamp cleanup PVC resourceNames so replicas:0 cannot widen RBAC
Address CodeRabbit review on #3170: the PVC cleanup Role generates its resourceNames from .Values.replicas. values.schema.json sets no minimum, so an explicit replicas: 0 rendered an empty resourceNames list — and an empty resourceNames in an RBAC rule matches every PVC in the namespace (a wildcard), silently undoing the least-privilege scoping the previous commit added. Clamp the count to max(1, replicas) via a shared $replicaCount used by both the Role resourceNames range and the delete command, so at least data-etcd-0 is always listed (it exists for any running cluster and the delete uses --ignore-not-found). Add a regression test asserting replicas: 0 still yields a name-scoped Role and a single-name delete rather than a namespace-wide grant. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей АртамоновandClaude Opus 4.8
|
4a06df92e7 |
fix(etcd): scope cleanup Role to the data-etcd PVC names and assert hook metadata
Address CodeRabbit review on #3170: - Tighten the cleanup Role's PVC rule from a namespace-wide get/list/delete to get/delete by explicit resourceNames (data-etcd-<ordinal>, one per replica), mirroring the least-privilege pattern already used for the TLS secrets. The Job only ever deletes those deterministic names, so list is unnecessary and a regression can no longer reach a sibling workload's PVC. - Extend the Role and RoleBinding unit tests to assert helm.sh/hook and helm.sh/hook-delete-policy (not just the weight), so the RBAC cannot regress into normal release resources, and assert the PVC resourceNames. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей АртамоновandClaude Opus 4.8
|
a92f060f63 |
fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant module delete (#3090)
When the etcd tenant module is disabled (etcd: false), the etcd-operator StatefulSet data PVCs (data-etcd-<ordinal>) are never garbage-collected, and the five etcd-*-tls secrets are left orphaned in the tenant namespace: etcd-ca-tls / etcd-client-tls are pre-install hooks this chart marks with helm.sh/resource-policy: keep, and etcd-peer-ca-tls / etcd-server-tls / etcd-peer-tls are cert-manager-issued secrets with no owner reference back to the chart. None are reclaimed by the Helm uninstall. Add a post-delete cleanup Job (modeled on the seaweedfs/monitoring sibling fixes) that deletes the data PVCs by name (one per replica) and the five TLS secrets by name. check-release-name pins the release to "etcd", so every target name is deterministic and no sibling workload shares the data-etcd-* prefix. The Job runs non-root with a read-only root fs, dropped capabilities and seccompProfile RuntimeDefault; its RBAC is scoped to PVC get/list/delete and the five secrets by resourceNames. Deletes use --ignore-not-found and failures are logged but never block teardown. The stuck-HelmRelease finalizer noted in the issue is the platform-wide sharded-helm-controller side-bug tracked separately; this change addresses the PVC and secret orphans the etcd chart owns. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
c7e8f4d975 |
fix(mariadb): re-guard replication so single-replica passes the operator webhook
PR #2279 made `replication.enabled: true` unconditional so the operator would always create -primary/-secondary services. But the mariadb-operator validating webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1, so single-replica MariaDB could no longer be created — the HelmRelease never became Ready. Re-guard replication on replicas>1 and teach the service-name consumers about the bare <release> service the operator provisions in both topologies: - mariadb.yaml: gate `replication` on replicas>1; restore single-replica external access via `service.type: LoadBalancer` and keep `primaryService` LoadBalancer for replicas>1 only. - backup-cronjob.yaml: route single-replica dumps to the bare service (no -secondary exists without replication). - dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus -primary/-secondary when replicas>1, matching the ApplicationDefinition. - mariadb-rd: list the bare service in services.include so single-replica installs are visible in the dashboard. Adds a single-replica e2e test that reproduces the webhook rejection and guards against regression. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
scooby87
|
fdbe6b299c |
fix(clickhouse)!: clean up orphaned keeper and data PVCs on delete (#3057) (#3072)
## What this PR does After deleting a `ClickHouse` app CR, data PVCs were left orphaned in the tenant namespace and consumed storage permanently: - the keeper PVC `default-chk-<release>-keeper-cluster1-0-0-0`, backing a `ClickHouseKeeperInstallation` reconciled by the clickhouse-keeper-operator; - the main ClickHouse data / log PVCs. The keeper operator does not delete data PVCs when the CR is removed (data-safety default) and the CRD exposes no `persistentVolumeClaimRetentionPolicy` equivalent, so the volumes leaked. This adds a `post-delete` cleanup hook (modeled on the existing MariaDB hook at `packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml`) that deletes every PVC carrying the release label `app.kubernetes.io/instance=<release>`. It also adds that label to the keeper `volumeClaimTemplates` in `chkeeper.yaml`: the keeper PVC previously had **no** release label (the chart set labels only on the keeper pod template, not the PVC), so the operator never stamped one and a single selector would have missed it — leaving the reported leak unfixed. ### Existing releases (label backfill migration) The altinity clickhouse-keeper-operator only stamps `volumeClaimTemplates.metadata.labels` onto PVCs it creates **after** the chart change; it never re-labels PVCs that already exist. So for releases installed before this fix, the keeper PVC stays unlabeled and the cleanup selector would still miss it. To close #3057 for existing customers too, this PR adds migration `48` (`48 -> 49`, `packages/core/platform/images/migrations/migrations/48`) that back-fills `app.kubernetes.io/instance=<release>` onto pre-existing keeper PVCs, and bumps `migrations.targetVersion` to `49`. The migration: - routes its version stamp through the shared `lib/cozystack-version.sh` helper (`stamp_cozystack_version 49`), per the `cozystack-version-stamp.bats` convention; - no-ops when the keeper CRD is absent, and tolerates a transient apiserver error on the keeper list (`|| true`) so it cannot abort a platform-wide upgrade — the backfill is best-effort (worst case is the pre-existing leak, no worse than today); - is idempotent (skips PVCs already labeled). The main ClickHouse data/log PVCs have carried `app.kubernetes.io/instance` since 2025-06-03 (well before the migration era), so they need no backfill. ### Hook failure handling The cleanup hook does **not** mask failures: `kubectl delete` failing makes the Job exit non-zero. The **Job** uses `helm.sh/hook-delete-policy: before-hook-creation` only (no `hook-succeeded`) so a failed cleanup Job and its logs survive for debugging; the **ServiceAccount/Role/RoleBinding** keep `before-hook-creation,hook-succeeded` so the RBAC artifacts are reclaimed after a successful uninstall instead of lingering in the namespace. ### Scope The hook intentionally reclaims **only PVCs** of the release. Other release state (Secrets, ConfigMaps) is owned/garbage-collected through the normal Helm uninstall path and is out of scope here. > [!WARNING] > **This is destructive — deleting a ClickHouse app now permanently deletes its data.** > The hook removes **all** PVCs of the release: the keeper PVC **and** the main > ClickHouse data and log PVCs. After deletion nothing is left behind to recover > data from. This is intentional and matches the existing MariaDB cleanup-hook > behaviour and the general cozystack model (deleting a managed app reclaims its > storage). Back up before deleting. Fixes #3057. ### Release note ```release-note fix(clickhouse): garbage-collect the keeper and data/log PVCs when a ClickHouse app is deleted, so no orphaned volumes are left behind. A migration back-fills the release label onto keeper PVCs from releases installed before this fix so they are cleaned up too. WARNING: deleting a ClickHouse app now permanently removes all of its PVCs (keeper + data + logs) and the data they hold. ``` |
||
|
Aleksei Sviridkin
|
9aa045fc30 |
feat(ci): gate sizeable API changes behind API-owner review (#3167)
## What this PR does Adds a CI gate that requires an approving review from a designated API owner (`lllamnyp` or `kvaps`) when a pull request makes a **sizeable** change to the Cozystack API surface. Sizeable means one of: 1. A new API group. 2. A new resource. 3. A breaking change to an existing resource's schema. 4. Removal of a resource or an entire API group. Detection is fully deterministic — no LLM or network calls, so it runs on every PR. ### How it works - **`cmd/api-gate` + `internal/apigate`** — a small Go tool that loads the API surface from two checkouts (merge base and PR head) and classifies the delta. Every resource's schema is normalized to a parsed OpenAPIv3Schema; the tool only unwraps it from its two container formats and reads identity (group/kind/plural) from the wrapper. | Surface | Files scanned | Schema source | | --- | --- | --- | | Apps (`apps.cozystack.io`) | `packages/system/*-rd/cozyrds/*.yaml` | `spec.application.openAPISchema` | | Typed groups (`cozystack.io`, `backups.*`, `network.*`, `gateway.*`) | every `*.cozystack.io` CRD discovered under a `crds/` or `definitions/` directory in `packages/` and `internal/` | served `spec.versions[].schema.openAPIV3Schema` | | Static Go-backed (`core.*`, `sdn.*`) | `pkg/apiserver/apiserver.go` storage registrations | none — new/removed group / resource only | CRDs are found by location + content rather than a fixed file list, so a first-party CRD that moves (e.g. into a chart's `crds/` dir) stays covered. Only `served` CRD versions count as live API surface, so the standard deprecation step (`served: true → false`) registers as a removed version. An empty snapshot is treated as an error, never a silent pass. - **Breaking classification is semantic, not textual.** Additive changes (new optional field, new enum value, widened type, relaxed or removed constraint, description/default edits) pass. Flagged as breaking: removed fields, type narrowing (including adding a type to a previously-unconstrained field), enum-value removal, newly-required fields, added/tightened `pattern`/min/max/length constraints, `additionalProperties` restricted to `false`, added `x-kubernetes-validations` (CEL) rules, and removed served versions. - **`.github/workflows/api-review-gate.yaml`** builds the tool, diffs the PR head against its true **merge base**, and — when the change is sizeable — requires an `APPROVED` review from an API owner. Only each reviewer's *latest* review *on the current head commit* counts, so a stale or superseded approval no longer satisfies the gate. It re-runs on `pull_request_review` so an approval flips the check without a new push. The token is scoped to the approval step alone (`persist-credentials: false` on checkout) so PR-controlled build steps can't read it. The allowlist lives in one place (`API_OWNERS`). The job runs on every PR (no path filter) so it can safely be marked a required check. ### Follow-up needed after merge Mark **"Require API owner review for sizeable API changes"** as a required status check in `main` branch protection for the gate to block merges. ### Tests Table-driven unit tests cover the schema-diff rules, the classifier (new group / new resource / breaking, including the double-count and pure-removal edge cases), all three parsers, and an on-disk end-to-end snapshot test. Verified against the real repo tree: identical trees pass; injected new-group/new-resource/breaking changes are all caught; an additive-only edit passes. ### Release note ```release-note feat(ci): require a review from an API owner (lllamnyp or kvaps) when a PR introduces a new API group, a new resource, or a breaking change to an existing API schema ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an API review gate workflow that flags “sizeable” API changes and requires at least one approving API-owner review. * Introduced an API-gate CLI to compare API snapshots, classify changes (new/removed resources and groups, plus breaking schema updates), and generate readable reports. * Enhanced snapshot parsing and deterministic reporting across multiple input sources. * **Tests** * Added comprehensive unit tests covering parsing, snapshot loading, schema diff detection, and classification/report output behavior. * **Chores** * Updated API gate ownership rules for the new workflow and related tooling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6ac13402a0 |
chore(packages/system/linstor): bump linstor-server to 1.33.3, linstor-csi to v1.11.2
Rebuilds the cozystack piraeus-server image off linstor-server v1.33.3 (latest 1.33 patch) and the linstor-csi image off v1.11.2, both multi-arch, and pins the new digests in values.yaml. Picks up the published security fixes plus the upstream 1.33.3 DRBD fixes (clone status check, shrink trigger). linstor-csi v1.11.2 go.mod requires Go >= 1.26, which the csi builder image (golang:1.26) already satisfies. All five piraeus-server patches re-apply cleanly against v1.33.3 in the Dockerfile glob order, and both linstor-csi patches re-apply against v1.11.2. Verified on a dev cluster: a rolling restart of all satellites/controller/csi to 1.33.3 / v1.11.2 kept every node Online with zero faulty resources; a replicated PVC provisioned, mkfs'd, and reached UpToDate across 3 nodes with no DRBD "device held open" abort (the secondary-after-mkfs patch handled the path). The csi v1.11.2 "no RWX without DRBD layer" change is a no-op for cozystack, which always uses the DRBD layer. Adaptations: - Makefile: LINSTOR_VERSION 1.33.2 -> 1.33.3, LINSTOR_CSI_VERSION v1.10.6 -> v1.11.2; add test target - images/linstor-csi/Dockerfile: ARG VERSION v1.10.6 -> v1.11.2 - values.yaml: pin both rebuilt multi-arch image digests - tests/linstor_test.yaml: new helm-unittest image pins Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9e89ae84ac |
fix(kubernetes): default worker disk storageClass to the application storageClass (#3129)
## What this PR does
Worker node VMs in a tenant Kubernetes cluster set `evictionStrategy:
External`, so they live-migrate. Live migration requires the worker
system DataVolume to be ReadWriteMany, and linstor-csi only grants RWX
on a DRBD-backed StorageClass. When a node group leaves `storageClass`
empty, the worker DataVolume template previously omitted
`storageClassName` and fell through to the management cluster's default
StorageClass — which is operator-configured and not guaranteed to be
DRBD-backed. On any cluster whose default StorageClass is not a
replicated/DRBD class, the worker disk lands on a non-RWX class and live
migration breaks.
This changes the empty-`storageClass` fallback from the management
cluster default to the application-level `storageClass` (default
`replicated`, a DRBD/LINSTOR class): `{{- with .group.storageClass |
default $.Values.storageClass }}`. A per-node-group `storageClass` still
wins when set; when both the node group and the application
`storageClass` are empty, `storageClassName` is omitted so the cluster
default still applies (explicit opt-out preserved).
Doc and codegen surfaces (`types.go`, `values.yaml`,
`values.schema.json`, `README.md`, and the generated `kubernetes-rd`
ResourceDefinition) are regenerated in lockstep with the new field
description. Chart unit tests cover the full contract: fallback to the
application class, a custom application class, per-node-group override
winning, and the both-empty omit path.
**Upgrade impact — one-time worker-pool roll.** The
`KubevirtMachineTemplate` name embeds a hash of the rendered worker
template body, and the MachineDeployment references that name. On the
shipped default (`md0` leaves `storageClass` empty) the body changes
from "no `storageClassName`" to `storageClassName: replicated`, so on
upgrade the template is renamed and Cluster API performs a one-time
rolling replacement of the worker pool — for every existing tenant
cluster, including those whose default StorageClass already was
`replicated` (the roll is driven by the body hash, not by whether the
effective class differs). This is the same class of roll any
worker-template change triggers and is unavoidable with this approach.
Two test pins document and guard the hash so future body changes are
deliberate.
### Release note
```release-note
fix(kubernetes): worker node disks now fall back to the application-level storageClass (a replicated/DRBD class) instead of the management-cluster default when a node group leaves storageClass empty, so live migration of worker VMs gets the ReadWriteMany volume it requires. Upgrading rolls each tenant cluster's worker pool once.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved worker disk `storageClassName` fallback for empty
per-node-group values: it now reliably uses the app-level
replicated/DRBD StorageClass (rather than relying on the
management-cluster default).
* Ensured KubeVirt data volume templates and rendering behavior are
consistent across single- and multi-node-group setups.
* **Documentation**
* Refined `storageClass`/RWX/DRBD guidance across configuration docs and
schema descriptions, including the intended non-immutable behavior
(optional/undefaulted).
* **Tests**
* Updated template test expectations to cover the new fallback vs
omission cases, including naming assertions for rendered templates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
83bd83062e |
feat(keycloak): enable UPDATE_EMAIL required action in cozy realm
The portal user-settings page changes a user's email via a Keycloak kc_action=UPDATE_EMAIL redirect, which only works when the UPDATE_EMAIL required action is enabled at the realm level. On a fresh realm the action is registered (update-email is a supported, default-on feature since Keycloak 26.4) but disabled by default. The edp-keycloak-operator has no CRD field for required actions, so add a one-shot, idempotent post-install/post-upgrade Job to keycloak-configure that enables it via kcadm.sh, waiting for the operator to reconcile the realm first. Assisted-By: Claude Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Алексей Артамонов
|
bf75d32a1a |
fix(dashboard): show LoadBalancer external IP on the app Services tab
The per-app Services tab queried Services with a spec-only type, so it never read status.loadBalancer.ingress and showed only the cluster-internal IP for external: true (type: LoadBalancer) services. This regressed the pre-1.4.0 behaviour (cozystack/cozystack#2789). Add a ServiceStatus type to the useK8sList query and surface an "External IP" column (ip / hostname / Pending for LoadBalancer, — otherwise), mirroring what ExternalIpsPage already does. Fixes #3111 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
7faa32508d |
fix(clickhouse): reclaim cleanup hook RBAC on successful uninstall
Address review [MINOR] on #3072: keep the cleanup Job on before-hook-creation (so a failed Job and its logs survive for post-mortem), but restore before-hook-creation,hook-succeeded on the ServiceAccount/Role/RoleBinding so those artifacts do not linger in the namespace after a successful uninstall. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
2e65ce108b |
fix(clickhouse): guard migration when keeper CRD absent and quote hook scalars
Address CodeRabbit review on #3072: - Migration 45 now skips the backfill when the clickhousekeeperinstallations CRD is not installed. The migration runs platform-wide via targetVersion, so on clusters that never deployed ClickHouse the kubectl get would otherwise fail under set -euo pipefail and abort the whole upgrade. - Quote the templated name/label/serviceAccountName scalars in the cleanup hook so the source parses under YAMLlint; rendered output is unchanged. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
a2ae1621af |
fix(clickhouse): surface cleanup hook failures and backfill legacy keeper PVC labels
Address review on #3072: - Cleanup hook no longer masks kubectl delete failures: the command now fails (exit 1) when PVC deletion fails instead of swallowing the error with || echo. The hook-delete-policy drops hook-succeeded (keeps only before-hook-creation) so a failed cleanup Job survives for inspection. - Add migration 45 (45 -> 46) that backfills app.kubernetes.io/instance onto keeper PVCs created before this change; the altinity keeper operator never re-stamps volumeClaimTemplate labels onto existing PVCs, so without this the cleanup selector misses pre-upgrade keeper PVCs and #3057 persists for existing releases. Bump migrations.targetVersion to 46. - Unify helm.sh/hook-weight key quoting across the four hook manifests. - Extend the unittest to assert the new hook-delete-policy and exit-on-failure. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
2c8f6b8ca1 |
fix(clickhouse)!: clean up orphaned keeper and data PVCs on delete (#3057)
After deleting a ClickHouse app CR, the keeper PVC and the main data/log PVCs were left orphaned in the tenant namespace, consuming storage permanently. The clickhouse-keeper-operator does not delete data PVCs on CR removal (data-safety default) and the CRD exposes no persistentVolumeClaimRetentionPolicy equivalent. Add a post-delete cleanup hook (modeled on the MariaDB hook) that deletes every PVC carrying the release label app.kubernetes.io/instance=<release>. Also stamp that label on the keeper volumeClaimTemplates in chkeeper.yaml: the keeper PVC previously had no release label (the chart labelled only the keeper pod template, not the PVC), so a single selector would have missed the very PVC the issue reports. BREAKING CHANGE: deleting a ClickHouse app now permanently removes all of its PVCs (keeper + data + logs) and the data they hold. Nothing is left behind to recover from. This matches the existing MariaDB cleanup-hook behaviour and the general cozystack model where deleting a managed app reclaims its storage. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
4783adddf7 |
fix(seaweedfs)!: clean up orphaned data PVCs and cert/db secrets on tenant module delete (#3092) (#3095)
## What this PR does
When the `seaweedfs` tenant module is disabled, the SeaweedFS
StatefulSet data PVCs (`data1-<release>-system-volume-*`) and a set of
certificate/credential secrets were left orphaned in the tenant
namespace.
This adds a `post-delete` cleanup hook
(`packages/extra/seaweedfs/templates/hooks/cleanup.yaml`) that:
- deletes the volume PVCs by their release-scoped labels
(`app.kubernetes.io/name=seaweedfs,
app.kubernetes.io/instance=<release>-system,
app.kubernetes.io/component=volume`);
- deletes the orphaned secrets by name (`<release>-s3-ingress-tls`,
`<release>-system-{admin,ca,client,filer,master,volume,worker}-cert`,
`<release>-system-db-secret`) — cert-manager TLS secrets carry only the
cluster-wide `controller.cert-manager.io/fao=true` label, so they are
removed by name rather than by an unsafe label sweep.
All names are templated from `.Release.Name` (the chart renders the
StatefulSet/cert HelmRelease as `<release>-system`). CNPG PostgreSQL
PVCs (`<release>-db-*`) are intentionally **not** targeted — CNPG cleans
those itself.
The cleanup Job is hardened: non-root, read-only root fs, dropped
capabilities, `seccompProfile: RuntimeDefault`, resource
requests/limits; RBAC scoped to PVC get/list/delete and the secrets by
`resourceNames`. Failures are logged but never block teardown.
> [!WARNING]
> **Disabling the seaweedfs module now permanently deletes its volume
data PVCs and the object data they hold.** This is intentional and fixes
the storage leak. Back up before disabling if the data matters.
Fixes #3092.
### Release note
```release-note
fix(seaweedfs): garbage-collect the orphaned SeaweedFS volume PVCs and the certificate/db secrets left behind when a tenant's seaweedfs module is disabled. WARNING: disabling seaweedfs now permanently removes its volume PVCs and the data they hold.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a Helm post-delete cleanup hook for SeaweedFS to remove orphaned
PersistentVolumeClaims and related certificate/database Secrets.
* **Bug Fixes**
* Cleanup deletions now ignore missing resources to reduce hook
failures.
* **Tests**
* Added a Helm/Kubernetes test suite validating the cleanup hook
behavior when the module is disabled.
* **Chores**
* Marked the SeaweedFS Makefile `test` target as phony.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Алексей Артамонов
|
fff2e7716f |
fix(seaweedfs)!: clean up orphaned data PVCs and cert/db secrets on tenant module delete (#3092)
When the seaweedfs tenant module is disabled, the StatefulSet volume PVCs and a set of certificate/credential secrets were left orphaned. Add a post-delete cleanup hook that deletes the volume PVCs by their release-scoped labels and the secrets by name (all templated from .Release.Name). CNPG PVCs are not targeted (CNPG cleans them). The cleanup Job is hardened (non-root, read-only root fs, dropped caps, resource limits); RBAC scoped via resourceNames; failures are logged, never blocking teardown. The PVC selector matches on name + instance only, without an app.kubernetes.io/component clause, so volume PVCs from custom storage pools and MultiZone topologies (component volume-<pool>/volume-<zone>[-<pool>]) are cleaned up too, not just the default pool. Both deletes use --wait=false and the Job has activeDeadlineSeconds so a finalizer-stuck resource cannot hang the post-delete hook. NOTE: disabling seaweedfs now permanently removes its volume PVCs and the data they hold. BREAKING CHANGE: disabling the seaweedfs tenant module now permanently deletes its volume PVCs and the object data they hold. Back up before disabling if the data matters. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
Timofei Larkin
|
dc615664a3 |
fix(ci): flag constraints added to open maps/arrays
childSchemas now recurses into items/additionalProperties whenever head defines a schema, diffing against an empty base when base had none. This catches constraining a previously-unconstrained array element or map value (base absent/true -> schema), while still treating additionalProperties:false -> schema and schema removal as safe relaxations. Also tighten the first-party group check to cozystack.io and its dot-delimited subdomains so a lookalike like fakecozystack.io is not gated. Both covered by tests. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Timofei Larkin
|
f3e55d2be4 |
fix(ci): harden API gate against tampering and schema gaps
Scope the gate's own files (cmd/api-gate, internal/apigate, the workflow) to the API owners in CODEOWNERS, and compile the detector from the trusted base checkout when present (head fallback only during bootstrap), so a PR cannot neuter the gate by editing its own detector code. Detect two more breaking vectors: removing nullable:true (rejects null) and adding a oneOf/allOf/not composition constraint. Both covered by regression tests. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Timofei Larkin
|
dec840e938 |
fix(ci): address review of the API gate
Fix schema-diff correctness: guard nil head on removed child schemas, flag adding a type to a previously-unconstrained field, stop reporting a head becoming unconstrained as narrowing, detect additionalProperties:false, and only recurse items/additionalProperties when both sides carry a schema. Gate resource and group removal as new categories. Filter CRD versions to served ones so the deprecation flow surfaces as a removed version, and flag added x-kubernetes-validations (CEL) rules. Discover first-party cozystack.io CRDs by walking crds/ and definitions/ dirs so a relocated CRD (e.g. cozystack-scheduler) is covered, and error on an empty snapshot. Surface unmapped apiserver storage prefixes and allow hyphens in plurals; drop Markdown from text-mode reports. Workflow: diff against the true merge base, count only each reviewer's latest review on the current head commit, and scope the token to the approval step with persist-credentials:false. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Andrei KvapilandClaude
|
5d36e13d06 |
refactor(keycloak): drop the generic db.* connection override
Remove the db.host/db.port/db.urlProperties override and the extraVolumes/extraVolumeMounts plumbing: they are no longer needed now that the database endpoint is handled natively. Keycloak's DB host/port come straight from the keycloak-db-app secret again, except when encryption is enabled, where the keycloak-kms-proxy owns the repointing. Theme and admin-console hostname handling are unaffected. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei KvapilandClaude
|
d459ad800a |
test(keycloak): cover encryption misconfiguration failures
Add render-time negative-path coverage for the encryption proxy: an unsupported kms.backend and a vault-transit backend missing address, keyName, or tokenSecretName must all fail the template render rather than silently producing a broken Deployment. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei KvapilandClaude
|
956759388f |
docs(keycloak): document encryption operational caveats and cap proxy memory
Address review feedback on the encryption values block: - Spell out the operational caveats an operator must understand before flipping encryption on: the single-replica proxy turns HA Keycloak into a single point of failure; existing data is not migrated automatically so enabling (or later disabling) on a populated database breaks logins until the offline backfill runs; and a lost static KEK is permanent data loss, so the KEK Secret must be backed up. - Add a memory limit to the proxy resources per the repository style guide. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei KvapilandClaude
|
e97c918829 |
fix(keycloak): fail fast on invalid KMS backend and harden proxy template
Address review feedback on the encryption proxy template: - Validate encryption.kms.backend at render time and fail with a clear message on any value other than static/vault-transit. Both the KEK Secret and the env wiring now key off the same validated $backend, so a typo can no longer render a Deployment that mounts a KEK Secret that was never created (which would silently break the whole IdP). - Guard the static KEK lookup against a Secret that exists without a usable 'kek' key, preserving the freshly generated KEK instead of nil/empty. - Default kms.vault to an empty dict and require address/keyName at render time so an incomplete Vault config fails with an actionable message rather than a generic nil dereference. - Publish the metrics port (9090) on the proxy Service so it can be scraped. - Clarify the backend-TLS note: the proxy negotiates TLS in-band; CNPG CA pinning is the remaining follow-up. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
f5b078d8c7 |
feat(keycloak): add optional KMS-encrypting DB proxy
Add an opt-in `encryption` flag to the keycloak system package. When encryption.enabled=true, a keycloak-kms-proxy Deployment + Service is placed on the wire between Keycloak and its CloudNativePG database, and Keycloak's KC_DB_URL_HOST/PORT are repointed at the proxy. The proxy transparently encrypts PII columns (username, email, first/last name, PII attributes, credentials) before they reach PostgreSQL and decrypts them on reads; the data-encryption key is wrapped by a KMS and never lives inside the database. The KMS backend is selectable: a self-generated static KEK (default, works on any cluster, suitable for evaluation) or HashiCorp Vault Transit (recommended for production, supports KEK rotation). The feature is off by default, so existing installs are unaffected. Skeleton/draft scope: backend TLS re-origination to CNPG, the Liquibase bootstrap window (KKP_LENIENT), an offline backfill Job for pre-existing data, and HA via a shared DEK set are tracked as follow-ups. Proxy project: https://github.com/cozystack/keycloak-kms-proxy Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Timofei Larkin
|
fa871249c6 |
feat(ci): gate sizeable API changes behind API-owner review
Add a deterministic api-gate tool (cmd/api-gate, internal/apigate) and a workflow that require an approving review from an API owner (lllamnyp or kvaps) when a PR introduces a new API group, a new resource, or a breaking schema change to an existing resource. Detection diffs the checked-in OpenAPIv3 schemas (apps cozyrds + CRD manifests) and the apiserver storage registrations; no network or LLM calls. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Aleksei Sviridkin
|
a31a73cb87 |
fix(tenant): gate VM-resource HelmReleases on victoria-metrics-operator readiness (#3141)
## What this PR does The tenant `etcd`, `ingress` and `monitoring` HelmReleases create VictoriaMetrics custom resources — `VMPodScrape` directly in `etcd` and `ingress`, and `VMCluster` / `VMAlert` / `VLCluster` / `VMAlertmanager` / `VMServiceScrape` via the `monitoring-system` child that `monitoring` creates. All of these are intercepted by the `victoria-metrics-operator` validating webhook, which is configured with `failurePolicy: Fail`. On a cold install the operator pod can be briefly unavailable on `:9443` (for example a transient API-server blip during startup leaves the webhook backend not yet serving). In that window the tenant releases reconcile, the webhook rejects their VM* resources with `connection refused`, and the releases churn through failed-install → rollback → retry. They carry `install.remediation.retries: -1` and recover eventually, but the churn can outlast a cold-install readiness deadline and surfaces as spurious install failures. This adds a Flux `spec.dependsOn` from each of those three tenant releases to the `victoria-metrics-operator` HelmRelease in `cozy-victoria-metrics-operator`, so they wait for the operator to be Ready before creating any VM* resource. The system monitoring components already gate on the operator the same way (a variant-level `dependsOn` in the platform package sources); this brings the tenant releases to parity. `failurePolicy: Fail` is preserved — validation is unchanged. The tenant releases are sharded (`sharding.fluxcd.io/key`) while the operator release is not. helm-controller resolves `dependsOn` through an uncached API read that bypasses the shard's watch-label-selector cache (since v1.1.0, PR fluxcd/helm-controller#1070), so the cross-shard reference is visible to the tenant shard's controller; this repo ships helm-controller v1.5.0. A helm-unittest suite pins the gate on all three releases and asserts that the tenant releases creating no VM* resources (`seaweedfs`, `gateway`) do not carry it, documenting the intentional narrowness of the gate. ### Release note ```release-note fix(tenant): gate tenant etcd, ingress and monitoring HelmReleases on victoria-metrics-operator readiness so they no longer fail to install when the operator's admission webhook is briefly unavailable on a cold install ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved tenant installation reliability by adding readiness gating: etcd, ingress, and monitoring Helm releases now wait for the Victoria Metrics operator to become Ready before applying dependent resources. * Prevents install/rollback failures caused by temporary admission webhook unavailability during cold starts. * **Tests** * Added Helm-unittest coverage to confirm the dependency gate is present only for templates that create VM* resources (and omitted for templates that don’t). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
03435dcc50 |
fix(flux): tune source-watcher concurrency + CPU to unblock platform install under load (#3163)
## What this PR does `source-watcher` is the Flux controller that turns each `PackageSource` into an `ExternalArtifact` (one `ArtifactGenerator` per source, ~100 builds on the management cluster during install). The `cozystack-platform` HelmRelease blocks until all of those artifacts are Ready, so source-watcher's throughput sits on the install's critical path. It was the only Flux controller left without concurrency/CPU tuning. In the bootstrap all-in-one pod it ran on the binary default (`--concurrent=10`) with the shared 100m CPU request; in the production `FluxInstance` it was absent from the reconciler patch that source-, kustomize- and helm-controller already receive. Under load its per-artifact builds finish at staggered times, leaving a random tail of not-yet-Ready artifacts that times out the platform install. A failed run's diagnostics show the shape of this: the `cozystack-platform` HelmRelease timed out with `timeout waiting for: PackageSource/… status: 'InProgress'` (a run-to-run varying tail — e.g. `bucket-application`, `linstor-gui`, `capi-provider-core`), while its ~200 ExternalArtifacts reached Ready in a staggered trickle spread over ~60 seconds rather than together. That staggered completion of independent per-artifact builds is the signature of a throughput-limited builder, not a single all-or-nothing pull. This raises source-watcher to `--concurrent=20` in both install paths: - **Bootstrap all-in-one pod** (`internal/fluxinstall/manifests/fluxcd.yaml`, regenerated from `flux-aio.cue` + the `flux-aio` Makefile): `--concurrent=20` plus a 500m CPU request. The AIO pod sets no CPU limits, so the request is what drives source-watcher's proportional CPU share when the node is saturated. - **Production `FluxInstance`** (`packages/system/fluxcd/values.yaml`): source-watcher is added to the shared reconciler patch, so it now gets the same `--concurrent=20`, `--requeue-dependency=5s` and 2000m/2048Mi limits as the other tuned controllers. The `flux-aio` module exposes a per-controller resources override for the watcher (set via CUE) but no args override, so the bootstrap `--concurrent` bump is a guarded post-render patch in the Makefile; the embedded manifest is the product of that pipeline. source-watcher's support for both `--concurrent` and `--requeue-dependency` was confirmed against its `cmd/main.go`, and the production `op: replace` on `resources/limits` is safe because the distribution ships source-watcher with a `resources.limits` block (same shape as source-controller). Optional follow-up, intentionally left out to keep this focused on the root cause: if the artifact-build tail still brushes the 10-minute platform HelmRelease timeout under load, the installer's HelmRelease install/upgrade timeout values give a safety margin. ### Release note ```release-note fix(flux): tune source-watcher concurrency and CPU so the platform install does not time out under load ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Increased the source watcher’s concurrency to improve throughput and better handle larger workloads. * Added higher CPU and memory settings for the watcher to support more stable operation under load. * **Bug Fixes** * Updated deployment patching so the watcher receives the same configuration adjustments as other controllers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
a40be5e212 |
fix(e2e): widen tenant-root monitoring bring-up waits to 15m under load (#3160)
## What this PR does The `Configure Tenant and wait for applications` install test brings up the entire root-tenant monitoring stack and waits for each component to become ready/operational. Those waits span the VictoriaMetrics operator resources (`vmalert`/`vmalertmanager`, `vlclusters/generic`, `vmcluster/shortterm`+`longterm`) and the Grafana pair (`grafana-db` CNPG cluster, `grafana-deployment`). Each converges only once its underlying workloads schedule and turn Ready, and during platform bring-up they contend for the same node resources as the rest of the install — so their convergence is load-sensitive. Only `vmalert`/`vmalertmanager` used a 15m budget; the other four used 10m, so this block carried a 10m/15m split even though all its resources contend for the same node capacity during install. Under install-time load (multiple concurrent e2e sandboxes on one runner) the monitoring bring-up is slow across the whole block rather than on a single resource, so the shorter budgets risk failing the whole install on PRs whose branches never touched monitoring — and per-resource bumps are whack-a-mole: raising one budget just moves the risk to the next component in the block. On a calm sandbox each of these reaches ready/operational in under a second, which is what distinguishes the two cases — slow-not-stuck convergence under contention, not a product regression. This lifts the whole tenant-root monitoring wait-set to a single uniform 15m budget and wraps each wait in an on-timeout diagnostic block that dumps the resource YAML (status conditions / `updateStatus`) plus tenant-root pod state. A genuinely stuck resource still fails — now at 15m, and with legible diagnostics instead of a bare `timed out` line — so this does not weaken the gate's ability to catch real breakage. The extra budget costs near-zero wall-time on the happy path and stays well inside the E2E job's 180m cap. The remaining upstream angle is the real driver: how long the VictoriaMetrics/VictoriaLogs stack and the Grafana bring-up take to become operational under resource contention during install. This change addresses the e2e robustness side; reducing that bring-up cost is a separate, product-side improvement. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved and standardized readiness checks during tenant setup so application bring-up waits behave more consistently. * Enhanced failure handling to provide clearer diagnostics (including resource details and pod status) when readiness checks time out. * **Tests** * Standardized longer readiness wait timeouts for multiple application components to reduce flakiness during e2e runs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
cd68d5efdc |
fix(e2e): drain tenant VMs/PVCs between kubernetes tests to fix the node-join flake (#3159)
## What this PR does Fixes the dominant flake in the tenant-Kubernetes e2e suite, where a run fails on worker-node-join under sandbox memory/storage pressure. The two heavy Kamaji tenant tests — `kubernetes-latest` then `kubernetes-previous` — run back-to-back in one CI sandbox, each provisioning a tenant cluster with KubeVirt worker VMs. The inter-test cleanup hook (`cozy_cleanup` in `hack/e2e-apps/run-kubernetes.sh`) deletes the tenant `Kubernetes` CR and waits only for the CR to disappear. That wait finalizes as soon as the CR's finalizers clear, which merely *triggers* teardown of the CAPK worker VMs and their DataVolume-backed disk PVCs — it does not await it. The virt-launcher pods keep their guest RAM reserved until the VMIs are gone, so the next tenant test's worker VMs begin scheduling against a sandbox the previous tenant has not yet vacated. Under the constrained sandbox this starves memory, a worker VM misses the node-join budget, and the MachineHealthCheck cannot remediate when both workers are unhealthy — so the run flakes on worker-node-join. This adds a bounded drain barrier to `cozy_cleanup`: after deleting the CR it now blocks until the tenant's VirtualMachines, VirtualMachineInstances and disk PVCs have actually drained from the `tenant-test` namespace, so the next test starts on a freed sandbox. The wait is bounded (5m) and best-effort: cozytest already runs `cozy_cleanup` wrapped in `|| true`, and the waiter returns loudly (warning plus a leftover dump) on timeout, so a stuck teardown can never hang the job past the deadline — it leaves the sandbox no worse than before. The probe is namespace-scoped: `tenant-test` is provisioned with etcd/monitoring/seaweedfs disabled so it carries no baseline PVCs, and the e2e apps run sequentially each cleaning up after themselves, so the only VMs/VMIs/PVCs present belong to the tenant being torn down. The pure drain exit-condition (`cozy_tenant_drained`) is factored out and unit-tested in `hack/run-kubernetes-drain_test.bats` (runs under `make unit-tests`), including the invariant that a failed probe's `err` sentinel is never misread as a completed drain. ### Screenshots Not applicable — e2e harness change, no UI. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes test cleanup so tenant resources are fully drained before the next test starts, reducing leftover VMs, VM instances, and PVCs. * Added more reliable timeout handling and clearer diagnostics when teardown or node readiness checks fail, making flaky test issues easier to identify. * **Tests** * Added coverage for tenant-drain detection to verify empty, whitespace-only, multi-resource, and error scenarios behave as expected. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6db5ae9c6f |
fix(tenant): gate VM-CR-creating HelmReleases on victoria-metrics-operator
The tenant etcd, ingress and monitoring HelmReleases create VM* custom resources (VMPodScrape directly in etcd/ingress; VMCluster, VMAlert, VLCluster, VMAlertmanager and VMServiceScrape via the monitoring-system child of monitoring) guarded by the victoria-metrics-operator validating webhook with failurePolicy: Fail. On a cold install the operator pod can be briefly unavailable on :9443, so these releases reconcile before the webhook backend is serving and are rejected with connection refused. They carry install.remediation.retries: -1 and would eventually recover, but the failed-install -> rollback -> retry churn can outlast a cold-install readiness deadline and surfaces as spurious install failures. Add a Flux spec.dependsOn from each of these releases to the victoria-metrics-operator HelmRelease in cozy-victoria-metrics-operator, so they wait for the operator to be Ready before creating VM* resources. The system monitoring components gate on the operator the same way, via a variant-level dependsOn. The cross-shard reference (tenant releases are sharded, the operator release is not) resolves via helm-controller's uncached dependency check (APIReader, v1.1.0+). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
8f60b27f9a |
feat(ingress): serve replicated wildcard on per-tenant ingress controllers
In operator wildcard mode each tenant's ingress controller now sets default-ssl-certificate to the wildcard Secret in its own namespace, so a child tenant serves the replicated wildcard instead of falling back to per-host ACME. ingress-nginx reads the default certificate only from its own namespace, and each ingress HelmRelease renders in the namespace that owns the controller, so the reference stays same-namespace for the root and every child alike. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
c841c70866 |
feat(operator): replicate operator wildcard certificate to tenant namespaces
The operator-provided wildcard MVP serves the root tenant only: the root ingress controller and Gateway read a TLS Secret from their own namespace. Child tenants run their own ingress controller / Gateway in a separate, default-deny namespace and cannot read that Secret cross-namespace. Add a controller that mirrors the operator wildcard Secret into every tenant namespace that owns a TLS termination point (its namespace.cozystack.io/ingress or .../gateway label equals its own name). The source is identified by name from the platform values channel (_cluster.wildcard-secret-name) with no extra operator input; replicas carry a managed-by label and refresh on rotation. An individual replica is removed when its namespace stops terminating TLS; every replica is torn down only on an explicit disable (clearing wildcard-secret-name). A missing or mistyped source, or a transiently-absent values channel, keeps existing replicas, so tenant TLS never drops on a transient gap. The controller's Secret cache is scoped to managed replicas and the values channel, not every cluster Secret. Only the platform controller's ServiceAccount gains Secret writes — no tenant RBAC is widened, and every consumer reads only its own-namespace copy. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
b25a132747 |
fix(velero): add startupProbe so slow startup does not crashloop the install gate (#3138)
## What Add a `startupProbe` to the velero server container (and supply it through the package values), so a slow server start under a heavy parallel install no longer trips the liveness probe. Pin the probe with a helm-unittest regression guard. ## Why Velero binds its metrics/health endpoint (`:8085`, `http-monitoring`) only after the server loads its plugins and connects to the API server. Under a heavy parallel platform install that can take minutes, but the upstream liveness probe starts checking `/metrics` at `initialDelaySeconds=10` and kills the container after `failureThreshold=5` (~130s total) — before `:8085` is bound. The kubelet SIGTERMs velero (exit code 0 / Completed — a graceful kill, not a crash), it BackOff-restarts, and never escapes the loop inside the install-gate window. Because `backupstrategy-controller` hard-depends on velero, this blocks the install gate. ## How While a `startupProbe` is failing, the kubelet holds off the liveness and readiness probes entirely. So the startupProbe grants the server an explicit startup budget — `failureThreshold * periodSeconds = 30 * 10s = 300s` of grace — and then hands over to the unchanged, tight upstream liveness (`10s / 30s / 5`) for steady-state crash detection. This is strictly better than loosening liveness, which would also blunt steady-state detection for the lifetime of every pod. The upstream velero chart (12.0.3) does not render a `startupProbe`, so the package patches the deployment template via `patches/add-startup-probe.patch`, re-applied after re-vendor by the Makefile `update` target — the same vendored-patch convention used by `seaweedfs`, `ingress-nginx`, and the existing `startupProbe` patches in `grafana-operator` and `kubevirt-cdi-operator`. The `startupProbe` value itself is supplied through the package values override. ## Notes Surfaced as a recurring (not flaky) install failure under heavy parallel load. Does not close a specific issue. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a configurable Velero `startupProbe` for the `/metrics` endpoint (`http-monitoring`) to better accommodate slow startup. * **Bug Fixes** * When metrics are disabled, the rendered container no longer includes `startupProbe`, `livenessProbe`, or `readinessProbe`. * **Tests** * Added invariant chart rendering tests to verify `startupProbe` behavior (including pinned timing fields) and confirm existing probe timing remains unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
41ff94eca9 |
fix(flux): tune source-watcher concurrency and CPU under load
source-watcher reconciles one ArtifactGenerator per PackageSource and builds the resulting ExternalArtifacts. On the management cluster that is ~100 artifact builds during install, and the platform HelmRelease blocks until all of them are Ready. It was the only Flux controller left untuned. In the bootstrap all-in-one pod it ran without an explicit --concurrent (binary default 10) and with the shared 100m CPU request; in the production FluxInstance it was missing from the --concurrent / CPU patch that source-, kustomize- and helm-controller already receive. Under load its per-artifact builds finish at staggered times, leaving a random tail of not-yet-Ready artifacts that times out the platform install. Raise source-watcher to --concurrent=20 in both deployments. In the bootstrap all-in-one pod also give it a 500m CPU request; the pod has no CPU limits, so the request is what drives its CPU share under contention. In the production FluxInstance add it to the shared reconciler patch, so it now receives the same --requeue-dependency=5s and 2000m/2048Mi limits the other tuned controllers already have. The flux-aio module exposes a resources override for the watcher (set via CUE) but no args override, so the bootstrap --concurrent bump is a post-render patch in the Makefile. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
09f9d0bdec |
fix(e2e): widen Grafana bring-up waits to 15m under load
The Configure-Tenant install test completes the tenant-root monitoring bring-up by waiting for the grafana-db CNPG cluster and the grafana-deployment Deployment. Both used a 10m budget while the VictoriaMetrics waits above them use 15m. They complete the same monitoring bring-up and contend for the same node resources during install, so under install-time load (concurrent e2e sandboxes on one runner) either can be slow, and the 10m/15m split within the same contend-for-resources bring-up risks failing the whole install on a PR that never touched monitoring. Raise both Grafana waits to the same uniform 15m budget as the vm-operator waits and wrap them in an on-timeout diagnostic block that dumps the resource YAML plus tenant-root pod state, so a genuinely stuck resource still fails at 15m with legible diagnostics instead of a bare "timed out" line. The extra budget costs near-zero wall-time on the happy path and stays inside the E2E job budget. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
1ffbb30d25 |
fix(e2e): widen VictoriaMetrics operational waits to 15m under load
vlclusters/generic and vmcluster wait for updateStatus=operational with a 10m budget, while their vmalert/vmalertmanager sibling in the same block already uses 15m. All three are vm-operator-managed monitoring resources that only become operational once their workloads schedule and turn Ready, and during platform bring-up they contend for node resources with the rest of the install. On a calm sandbox each converges in under a second, but under install-time load (concurrent e2e sandboxes on one runner) monitoring bring-up is slow, and a 10m/15m split inside a block whose resources all contend for the same node capacity risks failing the whole install on a PR that never touched monitoring. Unify vlclusters and vmcluster on the sibling's 15m budget so the whole block shares one headroom, and dump resource YAML plus pod state on timeout across all three waits so a genuine stuck-not-slow regression stays legible instead of a bare "timed out" line. The extra budget costs near-zero wall-time on the happy path and stays inside the E2E job budget. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f7271b8011 |
fix(velero): add startupProbe so slow startup does not crashloop the install gate
Velero binds its metrics/health endpoint (:8085, http-monitoring) only after the server loads its plugins and connects to the API server. Under a heavy parallel platform install that can take minutes, but the upstream liveness probe begins checking /metrics at initialDelaySeconds=10 and kills the container after failureThreshold=5 (~130s total) before :8085 is bound. The kubelet SIGTERMs velero (exitCode 0/Completed), it BackOff-restarts, and never escapes the loop inside the install-gate window. Because backupstrategy-controller hard-depends on velero, this blocks the gate. Add a startupProbe to the velero server container. While it is failing the kubelet holds off liveness and readiness entirely, granting failureThreshold * periodSeconds = 30 * 10s = 300s of startup grace, then hands over to the unchanged, tight upstream liveness (10s / 30s / 5) for steady-state crash detection. This is strictly better than loosening liveness, which would also blunt steady-state detection. The upstream velero chart (12.0.3) does not render a startupProbe, so the package patches the deployment template via patches/add-startup-probe.patch (re-applied after re-vendor by the Makefile update target, matching the seaweedfs/ingress-nginx convention) and supplies the startupProbe through the package values override. Pin the startupProbe budget and the unchanged liveness with a helm-unittest regression guard. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9c18260f95 |
fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets) (#2957)
## What The e2e install pipeline still flakes on a loaded runner, failing at a different point each run. Root-caused via repeated runs and a live SSH breakpoint; the platform itself is fine (the components do converge), the failures are slowness + self-inflicted churn under resource pressure. ### Fixes 1. **VPA updater OOM** — the `vertical-pod-autoscaler-updater` memory limit was `110Mi`. The updater watches every pod in the cluster, so on a full install it OOM-kills → `ProgressDeadlineExceeded` → Flux fails the HelmRelease "early due to stalled resources", cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raised to `512Mi` (request `200Mi`). 2. **VPA-for-VPA eviction churn** — the two VPA-for-VPA `VerticalPodAutoscaler` objects ran `updateMode: Auto`, evicting the VPA recommender(s) during install and adding pod churn on an already busy cluster. Switched to `Initial`, matching the existing monitoring/etcd `VPA->Initial` fixes. 3. **Install wait budgets** — components reach readiness but overshoot tight waits on a busy runner: keycloak HR chain `10m→20m`, tenant aggregate wait `10m→20m`, per-app tenant readiness waits `5m→10m`. (The crust-gather failure-snapshot timeout that kept whole jobs hanging to the 3h cancel is already on `main`.) ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased several installation and readiness wait times to better handle slower environment startup and reduce premature failures. * Updated vertical autoscaling behavior to avoid unnecessary pod restarts during deployment. * Raised resource limits for the autoscaling updater to improve reliability under load. * **Tests** * Aligned automated checks with the new autoscaling behavior and longer readiness expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
ec42c52967 |
fix(e2e): skip cilium-agent restart for the reserved:ingress IP variant (#3142)
## What this PR does The Cilium leak healer (`hack/e2e-cilium-endpoint-leak-healer.sh`, the interim CI band-aid for the Cilium endpoint-leak class) escalates an unhealed leaked-IP reuse to a Tier-2 `cilium-agent` restart. For one variant that restart is both futile and harmful. When the disputed IP is the node's per-node Cilium ingress IP (`CiliumNode.spec.ingress.ipv4`), it is held by the `reserved:ingress` endpoint — allocated from the node pod CIDR whenever Gateway API / Envoy is enabled (`enable-envoy-config`, `daemon/infraendpoints/infra_ip_allocation.go`). Restarting the agent cannot clear it: the agent re-derives the same ingress IP from the CiliumNode spec on every boot. Worse, the restart can re-trigger the race that caused the wedge in the first place — it rebuilds the host-scope (`kubernetes` IPAM) allocation bitmap empty (`pkg/ipam/hostscope.go`; `RestoreFinished()` is a no-op), then re-reserves the ingress IP via a path decoupled from pod-endpoint restore, so a CNI ADD served in that window can be handed the ingress IP again. That is the infra-IP-vs-pod-IP double-allocation behind the "IP already in use" sandbox rejection. This is an upstream Cilium bug in host-scope IPAM with Gateway API; no released Cilium fixes it as of v1.19.5 (the deployed version). The complete fix belongs upstream — the agent must reserve `spec.ingress.ipv4` in the host-scope allocation bitmap before the CNI server serves allocations, or reconcile it with the restored `reserved:ingress` endpoint, or source it from a dedicated range. That is tracked upstream in cilium/cilium#46799. This PR makes the CI healer correct for that variant: it detects an exact match of the wedged IP against the node's `CiliumNode.spec.ingress.ipv4` and skips the Tier-2 restart — surfacing the case and rescheduling the wedged pod best-effort instead. The match is gated so an empty ingress IP (CiliumNode absent or RBAC denied) never matches, so a false positive can never suppress a restart that would clear a real in-memory leak. The change adds a `ciliumnodes` read to the healer RBAC and a unit test pinning the exact-match-and-non-empty contract. ### Screenshots N/A — CI test-infrastructure change, no UI impact. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Enhanced endpoint leak healer to skip Tier-2 agent restarts when the disputed IP exactly matches a node’s reserved per-node Cilium ingress IP. * Added lightweight unit tests for the new IP-matching behavior. * **Bug Fixes** * Prevents unnecessary remediation when the disputed IP is empty or doesn’t exactly match the node ingress IP. * When a match is detected, the healer deletes the affected wedged pod as a best-effort reschedule step. * **Chores** * Updated healer permissions to read Cilium node ingress information cluster-wide. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
93d2c30a8f |
fix(kubevirt-cdi): order behind cozystack-basics so cdi-clone-dv RoleBinding finds cozy-public (#3137)
## What Add a `cozystack.cozystack-basics` edge to the `kubevirt-cdi` `default` variant's `dependsOn` in the platform package, plus a helm-unittest regression guard. ## Why `kubevirt-cdi` ships the `cdi-clone-dv` RoleBinding into the `cozy-public` namespace, but `cozy-public` is provisioned by `cozystack-basics`. The variant previously only depended on `cozystack.networking`, so kubevirt-cdi reconciled long before `cozystack-basics` created the namespace. On a fresh install the RoleBinding could race namespace creation and fail with `namespaces "cozy-public" not found`. Flux retries normally mask this, but under slow reconcile it surfaces as a hard `UpgradeFailed`. This mirrors the existing `backupstrategy-controller` ordering, which already depends on `cozystack-basics` for the identical reason (it writes a Bucket into `tenant-root`). There is no dependency cycle: `cozystack-basics`' own dependencies (`tenant-application`, `cozystack-engine`, `gateway-api-crds`) do not reach `kubevirt-cdi`. ## Notes Surfaced via a slow-reconcile install in #3124's E2E. Does not close a specific issue. Tradeoff: kubevirt-cdi now installs after the full `cozystack-basics` chain instead of just `networking`. Its only consumers (`vm-default-images`, `vm-disk-application`) install later still, so the added latency introduces no new ordering problem — the same correctness-over-speed tradeoff already accepted for `backupstrategy-controller`. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved upgrade reliability for kubevirt-cdi by enforcing correct dependency ordering before applying its RoleBinding-related resources. * Reduced intermittent `UpgradeFailed` outcomes during slower reconciliation scenarios. * **Tests** * Added a regression test to verify kubevirt-cdi manifest `dependsOn` includes the expected prerequisites for consistent deployment. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
d54e3e4b58 |
test(e2e): capture worker VM/CSR state on tenant node-join failure
When a tenant Kubernetes cluster hits the 12m node-Ready deadline with fewer than two nodes joined, the failure path dumped only `describe nodes` and `get hr`. That cannot tell apart the two node-join failure sub-modes: a worker VM that never booted (virt-launcher Pending or OOMKilled) versus a VM that booted fine but whose kubelet never registered a Node. The tenant cilium-operator HelmRelease reports "InProgress" identically in both, so the current dump is not enough to diagnose the failure or design a fix. Dump scoped, guarded diagnostics before failing fast: - worker VM/VMI/virt-launcher state on the management cluster, which is what actually splits the two sub-modes; - tenant kubelet CSRs plus the talos-csr-signer sidecar log, which reveal a boot-but-no-Node stall. In-guest Talos/kubelet capture is deliberately left out: talosctl needs a tenant-cluster talosconfig the runner does not have (the tenant workers use a separate Talos CA from the management sandbox), so the gap is documented in the failure path rather than shipped as a call that would fail mTLS and capture nothing. Every capture is guarded with `|| true` so a capture failure never masks the real exit. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei KvapilandClaude
|
d6f66d2dcd |
fix(vertical-pod-autoscaler): stop the updater OOM + VPA-for-VPA eviction churn
Two root causes of e2e install flakiness, both in the VPA package: 1. The updater memory limit was 110Mi. The updater watches every pod in the cluster, so on a full cozystack install it OOM-kills, which trips the Deployment's ProgressDeadlineExceeded and makes Flux fail the HelmRelease 'early due to stalled resources' - cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raise the limit to 512Mi (request 200Mi); the recommender is already unlimited. 2. The two VPA-for-VPA VerticalPodAutoscaler objects ran updateMode: Auto, which evicts the VPA recommender(s) during install and adds pod churn on an already busy cluster. Switch to Initial, matching the existing monitoring/etcd VPA->Initial fixes. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> Signed-off-by: Aleksei Sviridkin <f@lex.la> |