mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
ci/cozystack-api-pin-securitygroup
1075
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
6db5ae9c6f |
fix(tenant): gate VM-CR-creating HelmReleases on victoria-metrics-operator
The tenant etcd, ingress and monitoring HelmReleases create VM* custom resources (VMPodScrape directly in etcd/ingress; VMCluster, VMAlert, VLCluster, VMAlertmanager and VMServiceScrape via the monitoring-system child of monitoring) guarded by the victoria-metrics-operator validating webhook with failurePolicy: Fail. On a cold install the operator pod can be briefly unavailable on :9443, so these releases reconcile before the webhook backend is serving and are rejected with connection refused. They carry install.remediation.retries: -1 and would eventually recover, but the failed-install -> rollback -> retry churn can outlast a cold-install readiness deadline and surfaces as spurious install failures. Add a Flux spec.dependsOn from each of these releases to the victoria-metrics-operator HelmRelease in cozy-victoria-metrics-operator, so they wait for the operator to be Ready before creating VM* resources. The system monitoring components gate on the operator the same way, via a variant-level dependsOn. The cross-shard reference (tenant releases are sharded, the operator release is not) resolves via helm-controller's uncached dependency check (APIReader, v1.1.0+). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
5b26ec3462 |
fix(mariadb): clean up orphaned operator-generated secrets (#3056) (#3071)
## What this PR does After deleting a `MariaDB` app CR with metrics and/or replication enabled, two Secrets created by the mariadb-operator were left orphaned in the tenant namespace: - `<release>-metrics-password` - `<release>-repl-password` These are generated by the operator (`k8s.mariadb.com/v1alpha1`) as a side effect of `metrics.enabled` / `replication.enabled`, so they are **not** Helm-owned and survive `helm uninstall`. The existing post-delete hook (`templates/hooks/cleanup-pvc.yaml`) only swept PVCs, leaving the Secrets behind. This PR extends that hook to also clean them up: - grants the cleanup `Role` RBAC on `secrets`; - deletes both password Secrets by name with `--ignore-not-found`; - adds a `helm-unittest` suite locking in the behaviour, plus a `test` Makefile target so the package's tests actually run in CI (`hack/helm-unit-tests.sh` only runs packages that define a `test` target). Fixes #3056. ### Release note ```release-note fix(mariadb): garbage-collect operator-generated metrics-password / repl-password Secrets when a MariaDB app is deleted ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved MariaDB uninstall cleanup to remove leftover operator-generated password Secrets in addition to persistent volumes/claims. * Hardened the cleanup hook Job with stricter pod/container security settings, least-privilege Secret permissions, idempotent deletions for missing resources, and correct non-zero exit on real failures. * **Tests** * Added a cleanup hook test suite validating Secret deletion behavior, failure propagation, and that cleanup RBAC grants least-privilege access only to the required Secrets. * Added a Helm unit test `test` target for the MariaDB chart. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
bef42229d4 |
fix(harbor): clean up orphaned jobservice PVC and ACME solver resources on delete (#3061) (#3077)
## What this PR does After deleting a `Harbor` app CR, two things were left orphaned in the tenant namespace: - the **jobservice / trivy PVCs** (`<release>-jobservice`, and `<release>-trivy` when Trivy is enabled), because `packages/apps/harbor/templates/harbor.yaml` sets `persistence.resourcePolicy: "keep"`, which annotates the upstream Harbor subchart's PVCs with `helm.sh/resource-policy: keep` so Helm/Flux deliberately never deletes them; - cert-manager's **ACME http01-solver** resources (`cm-acme-http-solver-*`), via the same stuck-challenge mechanism as the Bucket issue (#3060) when Harbor issues a non-wildcard cert. This adds a single `post-delete` cleanup hook that: - **deletes the upstream Harbor subchart's kept PVCs** — selecting by `app.kubernetes.io/instance=<release>-system` (the subchart runs as the `<release>-system` HelmRelease, so its PVCs carry that label, **not** `<release>`). A hook is required precisely because `resourcePolicy: keep` blocks normal reclaim. This intentionally covers both the jobservice and trivy PVCs: both hold only regenerable data (jobservice file logs; trivy's vulnerability-DB cache), and registry images live in S3. - **deletes this release's `Certificate`** (deterministic name `<release>-ingress-tls`); cert-manager then cascade-deletes the `Order`, `Challenge` and solver `Pod`/`Service`/`Ingress` objects it owns via owner references. - **host-scoped belt-and-braces sweep** for any ACME http01 solver objects cert-manager may have orphaned (e.g. the Certificate was already gone, so deleting it did not cascade). The sweep is scoped **strictly to this release's Harbor host** — Challenges filtered by `.spec.dnsName`, solver Ingresses by `.spec.rules[0].host` — **not** a namespace-wide `acme.cert-manager.io/http01-solver=true` label sweep, because a Cozystack tenant namespace can host many apps and a blanket label sweep would delete sibling apps' in-flight challenge solvers. The Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`, with CPU/memory requests and limits set. RBAC is scoped to exactly the resources it deletes (`persistentvolumeclaims`, `certificates`, `orders`, `challenges`, solver `ingresses`) and verified against the vendored cert-manager CRDs. Failures are logged but never block teardown. ## Cleanup scope — what is and isn't removed **Removed:** the upstream Harbor subchart's kept PVCs (jobservice + trivy) and this release's cert-manager ACME artifacts (Certificate, and host-scoped Order/Challenge/solver-Ingress). **Not touched (by design):** - **CNPG database PVCs** (`<release>-db-N`) — owned by the CNPG `Cluster` and governed by CNPG's own reclaim policy; they carry CNPG labels, not `app.kubernetes.io/instance=<release>-system`. Re-installing Harbor with a different DB size into the same namespace can therefore leak the old DB PVCs; this is left to CNPG's lifecycle on purpose (the data is the user's). - **Redis PVCs** — managed by the redis-operator with their own labels, likewise untouched. - **Solver `Pod`/`Service`** — not deleted directly; they are removed by cert-manager's owner-ref cascade when the Certificate/Challenge is deleted, so the hook grants no RBAC for them. > [!NOTE] > The PVC sweep is eventually-consistent: while the inner `<release>-system` > HelmRelease is still being uninstalled by Flux, a PVC may briefly stay in > `Terminating` until its consumer Pods (jobservice/trivy) exit. The Job's > deletion request always succeeds; reclaim completes once the pods are gone. > [!WARNING] > **This overrides the intentional `persistence.resourcePolicy: "keep"` setting > and force-deletes the jobservice/trivy PVCs on app deletion.** Those PVCs hold > only jobservice file logs and the trivy vuln-DB cache (registry images live in > S3), so data-loss risk is low — but if retaining them across re-installs is > desired, this is a conscious product decision to revisit. The ACME cleanup > affects only cert-manager's transient artifacts. Fixes #3061. > Follow-up tracked in #3088: digest-pin the cleanup `kubectl` image and route it > through `cozy-lib.images-registry` (pre-existing debt shared with the MariaDB > cleanup hook), out of scope for this PR. ### Release note ```release-note fix(harbor): clean up the orphaned jobservice/trivy PVCs and cert-manager ACME http01-solver resources left behind after a Harbor app is deleted. NOTE: this force-deletes the jobservice/trivy PVCs, overriding persistence.resourcePolicy=keep (jobservice file logs and trivy vuln-DB cache only; registry images are in S3). CNPG database PVCs are not touched. ``` |
||
|
|
3e20a0cd41 |
fix(kubernetes): make node group resources optional when instanceType is set (#3121)
## What this PR does
A tenant Kubernetes node group can be sized two ways: by `instanceType`
or by an explicit `resources` block. The generated values schema,
however, marked `resources` as required on every node group, so a node
group sized purely by `instanceType` was rejected at install time:
```text
at '/nodeGroups/<name>': missing property 'resources'
```
The same failure occurs through the typed `Kubernetes` CR: an empty
`resources: {}` is dropped during the CR-to-HelmRelease translation, so
by validation time the key is absent and the required check fails — even
though sizing was already fully specified by `instanceType`.
This marks `resources` optional in the node group definition and
regenerates the dependent artifacts (`values.schema.json`, API types,
README, and the `ApplicationDefinition`). No change is needed to emit
the VM: the chart already renders `domain.cpu`/`domain.memory` only when
`resources` is present and `instancetype` only when set, so a node group
sized purely by `instanceType` now validates and installs. Node groups
that specify `resources` are unaffected.
`instanceType` and `resources` are alternative, mutually exclusive
sizing sources — KubeVirt rejects a VirtualMachine that references an
instancetype and also sets explicit `domain.cpu`/`domain.memory`, since
instancetype CPU/memory cannot be overridden. To avoid emitting a VM
that the instancetype admission webhook would later reject, the chart
now fails fast at render time with a clear message when a node group
sets both a non-empty `instanceType` and explicit `resources`.
Tests: helm-unittest coverage proves a node group with `instanceType`
and no `resources` passes schema validation and renders without explicit
`domain.cpu`/`memory`; a complementary case confirms explicit
`resources` are still honoured; and a `failedTemplate` case pins the
render-time rejection of the conflicting both-set combination. The
omit-resources case fails against the pre-fix schema with the exact
error above, so it is a real regression guard.
Addresses #3046.
### Screenshots
N/A — schema and chart change, no UI impact.
### Release note
```release-note
fix(kubernetes): allow a node group to omit `resources` when it is sized by `instanceType` (previously failed schema validation with "missing property 'resources'"); reject the unsupported combination of `instanceType` and explicit `resources` at render time
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Worker node groups can optionally size CPU/memory via `resources`, and
can omit `resources` when using `instanceType`.
* **Bug Fixes**
* Render now validates CPU/memory rules: `instanceType` with only one of
`resources.cpu`/`resources.memory` fails, and `instancetype` is omitted
when both `resources.cpu` and `resources.memory` are set (with
`instanceType` ignored).
* **Documentation**
* Updated API/Helm docs and JSON schema to clarify `resources` vs
`instanceType` precedence and the “set both CPU and memory (or neither)”
requirement.
* **Tests**
* Added helm-unittest scenarios covering omission, precedence, and
render-time failure cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
0892c5a38c |
fix(qdrant)!: clean up orphaned data PVC on delete (#3059) (#3075)
## What this PR does After deleting a `Qdrant` app CR, the StatefulSet data PVC (`qdrant-storage-<release>-0`) was left orphaned in the tenant namespace, consuming storage permanently. StatefulSet-templated PVCs are never garbage-collected on StatefulSet deletion by default. The vendored upstream Qdrant chart does **not** support `persistentVolumeClaimRetentionPolicy` (verified: its `statefulset.yaml` never renders that field), so the OpenBAO-style value passthrough is not possible here. Instead this adds a `post-delete` PVC cleanup hook (modeled on the MariaDB hook) that deletes the release's PVCs by label. The chart's volumeClaimTemplate only stamps `app: qdrant` (shared by every Qdrant in a namespace), so to select PVCs safely per release this also passes `persistence.additionalLabels.app.kubernetes.io/instance: <release>` through the HelmRelease — giving the data PVC a release-unique label the hook selects on. The cleanup Job is hardened: runs as non-root (uid 65534), read-only root fs, all capabilities dropped, `seccompProfile: RuntimeDefault`; RBAC scoped to `persistentvolumeclaims`. A real cleanup failure is logged (not silently swallowed) but never blocks namespace teardown. > [!WARNING] > **Deleting a Qdrant app now permanently removes its data PVC and the vector data it holds.** > This is intentional and fixes the storage leak; the hook only fires on full > release deletion (not on scale-down). Back up before deleting. Fixes #3059. ### Release note ```release-note fix(qdrant): garbage-collect the StatefulSet data PVC when a Qdrant app is deleted, instead of leaving it orphaned. WARNING: deleting a Qdrant app now permanently removes its data PVC and the data it holds. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a post-delete cleanup hook for Qdrant to remove leftover StatefulSet data PVCs after deletion. * Included hardened Kubernetes settings for the cleanup job to improve runtime safety. * **Tests** * Added Helm chart tests that validate the post-delete cleanup job and its RBAC permissions. * Enhanced end-to-end verification to confirm the data PVC is fully removed after deleting a Qdrant instance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b25f294d46 |
fix(harbor): select kept PVCs by release label so the trivy PVC is also cleaned
The post-delete cleanup hook selected PVCs by app.kubernetes.io/instance=<release>-system, a label only the jobservice PVC carries (via the chart's "harbor.labels" helper). The trivy PVC is a StatefulSet volumeClaimTemplate labeled via "harbor.legacy.labels" with a selector using "harbor.matchLabels" — neither sets app.kubernetes.io/instance — so the sweep silently missed it. Trivy is enabled by default, so a default Harbor delete left the trivy PVC orphaned, contradicting the PR's own claim. Select on release=<release>-system instead: both kept PVCs carry it, while CNPG (db) and Redis PVCs use different label schemes and the registry is on S3, so only the two intended PVCs match. The unit test now asserts the corrected selector and guards against regressing to the instance label. Also drop the by-name delete of Orders/Challenges (their hash-suffixed names make a by-name delete a no-op; the Certificate owner-ref cascade plus the host-scoped sweep handle them), narrow the hook RBAC to challenges only, and note the Gateway-mode no-op. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
11c417e258 |
test(mariadb): assert both cleanup failure branches and exact RBAC rule set
Strengthen the cleanup hook suite per review: - assert the PVC and Secret delete branches each set cleanup_failed=1, not just the final exit, so a regression in either branch is caught; - pin the Role to exactly two rules so a broader unscoped secrets rule can't slip past the subset `contains` check. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
ae7c232409 |
fix(qdrant): anchor PVC ordinal so cleanup never deletes a sibling's data
The post-delete hook selected PVCs with an unanchored name prefix (qdrant-storage-<release>-). Because release names are qdrant-<app>, deleting release "qdrant-test" matched not only its own PVC (qdrant-storage-qdrant-test-0) but also a sibling "qdrant-test-2"'s PVC (qdrant-storage-qdrant-test-2-0) in the same namespace — irreversibly deleting the sibling's data (or wedging it in Terminating via pvc-protection). Anchor the ordinal with grep -E "...-[0-9]+$" so only the release's own data PVCs match. A release name that is a prefix of a sibling no longer collides. - cleanup-pvc.yaml: grep -E with anchored "-[0-9]+$". - tests: assert the anchored selector and add a notMatchRegex guard against regressing to the unanchored prefix. - e2e: gate the post-delete wait on a NotFound result specifically, so a transient API/auth error is not misread as "PVC reclaimed". Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
c2a8a3b41f |
fix(mariadb): fail cleanup hook on real delete errors
The post-delete cleanup hook used `kubectl delete ... --ignore-not-found || echo WARNING`, which swallowed genuine RBAC/API failures and let the Job succeed with orphaned password Secrets still present. Track delete failures and propagate them via a non-zero exit, while --ignore-not-found still tolerates already-absent resources. Add a helm-unittest assertion locking in the non-zero exit-on-failure behaviour. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
a349c753ad |
fix(kubernetes): make node group resources optional and prefer them over instanceType
A node group is sized either by instanceType or by explicit resources, but the generated values schema required resources on every node group. A node group with instanceType and no resources — or an empty resources dropped during typed-CR to HelmRelease translation — failed schema validation with "missing property 'resources'" even though sizing was fully specified. Mark resources optional in the node group definition and regenerate the schema, API types, README, and ApplicationDefinition. The chart template emits domain.cpu/memory only when resources is present, so a node group sized purely by instanceType now passes validation and installs. When a node group sets both instanceType and explicit resources, explicit resources take precedence: the instancetype is omitted from the rendered VM and the per-group instancetype lookup is skipped, so KubeVirt accepts the explicit domain.cpu/memory with no instancetype to conflict with. instanceType stays required and defaulted (no schema change), so the console's required instanceType dropdown keeps working — sending the default instanceType plus explicit resources simply sizes the VM by resources. This replaces an earlier render-time hard failure on the both-set case, which was unreachable to clear from the dashboard (the required dropdown cannot be emptied). Explicit resources are all-or-nothing alongside an instanceType: a partial spec (only cpu, or only memory) on a group that carries a non-empty instanceType would emit one domain override next to the instancetype — which KubeVirt rejects — so it now fails at render time with a clear error. Partial resources on a group without an instanceType (used to tune kubelet reservations) are unaffected. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
7c67aafed5 |
fix(kubernetes): run the Talos reconcile Job in the main install phase (#3145)
## What this PR does The tenant worker `MachineDeployment` `md0` intermittently never reaches `status.replicas=2`. CAPI blocks MachineSet creation until the `TalosConfigTemplate` that the deployment's `bootstrap.configRef` points to exists, and on failing installs that template is never created. ### Root cause — an ordering deadlock The `TalosConfigTemplate` is created only by the `talos-reconcile` **post-install Helm hook** (the chart can no longer render it at template time — three of its inputs are produced asynchronously after apply). A post-install hook runs only after Helm has applied the release's main resources and, when Helm waits, after they become Ready. But the same release also emits in-tenant addon HelmReleases — notably **cilium** — whose readiness needs tenant worker nodes, and those nodes can only be created once this hook has produced the `TalosConfigTemplate`. So when the main-resource wait is in effect, the install deadlocks: it waits for cilium to be Ready, cilium waits for worker nodes, the nodes wait for the `TalosConfigTemplate`, and the template waits for the post-install hook — which never runs, because the install never finishes the wait that precedes the hook phase. The install times out and `md0` never scales. The release sets `helm-install-disable-wait` precisely to skip that main-resource wait so the hooks run, but its effect is unreliable — which is what makes the failure bimodal: when the wait is skipped the hook runs and the template lands in ~46s; when it is not, the deadlock strands the install. A timestamped poller over a failing run (both `kubernetes-latest` and `kubernetes-previous`, 640 polls) captured: `TalosConfigTemplate` absent in 640/640 polls; both post-install hook Jobs (`talos-reconcile` and `bootstrap-token`) never created — the install never reached the post-install hook phase; all main resources applied (MachineDeployment, KubevirtMachineTemplate, MachineHealthCheck, WorkloadMonitor, child HelmReleases, Talos PKI/secrets) and the four hook inputs ready early and stable; the parent HelmRelease stuck at `Running 'install' action 20m` then looping into `upgrade`, with the cilium sub-release timing out on `cilium-operator` (which has no worker node to schedule on) — downstream of the missing template. ### The fix Render the `talos-reconcile` Job and its RBAC / ServiceAccount / egress `CiliumNetworkPolicy` as ordinary **main-phase** resources instead of post-install hooks. The Job is then created during the apply itself — not after the wait — and runs on the management cluster, which has nodes (unlike the tenant `cilium-operator`). It produces the `TalosConfigTemplate`, `md0` scales, workers join, cilium goes Ready, and the main-resource wait (if any) closes within the install budget: the Job completes in tens of seconds, and node boot plus cilium readiness is a few minutes — comfortably inside the 20m install timeout. This works whether or not `helm-install-disable-wait` takes effect, so it does not depend on that annotation's reliability. Because a main-phase Job is immutable, it carries a content-hash name suffix over its whole rendered spec (mirroring the `KubevirtMachineTemplate` idiom in `cluster.yaml`): an identical render keeps the same name and Helm no-ops it; any spec change yields a new name and a fresh Job, so no change ever lands on the same name and trips an immutable-field patch error. The applied `TalosConfigTemplate` keeps its stable `<release>-<group>` name (referenced by `bootstrap.configRef`), so an unchanged render re-applies a byte-identical template — a CAPI no-op that never rolls existing workers. The Job name is truncated to stay within the 63-char DNS-label limit while always preserving the hash. The runtime input-waits, the single `kubectl apply` of the template with a `KamajiControlPlane` ownerReference for GC, and the `extraHostEntries` are all unchanged. `helm-install-disable-wait`'s unreliable effectiveness is a real latent issue (it explains the created-vs-never bimodality), but this fix breaks the deadlock independent of it, so that is a separate follow-up rather than a blocker here. Supersedes #3139 (a 10-minute wait-budget bump that treated the symptom; the deadlock is structural and a larger budget would not have helped). ### Screenshots Not applicable — no UI changes. ### Testing helm-unittest extended to pin: no `helm.sh/hook` on any of the seven talos-reconcile documents, the content-hash name, `ttlSecondsAfterFinished`, the preserved template apply + ownerReference + extraHostEntries, the hash rotating on a spec change, and the long-name truncation staying within 63 chars. Full chart suite green (128 tests). The applied template is byte-identical to the previous hook output (verified by diff), so existing tenants are upgrade-safe. ### Release note ```release-note fix(kubernetes): create the tenant worker TalosConfigTemplate from a main-phase Job instead of a post-install hook, so worker MachineDeployments reliably scale during install ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes addon install behavior by running reconcile work during the main chart phase, reducing readiness deadlocks and upgrade issues. * Added more reliable reconciliation handling with stable, change-detected job recreation and updated network access for the reconcile workload. * Refined chart and test guidance to match the new install flow and Helm behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
50748fed98 |
fix(kubernetes): make the default md0 node group removable (#2936)
Same-repo mirror of #2753 by @mattia-eleuteri, opened so the `Build` CI job can run. Fork PRs skip the OCIR registry login (`if: !github.event.pull_request.head.repo.fork` in `pull-requests.yaml`) and so can't push the per-PR images the build needs — meaning a fork PR can never pass `Build`, regardless of the change. The commits here are unchanged and authored by @mattia-eleuteri; full description and discussion in #2753. This includes migration **44** (renumbered from 43 after the seaweedfs-db migration landed on `main`) and bumps `targetVersion` to **45**. Supersedes #2753. ```release-note fix(kubernetes): the default `md0` node group is now applied only when no node groups are configured, so it can be removed or replaced. Migration 44 pins `md0` explicitly on existing clusters to preserve their topology. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Empty `nodeGroups` settings now automatically provision a default `md0` worker group. * Existing environments are updated to keep this default group available during upgrades. * **Bug Fixes** * User-defined node groups now fully replace the default instead of being merged with it. * Improved handling of node-group resources and permissions in Kubernetes app rendering. * **Documentation** * Updated Kubernetes app docs and schema details to reflect the new default behavior. * **Tests** * Added coverage for default and custom node-group configurations, plus migration behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
81198b294f |
fix(bucket): clean up orphaned cert-manager ACME solver resources on delete (#3060) (#3076)
## What this PR does After deleting a `Bucket` app CR, cert-manager's temporary ACME HTTP-01 solver resources (`cm-acme-http-solver-*` Ingress/Pod/Service) were left orphaned in the tenant namespace. They are created by cert-manager to answer an ACME challenge and are owned by the `Challenge`/`Order` → `Certificate`, not by the app Ingress. If a challenge gets stuck and the app is deleted, the parent Ingress is removed but the orphaned challenge leaves the solver objects with no live owner to cascade-delete them, so Helm/Flux uninstall cannot reap them. This adds a `post-delete` cleanup hook that deletes the release's `Certificate`/`Order`/`Challenge` (cert-manager then cascade-removes the solver objects) and then sweeps any leftover solver objects by the reliable `acme.cert-manager.io/http01-solver=true` label. RBAC is verified against the vendored cert-manager CRDs (`cert-manager.io` → certificates; `acme.cert-manager.io` → orders, challenges) plus core `services`/`pods` and `networking.k8s.io` ingresses. The cleanup Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`. Failures are logged but never block teardown. The preventive wildcard-cert guard the issue mentions already exists in the vendored `bucket-system` Ingress template (it drops the ACME annotations in wildcard mode), so no preventive change is needed — only cleanup. No user data is affected: only cert-manager's transient challenge artifacts are removed. Fixes #3060. ### Release note ```release-note fix(bucket): clean up orphaned cert-manager ACME http01-solver resources (Ingress/Pod/Service) left behind after a Bucket app is deleted. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a bucket app `test` command that runs Helm unit tests. * Introduced a post-delete ACME cleanup hook to remove release-scoped cert-manager TLS and HTTP-01 solver resources, using dedicated least-privilege access controls. * **Bug Fixes** * Improved cleanup safety with bounded execution and hardened container security. * **Tests** * Added a Helm test suite covering rendered hook documents, including host-scoped targeting and negative assertions to prevent overly broad deletions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
544ba5bf16 |
fix(harbor): preserve jobservice and trivy storageClass on upgrade (#2930)
## What this PR does The `apps/harbor` chart never passed `storageClass` to the jobservice `jobLog` PVC, so the upstream chart rendered it without a `storageClassName`. On first install the default-StorageClass admission then stamps the cluster default onto the live PVC. Because a PVC's `storageClassName` is immutable, the rendered manifest (no class) and the live object (admission-defaulted class) drift apart, which surfaced as upgrade failures on the `harbor-*-system` HelmRelease. This propagates `storageClass` to the jobservice PVC, with one safeguard: a `lookup` reads the live jobservice PVC and reuses its current `storageClassName` when the PVC already exists, only falling back to the configured `storageClass` for a fresh install. Keeping the rendered value equal to the live one means an existing PVC whose admission-defaulted class differs from the configured `storageClass` is preserved rather than triggering an immutable-field rejection on upgrade. A live PVC pinned to an explicit `""` (no dynamic provisioning) round-trips through the upstream `"-"` sentinel instead of collapsing to an omitted field, and the spec access is guarded so a PVC with no `storageClassName` key stays omitted. The trivy `StatefulSet`'s `volumeClaimTemplate` carried the same latent drift — `.Values.storageClass` was forwarded blindly into a VCT, which cannot be patched at all — so it now gets the identical lookup-preserve treatment, keyed off `data-<release>-trivy-0`. This closes both halves of #2368. Also adds a helm-unittest regression suite and wires a `test:` target so `make unit-tests` stops skipping `packages/apps/harbor`. Validated with `helm template` across storageClass/trivy permutations and the new unit tests. The jobservice preserve path was additionally confirmed with a server-side dry-run against a running harbor release: `lookup` resolved the live PVC's class, and with `storageClass` overridden to a different class the jobservice render stayed pinned to the live PVC's class — confirming the immutable-field rejection is avoided. The trivy change applies the same proven pattern. Note: under `helm template`/helm-unittest `lookup` returns empty, so the unit suite exercises only the fresh-install fallback; the preserve branches are covered by the live `lookup`. Fixes #2368 ### Release note ```release-note fix(harbor): preserve the jobservice PVC and trivy volumeClaimTemplate storageClass on upgrade, fixing upgrade failures caused by storageClassName drift ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Harbor now preserves the existing jobservice storage class during upgrades and ensures it’s applied on initial installation when available, with consistent handling for Trivy persistence as well. * **Tests** * Added a Helm unittest regression suite covering storage class propagation and the correct conditional rendering of persistence blocks when storage class and Trivy settings vary. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
c818ece25b |
test(kubernetes): cover md0 removability with user-defined nodeGroups
The existing case pins the empty-nodeGroups -> default md0 path (the helper's else-branch). This adds the complementary if-branch: when a user supplies their own nodeGroups, they are authoritative and the built-in md0 is NOT merged in, so md0 is removable. The assertion selects every rendered MachineDeployment and requires it to be the user's own group, so a regressed md0 leak surfaces as a second MachineDeployment named <release>-md0 that fails the assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
ab55ff1924 |
test(kubernetes): cover the default md0 node group render
Assert that an empty nodeGroups renders the built-in md0 group via the kubernetes.nodeGroups helper else-branch (the behaviour that makes md0 removable): the default md0 MachineDeployment is emitted and the default 20Gi system disk flows through. A minimal VirtualMachineClusterInstancetype plus dummy CAPI objects are registered so the per-group instanceType lookup and the old-template preservation LISTs resolve under helm-unittest. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
7e3c5f99db |
fix(kubernetes): apply default md0 node group only when nodeGroups is empty
The chart shipped a default nodeGroups.md0 in values.yaml. Helm deep-merges
values, so the default md0 was re-added on top of any user-supplied
nodeGroups; combined with Kubernetes stripping null values, md0 could never
be removed by configuration. Move the default into the template
("kubernetes.nodeGroups" helper) so it is applied only when nodeGroups is
empty. User-supplied node groups now fully replace the default, letting users
name and omit groups (including md0) freely.
Test fixtures that relied on the chart-default md0 to supply diskSize via
values merge now set it explicitly.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
|
||
|
|
32c4d60b43 |
fix(kubernetes): run the Talos reconcile Job in the main install phase
The talos-reconcile Job is the only thing that creates the TalosConfigTemplate the worker MachineDeployment's bootstrap.configRef points to; CAPI blocks MachineSet creation until it exists, so tenant workers cannot scale until the Job has run. The Job was a post-install/post-upgrade Helm hook, which runs only after Helm has applied the release's main resources and, when Helm waits, after they become Ready. But the same release emits in-tenant addon HelmReleases — notably cilium — whose readiness needs worker nodes, and those nodes need the TalosConfigTemplate this Job produces. So when the main-resource wait is in effect the install deadlocks: it waits for cilium to be Ready, cilium waits for workers, the workers wait for the template, and the template waits for this hook — which never runs because the install never finishes the wait that precedes the hook phase. The install times out and the MachineDeployment never scales. The release sets helm-install-disable-wait to skip that wait, but its effect is unreliable, which makes the failure bimodal: the template lands in ~46s when the wait is skipped, and the install is stranded when it is not. Render the Job and its RBAC/ServiceAccount/egress policy as ordinary main-phase resources. The Job is then created during the apply itself, not after the wait, and runs on the management cluster where nodes exist (unlike the tenant cilium-operator), so it produces the template, the MachineDeployment scales, workers join, cilium goes Ready, and the wait (if any) closes within the install budget — independent of whether helm-install-disable-wait takes effect. The Job is immutable, so it carries a content-hash name suffix over its rendered spec (mirrors the KubevirtMachineTemplate idiom in cluster.yaml): an identical render keeps the name and Helm no-ops it, while any spec change yields a new name and a fresh Job, so no spec change ever lands on the same name and triggers an immutable-field patch error. The applied TalosConfigTemplate keeps its stable <release>-<group> name, so an unchanged render re-applies a byte-identical template — a CAPI no-op that never rolls existing workers. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
2587458bd4 |
fix(qdrant)!: reclaim data PVC by name prefix, not volumeClaimTemplate label
The previous approach routed a release-unique label through qdrant.persistence.additionalLabels, which the upstream chart renders into the StatefulSet's volumeClaimTemplates[].metadata.labels. That field is immutable on an existing StatefulSet, so for any already-deployed Qdrant the API server would reject the upgrade and the <release>-system HelmRelease (retries: -1) would retry-fail forever — permanently wedging reconciliation. Label changes on a volumeClaimTemplate also never apply to PVCs that already exist, so the cleanup hook could not reclaim those instances anyway. Switch the post-delete hook to select PVCs by the deterministic StatefulSet naming prefix (qdrant-storage-<release>-) instead. PVC names derive purely from the release name, so this needs no StatefulSet change (upgrade-safe) and also reclaims PVCs from instances created before the hook existed. - qdrant.yaml: drop the additionalLabels passthrough entirely. - hooks/cleanup-pvc.yaml: delete by name prefix via kubectl get -o name | grep. - tests: assert the name-prefix command instead of the label passthrough. - e2e: assert qdrant-storage-<name>-0 is gone after delete (async-waited). BREAKING CHANGE: deleting a Qdrant app now permanently removes its data PVC and the vector data it holds. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
6a43adbd09 |
fix(bucket): drop no-op label-based order/challenge delete
Per maintainer review: the label selector cert-manager.io/certificate-name=<release>-ui-tls matches nothing — Challenge objects carry no labels, and on Orders certificate-name is an annotation (not a label). Orders are already reaped by the Certificate owner cascade and Challenges by the host-scoped .spec.dnsName sweep, so the line deleted nothing and its comment misstated cert-manager labeling. Remove the dead delete, replace the inaccurate comment, and drop the test assertion that pinned it; add negative assertions so a label-based order/challenge delete cannot be reintroduced. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
4db565282b |
fix(postgres): garbage-collect init-job after completion (#3055) (#3069)
## Problem After deleting a `Postgres` app CR, the chart-managed init Job `<release>-init-job` and its completed Pod remain in the tenant namespace indefinitely (reported in #3055). The init Job is a Helm hook (`post-install,post-upgrade`) whose only delete policy is `before-hook-creation`. That policy deletes the previous hook only right before a new one is created (next upgrade) — it never fires on `helm uninstall`. With no `hook-succeeded` and no TTL, the completed Job and Pod leak. ## Root cause `packages/apps/postgres/templates/init-job.yaml` — the Job has no garbage-collection mechanism for its completed state. ## Fix Add `ttlSecondsAfterFinished: 120` to the Job spec. The native Kubernetes TTL controller then GCs the Job and its Pod ~2 minutes after completion, regardless of the Helm hook lifecycle. The init Job is a one-shot, idempotent reconciler of users/databases/roles (`init-script.yaml`) — once it has run it is not needed until the next upgrade (when Helm recreates it), so GC-ing the finished Job is safe. This approach deliberately **does not** touch the hook-delete-policy / hook-timeout semantics, which the `bootstrap.recovery` path depends on (see the header comment in `init-job.yaml`). Under `bootstrap.enabled` the Job is skipped entirely, so the change is inert there. ## Tests Added a `helm-unittest` suite `packages/apps/postgres/tests/init_job_cleanup_test.yaml`: - asserts the init Job renders `spec.ttlSecondsAfterFinished: 120` - asserts the init Job is skipped under `bootstrap.enabled: true` Verified locally with `helm template` (TTL present on default render; Job absent under bootstrap recovery). ## Test plan - [x] `helm template` renders `ttlSecondsAfterFinished: 120` on the init Job - [ ] `helm unittest packages/apps/postgres` green in CI - [ ] e2e `test-apps-postgres` green Fixes #3055 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Postgres bootstrap init Jobs now automatically clean up after completion using a post-finish TTL. * **Tests** * Added a new integration test suite to verify the init Job renders with the expected cleanup TTL, and that the init Job is skipped when bootstrap is enabled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
49555e562b |
fix(openbao): garbage-collect the data PVC on delete (#3058) (#3073)
## What this PR does
After deleting an `OpenBAO` app CR, the StatefulSet data PVC
(`data-<release>-0`)
was left orphaned in the tenant namespace, consuming storage
permanently. The
OpenBAO app renders a Flux `HelmRelease` over the vendored upstream
OpenBAO chart,
and StatefulSet-templated PVCs are never garbage-collected on
StatefulSet deletion
by default — the chart set no `persistentVolumeClaimRetentionPolicy`.
This passes a retention policy through the HelmRelease values. The
upstream chart
renders it onto the StatefulSet from
`server.persistentVolumeClaimRetentionPolicy`
(verified in the vendored `server-statefulset.yaml`) — **not** from
`server.dataStorage.*`:
```yaml
server:
persistentVolumeClaimRetentionPolicy:
whenDeleted: Delete
whenScaled: Retain
```
A `test` target is added to the package Makefile so the new
helm-unittest suite
(asserting the passthrough) actually runs in CI —
`hack/helm-unit-tests.sh` only
runs packages that define a `test` target.
> [!WARNING]
> **Deleting an OpenBAO app now permanently removes its data PVC and the
data it holds.**
> This is intentional and fixes the storage leak. `whenScaled: Retain`
is kept on
> purpose so scaling an HA/raft cluster down does **not** destroy a
member's data —
> only full deletion of the release reclaims storage. Back up before
deleting.
Fixes #3058.
### Release note
```release-note
fix(openbao): set persistentVolumeClaimRetentionPolicy so the data PVC is removed when an OpenBAO app is deleted, instead of being orphaned. WARNING: deleting an OpenBAO app now permanently removes its data PVC and the data it holds (scale-down still retains data).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable PVC retention behavior for OpenBao: storage is
deleted when the release is removed and retained when scaling down.
* **Tests**
* Added a Helm unit testing workflow to run chart/unit tests.
* Introduced assertions to confirm PVC retention settings are correctly
passed through and rendered as expected.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
959a9b1fdb |
fix(kubernetes): clear kamaji datastore-secret finalizer to unblock namespace deletion (#3062) (#3078)
## What this PR does After deleting a `Kubernetes` app CR / tenant, the secret `<release>-datastore-config` kept the finalizer `finalizer.kamaji.clastix.io/datastore-secret`, which was never removed — **blocking deletion of the entire namespace** (observed stuck in `Terminating` for 115 minutes). Kamaji adds this finalizer to the datastore connection Secret and normally strips it as part of TenantControlPlane (TCP) deletion; if the TCP is force-deleted or Kamaji does not reconcile in time, the finalizer hangs and the Secret can never be garbage-collected, wedging the namespace. This extends the existing `pre-delete` cleanup hook (`packages/apps/kubernetes/templates/delete.yaml`): - **Step 4** now deletes the TenantControlPlane with `--wait=true --timeout=120s` so Kamaji's own finalizer-cleanup chain has a chance to run before we fall through. - **Step 4b (backstop)** patches `metadata.finalizers: null` on `<release>-datastore-config` only if it still exists. By this point the TCP is gone, the Secret is owner-ref'd to the deleted TCP and only awaiting GC, so clearing the finalizer merely lets GC reap it — it cannot strip a finalizer while a live TCP still needs the datastore. The Secret holds only connection config, not tenant data (which lives in the external etcd datastore), so no data is lost. - **RBAC** adds a `secrets` rule scoped via `resourceNames` to exactly `<release>-datastore-config`, verbs `get`/`patch` only. The Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`, writable `/tmp` for kubectl cache. Failures at both steps are logged (the Step 4b warning explicitly mentions the namespace may stay Terminating) but the hook still completes so it cannot itself block teardown. This is the most impactful of the create→delete leak sweep findings: without it, tenant teardown hangs and the create→delete cycle requires manual intervention. Fixes #3062. ### Release note ```release-note fix(kubernetes): clear the leftover Kamaji `finalizer.kamaji.clastix.io/datastore-secret` finalizer on the datastore-config Secret during tenant teardown, so deleting a Kubernetes app / tenant no longer leaves the namespace stuck in Terminating. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Extended the pre-delete cleanup workflow to more reliably remove `TenantControlPlane` and handle slow deletions with a bounded wait. * **Security** * Hardened the cleanup job by running as non-root with restricted security settings, dropped capabilities, and a safe writable temp location. * **Bug Fixes** * Added a conditional backstop to remove leftover `finalizers` from the relevant datastore Secret so namespace termination isn’t blocked. * **Tests** * Added a YAML test suite to validate the wait-and-patch behavior and ensure the patch runs only after the `TenantControlPlane` is gone. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
bdf520bf74 |
fix(harbor): preserve jobservice and trivy storageClass on upgrade
The cozy-harbor wrapper never passed storageClass to the jobservice jobLog PVC, so it was created without a storageClassName and the default-StorageClass admission stamped the cluster default. Because PVC storageClassName is immutable, this caused rendered-vs-live drift on upgrade. Preserve an existing PVC's class via lookup and fall back to the configured storageClass for fresh installs. The trivy StatefulSet's volumeClaimTemplate had the same latent drift -- .Values.storageClass was forwarded blindly into a VCT that cannot be patched at all -- so it gets the same lookup-preserve treatment, keyed off data-<release>-trivy-0. A live object pinned to an explicit "" (no dynamic provisioning) now round-trips through upstream's "-" sentinel instead of collapsing to an omitted storageClassName, and the spec lookup is guarded with `| default dict` so a PVC carrying no storageClassName key stays omitted. Adds a helm-unittest regression suite and wires a test target so make unit-tests stops skipping packages/apps/harbor. Fixes #2368 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
243ce65071 |
feat(kubernetes): expose tenant apiserver authentication in app values
Tenant Kubernetes clusters had no way to configure the kube-apiserver authentication through the app values: no knob for OIDC, the structured AuthenticationConfiguration, extra apiserver flags, or extra volumes. The only workaround was patching the underlying KamajiControlPlane out-of-band, which is invisible to GitOps and lost on recreation. Add three opt-in fields under controlPlane.apiServer, all empty by default so existing clusters render unchanged: - extraArgs -> KamajiControlPlane spec.apiServer.extraArgs - extraVolumeMounts -> spec.apiServer.extraVolumeMounts - extraVolumes -> appended to spec.deployment.extraVolumes This unblocks OIDC SSO via the --oidc-* flags and, more importantly, structured authentication: --authentication-config mounted from a ConfigMap/Secret plus --requestheader-uid-headers, which is the only way to map a user UID for aggregated/extension apiservers. Because the control-plane Deployment runs on the management cluster, the passthrough is validated to preserve tenant<->management isolation: - extraVolumes allow only configMap and secret sources; host-reaching sources (hostPath, csi, nfs, persistentVolumeClaim) and token projection (projected) are rejected, so a tenant with self-service access to the Kubernetes CR cannot mount management-node paths or project arbitrary tokens into the apiserver pod. - each extraVolume must have a unique, non-empty name and exactly one source; the reserved names talos-ca and talos-tls-cert are rejected to avoid a duplicate volume name in the rendered podspec. - each extraVolumeMount must reference a user-declared extraVolume; the chart-managed talos secret volumes cannot be mounted. Regenerate values.schema.json, README, the Go types and deepcopy, and the ApplicationDefinition. Add helm unittest coverage for the default (no-op) render, each field's passthrough, the name/source guards, and the mount-reference guard (valid and rejected inputs). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
6423cb53e0 |
fix(kubernetes): never auto-select the kubevirt alias as tenant default SC
The legacy "kubevirt" StorageClass alias was inserted into the candidate map before the default was chosen, so the sortAlpha fallback could pick it when propagated class names sort after "kubevirt" (e.g. nvme, ssd) and no configured/replicated class is present. That made the tenant default point at an infraStorageClass that may not exist on the infra cluster, leaving default PVCs Pending. Choose the default from the propagated set before adding the alias. Also quote the templated StorageClass name and infraStorageClassName in kubevirt-csi-node, and document the propagation upgrade caveat, render-time evaluation limitation, and allowRemoteVolumeAccess assumption in the kubernetes README. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
6a1b56f2b5 |
test(kubernetes): cover LINSTOR StorageClass propagation
- csi_test.yaml: assert the fallback storageClasses map yields a default "replicated" plus the non-default "kubevirt" alias, and that the alias tracks the storageClass selector. - kubevirt-csi-node: add a Makefile test target (otherwise the package is silently skipped by the unit-test runner) and storageclasses_test covering per-entry rendering, the single-default invariant, and the absence of any enforcement allowList in driver-config. - run-kubernetes.sh: verify in e2e that "replicated" propagates as the default kubevirt-CSI class, the "kubevirt" alias is retained, and the node-local "local" infra class is filtered out. Refs: #2094 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
021f6141ca |
feat(kubernetes): propagate remote-accessible LINSTOR StorageClasses to tenant clusters
Tenant Kubernetes clusters previously exposed a single hardcoded "kubevirt" StorageClass mapped to one infra class. Discover all remote-accessible LINSTOR infra StorageClasses (provisioner linstor.csi.linbit.com, allowRemoteVolumeAccess != false) via Helm lookup and propagate each to the tenant under the same name. - csi.yaml builds a storageClasses map filtered to remote LINSTOR classes; node-local classes (e.g. "local") are excluded because the backing volume cannot follow a live-migrated worker VM. - The class named by .Values.storageClass (default "replicated") is marked the tenant default; the legacy "kubevirt" StorageClass is kept as a non-default alias so existing tenant PVCs keep binding. - A non-destructive fallback preserves the legacy mapping when the lookup returns empty (dry-run), so tenant StorageClasses are never wiped on a transient reconcile. - infraStorageClassEnforcement is intentionally left disabled (AllowAll) to avoid breaking tenants that already self-served custom StorageClasses under the current default. Refs: #2094, #1424 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
1df8c7c557 |
feat(kubernetes): bootstrap tenant workers with Talos via CABPT (#2931)
## What this PR does > **Supersedes #2610.** Same work, rebased cleanly onto current `main` so it now sits on top of #2867 (the `kube-controller-manager` VAP type-checker fix). That fix is what was killing the base install in #2610's E2E *before* the `kubernetes` app test ever ran, leaving the Talos worker-bootstrap rewrite verified only by code reading. #2610's 67-commit branch is squashed here into 8 logical commits with original authorship preserved (@kvaps and @IvanHunters as authors); the tree is **byte-identical** to #2610's head — no code changes, just the rebase onto `main` (to pick up #2867) plus history cleanup. The detailed review history lives on #2610. Phase 1 of the Kubernetes-app split design (see cozystack/community#8): replace the Ubuntu+kubeadm worker bootstrap path with Talos, driven by `cluster-api-bootstrap-provider-talos` (CABPT) and a `clastix/talos-csr-signer` sidecar embedded in the Kamaji control-plane pod. Existing tenants keep working — old machines roll out and get replaced by Talos workers without manual intervention. Highlights: - Add CABPT (v0.6.12) as a second `BootstrapProvider` alongside the existing kubeadm one in `capi-providers-bootstrap`. - Bump the Kamaji control-plane provider with an upstream-bound patch that exposes `KamajiControlPlane.spec.network.additionalServicePorts`, used to publish trustd (50001/TCP) on the apiserver service. - Generate Talos PKI (Ed25519 CA + trustd TLS) via cert-manager and stable random Talos secrets (`token`, `clusterId`, `clusterSecret`, `bootstrapToken`) using the helm lookup-and-reuse pattern. - Materialise a kubeadm-format `bootstrap-token-<id>` Secret inside the tenant `kube-system` via a Helm post-install/upgrade Job. - Render a `TalosConfigTemplate` (worker machineconfig, `generateType: none`) and switch the `MachineDeployment.bootstrap` reference from `KubeadmConfigTemplate` to `TalosConfigTemplate`, gated on Talos secrets and the Kamaji apiserver service being ready. - Boot workers from the Talos openstack image via a CDI `DataVolume` (`source.http.url` → `factory.talos.dev`), expose the system disk as virtio-blk with `blockSize.custom: logical=512, physical=4096` so 4Ki-native block storage backends (LINSTOR/DRBD) play nicely with QEMU's O_DIRECT writes and SeaBIOS still boots, drop the separate kubelet disk (Talos lays out EPHEMERAL itself). - Pin pod/service CIDRs in the worker machineconfig to the tenant ranges (`10.243.0.0/16` / `10.95.0.0/16`) to avoid Talos's address-overlap diagnostic against the host pod CIDR. - Set `cluster.controlPlane.endpoint` and `cilium.k8sServiceHost` to the Kamaji apiserver Service ClusterIP (looked up at render time), so bootstrap survives the chicken-and-egg moment where tenant DNS does not exist yet and the host CoreDNS does not serve the tenant zone. End-to-end on a dev cluster: a fresh tenant brings up a Talos worker, kubelet CSRs get approved (apiserver-client by Kamaji, kubelet-serving by the talos-csr-signer sidecar), Cilium initialises with hostNetwork against the apiserver ClusterIP, CoreDNS comes up, and the node transitions to `Ready`. ### Release note ```release-note feat(kubernetes): bootstrap tenant workers with Talos Linux instead of Ubuntu+kubeadm. Existing clusters roll over to Talos workers automatically; new tenants come up on Talos from the start. Powered by cluster-api-bootstrap-provider-talos and the talos-csr-signer sidecar in the Kamaji control-plane pod. Note: the tenant `kubernetes` HelmRelease now reports Ready as soon as Helm completes (Install/Upgrade DisableWait) and no longer blocks on worker/addon readiness — worker-rollout health is tracked by WorkloadMonitor, not the HelmRelease. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes **New Features** * Added Talos worker configuration for `talos.version` and `talos.schematicID` * Introduced `nodeHealthCheck` (`maxUnhealthy`, `nodeStartupTimeout`) for worker remediation tuning * Added image overrides for `kubectl` and `talosCsrSigner` * Exposed additional tenant control-plane Service ports support **Breaking Changes** * Removed Kubernetes v1.30; supported versions start at v1.31 * Renamed `nodeGroups.ephemeralStorage` → `nodeGroups.diskSize` with consolidated Talos system-disk semantics **Removals** * Removed Ubuntu container-disk images and related containerd patch mechanisms * Air-gapped tenant workers temporarily unsupported during the Talos transition **Documentation** * Updated chart docs, parameters, and GitOps upgrade guidance for version bumps <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
26dc006f39 |
chore(qdrant): drop unrelated .PHONY: test from Makefile
The .PHONY: test tweak is out of scope for the PVC cleanup fix; remove it to keep this PR focused (flagged by review). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
58b2a3d779 |
fix(bucket): harden ACME cleanup hook and fix cert-manager label selector
Address review feedback on the post-delete cleanup hook: - Target Orders/Challenges by cert-manager's own cert-manager.io/certificate-name=<release>-ui-tls label instead of the Helm app.kubernetes.io/instance label, which cert-manager never sets on those resources (the old selector was a silent no-op). - Bound the Job with activeDeadlineSeconds=120 and backoffLimit=0 so a stuck finalizer or hung API call cannot block the uninstall. - Add --wait=false --request-timeout=30s to every kubectl delete/get so the hook never hangs on slow or finalizer-blocked deletes. - Surface list failures in the job log instead of redirecting to /dev/null. - Tests: assert the bounded execution and security-hardening contract, pin the rendered document count to 4, and update selector assertions. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
4b9d67f324 |
fix(kubernetes): clear kamaji datastore-secret finalizer to unblock namespace deletion (#3062)
After deleting a Kubernetes app/tenant, the <release>-datastore-config Secret kept the finalizer finalizer.kamaji.clastix.io/datastore-secret, blocking deletion of the whole namespace. Extend the pre-delete hook to wait for the TenantControlPlane to be removed (so Kamaji can strip the finalizer itself), then, as a backstop, clear the finalizer on the datastore-config Secret if it still lingers. RBAC is scoped via resourceNames to that one Secret (get/patch). The Job runs non-root with a read-only root fs, dropped capabilities and a writable /tmp for the kubectl cache; failures are logged. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
|
d113273aa9 |
fix(harbor): clean up orphaned jobservice PVC and ACME solver resources on delete (#3061)
The jobservice PVC (resourcePolicy: keep) and cert-manager ACME http01-solver objects were left orphaned after a Harbor app CR was deleted. Add a post-delete hook that deletes the release PVCs (selecting by app.kubernetes.io/instance=<release>-system, the label the upstream subchart stamps) and the release Certificate/Order/Challenge plus the http01-solver sweep. A hook is required because resourcePolicy: keep blocks normal reclaim. The cleanup Job runs non-root with a read-only root fs and dropped capabilities; failures are logged but never block teardown. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
|
4425d3afa7 |
fix(bucket): clean up orphaned cert-manager ACME solver resources on delete (#3060)
cert-manager ACME http01-solver Ingress/Pod/Service objects were left orphaned after a Bucket app CR was deleted (a stuck challenge leaves them without a live owner to cascade-delete). Add a post-delete hook that deletes the release Certificate/Order/Challenge and sweeps leftover solver objects by the acme.cert-manager.io/http01-solver label. RBAC verified against the vendored cert-manager CRDs. The cleanup Job runs non-root with a read-only root fs and dropped capabilities; failures are logged but never block teardown. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
|
cfd9ae30d6 |
fix(mariadb): bind metrics exporter to IPv4 for probe reachability (#3064)
## What this PR does The `mariadb` chart enables a Prometheus `mysqld-exporter` sidecar. The exporter binds the IPv6 wildcard `[::]:9104` by default, while mariadb-operator hardcodes the exporter's liveness/readiness probe as an `httpGet` on port 9104 that kubelet dials over the pod's IPv4 address. On a single-stack-IPv4 data plane the `[::]` listener intermittently refuses the IPv4 connect, so the probe fails roughly one second after the container starts, the pod flaps to NotReady, and the metrics Service keeps the pod IP in `notReadyAddresses` — its endpoint set never populates and Prometheus never scrapes the exporter. This pins the exporter to the IPv4 wildcard via the only chart-side lever the operator exposes for this — `spec.metrics.exporter.args`. The operator appends user-supplied args to its defaults (it does not set `--web.listen-address` itself), so there is no duplicate flag and the DB connection arg is preserved. A `0.0.0.0:9104` socket accepts the kubelet probe deterministically, independent of the exact IPv6 `v6only` submechanism. Conscious tradeoff: this makes the exporter IPv4-only, which is correct for cozystack's single-stack-IPv4 data plane (Cilium runs with `ipv6.enabled=false`). A future IPv6-primary deployment of this app would need `[::]` or both addresses (the exporter's `--web.listen-address` is repeatable in exporter-toolkit ≥ v0.10.0). Adds a helm-unittest pinning the rendered `args`, plus the `test:` Makefile target so CI's chart-test discovery runs it (the chart previously shipped no tests). ### Release note ```release-note fix(mariadb): bind the metrics exporter to IPv4 so its readiness probe is reachable and the metrics endpoint populates reliably ``` |
||
|
|
178b8360a3 |
fix(qdrant): clean up orphaned data PVC on delete (#3059)
The StatefulSet data PVC (qdrant-storage-<release>-0) was left orphaned after a Qdrant app CR was deleted. The vendored upstream chart does not support persistentVolumeClaimRetentionPolicy, so add a post-delete PVC cleanup hook (modeled on the MariaDB hook) and pass persistence.additionalLabels through the HelmRelease so the data PVC carries a release-unique app.kubernetes.io/instance label the hook selects on. The cleanup Job runs non-root with a read-only root fs and dropped capabilities; failures are logged but never block namespace teardown. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
|
7ef2b2f40f |
fix(mariadb): clean up orphaned operator-generated secrets (#3056)
The mariadb-operator creates <release>-metrics-password and <release>-repl-password Secrets as a side effect of metrics/replication being enabled. These are not Helm-owned and survive `helm uninstall`, leaving orphaned Secrets in the tenant namespace after the app CR is deleted. Extend the existing post-delete cleanup hook to sweep them: grant the cleanup Role RBAC on secrets and delete both password Secrets by name. Add a helm-unittest suite locking in the behaviour (and a `test` Makefile target so the package's tests run in CI). Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
|
9dfb078cbb |
fix(openbao): garbage-collect the data PVC on delete (#3058)
After deleting an OpenBAO app CR, the StatefulSet data PVC (data-<release>-0) was left orphaned in the tenant namespace, consuming storage permanently. StatefulSet-templated PVCs are never garbage-collected on StatefulSet deletion by default, and the chart set no persistentVolumeClaimRetentionPolicy. Pass a retention policy through the HelmRelease values. The vendored upstream chart renders it onto the StatefulSet from server.persistentVolumeClaimRetentionPolicy (not server.dataStorage.*): whenDeleted: Delete reclaims the data PVC on release deletion, while whenScaled: Retain keeps a member's data PVC on HA scale-down. Also add a 'test' Makefile target so the package's helm-unittest suite runs in CI (hack/helm-unit-tests.sh only runs packages that define a 'test' target). Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
|
af1cc790a1 |
fix unittest assetions
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
cee4227da2 |
fix(postgres): bump init-job TTL to 1h and fix unittest include
Address review feedback on #3069: - ttlSecondsAfterFinished 120 -> 3600 so a failed init Job leaves a 1h debugging window before the TTL controller collects it (the TTL also reaps failed Jobs, not just successful ones). - Load templates/init-script.yaml in the unittest suite: the Job's checksum/config annotation includes that template, so rendering init-job.yaml in isolation failed in CI. Select the Job via documentSelector and assert the 3600 TTL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
716cd32f43 |
chore(postgres): remove inline comment from init-job ttl
Drop the explanatory comment added alongside ttlSecondsAfterFinished; the rationale lives in the PR/commit history. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
3c7efb9acd |
fix(postgres): garbage-collect init-job after completion (#3055)
The post-install/post-upgrade init Job is a one-shot reconciler of users/databases/roles. Its hook-delete-policy was `before-hook-creation` only, which never fires on uninstall, so the completed Job and its Pod lingered in the tenant namespace forever after the Postgres CR was deleted (#3055). Add `ttlSecondsAfterFinished: 120` so the Kubernetes TTL controller GCs the Job and its Pod ~2 minutes after completion, regardless of the Helm hook lifecycle. This deliberately avoids touching the hook-timeout semantics that the bootstrap.recovery path depends on. Add a helm-unittest suite locking in the TTL and the bootstrap-skip behaviour to prevent regression. Fixes #3055 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
393b50de72 |
fix(tenant): grant watch verb to cleanup Role so pre-delete hook can wait (#2923)
## What this PR does Two fixes to the tenant chart's `pre-delete` cleanup hook (the Job that deletes a tenant's HelmReleases when the tenant is removed). **1. Grant the `watch` verb so the hook stops hanging.** The hook runs `kubectl delete helmreleases ... --wait=true`. Since Kubernetes 1.25 that flag opens a watch on the target resources to block until finalization. The hook's Role granted only `get`, `list`, `delete`, so the watch was rejected with HTTP 403, `kubectl delete` hung indefinitely, the Job never finished, and tenant deletion stalled on the pre-delete hook. Adding `watch` brings the Role in line with the `kubernetes` app chart's cleanup Role, which already grants it for the same `--wait` deletes. **2. Replace the unmaintained image.** The Job ran kubectl from `bitnami/kubectl:latest`. The bitnami Docker Hub catalog has moved to a legacy/sunset state, so that pull is at risk of breaking — which would stop the hook from starting at all — and `:latest` leaves the kubectl version unpinned. It now uses `docker.io/clastix/kubectl:v1.32`, the version-pinned image already used by every other first-party cleanup/delete hook in the repo (kubernetes, keycloak-configure, dashboard, mariadb). That image is Alpine-based and ships only `/bin/sh`, so the hook script (pure POSIX) now invokes `/bin/sh` instead of `/bin/bash`. A helm-unittest suite pins all three invariants (watch verb, image, shell) so none can regress. ### Screenshots N/A — no UI impact. ### Release note ```release-note fix(tenant): grant the watch verb to the cleanup Role and move it to a maintained kubectl image so tenant deletion no longer hangs on the pre-delete hook ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Expanded pre-delete cleanup RBAC permissions to include watching HelmRelease resources. * Hardened the cleanup Job runtime security settings for stronger container confinement. * Pinned the cleanup Job to a specific kubectl image version and switched the command interpreter for safer execution. * **Tests** * Added/extended automated validation for the pre-delete cleanup hook, covering RBAC rules, kubectl image pinning, command usage, and securityContext hardening. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
baf47ce638 |
chore(platform): raise Go toolchain to 1.26 across first-party images (#2941)
## What this PR does Completes the Go-toolchain modernization for first-party images — the build-time counterpart to #2852, which raised the root and `api/` modules to Go 1.26.4. 1. **First-party image Go modules → go 1.26.4** (`token-proxy`, `kube-ovn-webhook`, `kubevirt-csi-driver`): raise the `go` directive and refresh dependencies. `kube-ovn-webhook` moves `k8s.io/*` 0.26 → 0.36.2 (only long-stable admission/core API is used); `kubevirt-csi-driver` keeps its existing replace-based k8s alignment at 0.36.0 (main's refactor owns that module's pin) and raises the `go` directive, with the `golang.org/x/net` 0.56.0 security bump — which clears the reachable GO-2026-5026 — reapplied. Adds two `.gitignore` files so the locally-built binaries can't be committed. 2. **Go builder images → `golang:1.26`** across 8 images still on EOL/behind toolchains (golang 1.20, 1.20.6, 1.23.4, 1.24, 1.25): `cluster-autoscaler`, `kubevirt-cloud-provider`, `redis-operator`, `multus-cni`, `keycloak-operator`, `objectstorage`, `kilo`, `linstor-csi`. The builder image tag — not the go.mod minimum — sets the standard-library version baked into the binary, so these still shipped unpatched stdlib on a plain rebuild. Verified: `GOOS=linux go build`/`go vet` pass for all three modules and `go mod tidy` is a no-op; the three biggest builder jumps (redis-operator, kubevirt-cloud-provider, cluster-autoscaler — all formerly Go 1.20.x/1.23.x) were verified with a local `docker build`; the remaining five (1.24/1.25 → 1.26) are covered by CI image builds. ### Release note ```release-note chore(platform): raise the Go toolchain to 1.26 across first-party images to clear standard-library security advisories ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated several container build environments to a newer Go release for consistency across Kubernetes-related components. * Refreshed module and dependency versions in a few services to keep builds current. * Expanded ignore rules for generated image artifacts so build outputs aren’t tracked. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
c9464e018d |
fix(mariadb): bind metrics exporter to IPv4 for probe reachability
The mariadb-operator hardcodes the mysqld-exporter liveness and readiness probe as an httpGet on port 9104, which kubelet dials over the pod's IPv4 address. The exporter's default [::] (IPv6 wildcard) listener intermittently refuses the IPv4 connect on single-stack-IPv4 data planes, so readiness fails shortly after start, the pod flaps to NotReady, and the metrics Service keeps the pod in notReadyAddresses — its endpoint never populates. Pin the exporter to the IPv4 wildcard via --web.listen-address=0.0.0.0:9104 so the kubelet probe connects deterministically. The operator appends user args to its defaults and does not itself set --web.listen-address, so there is no duplicate-flag conflict. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
59ded0a223 |
fix(kafka): set entity-operator resources to avoid OOM under tenant LimitRange (#2934)
## What this PR does
`packages/apps/kafka` rendered the Strimzi `Kafka` CR with
`entityOperator.topicOperator: {}` / `userOperator: {}` and no
`resources`. Tenant namespaces ship a `LimitRange` that defaults
containers to `memory: 128Mi`, so the topic-operator and user-operator
JVM containers (Kafka 3.9.1 + fabric8/vertx) are **OOMKilled (exit
137)** on startup. The `<cluster>-entity-operator` Deployment never
becomes Ready (CrashLoopBackOff), so:
- `KafkaTopic` / `KafkaUser` CRs never reconcile — topic creation
effectively fails;
- the cluster-operator logs `Exceeded timeout of 300000ms while waiting
for Deployment ...-entity-operator ... to be ready`;
- on teardown, `KafkaTopic`s get stuck permanently with the
`strimzi.io/topic-operator` finalizer (no operator left to clear them),
blocking namespace deletion.
The Kafka broker + ZooKeeper come up fine — only the entity-operator
dies. The `apps.cozystack.io/Kafka` schema exposes no field for
entity-operator resources, so there is no user-side workaround.
## Fix
Set explicit `resources` (requests `256Mi`/`100m`, limit `512Mi` memory)
on `entityOperator.topicOperator` and `userOperator` so they no longer
inherit the 128Mi namespace default. Adds a `helm-unittest` assertion
for the rendered limits.
## Testing
Reproduced on a clean cluster (Cozystack v1.4.3 and v1.5.0-rc.2; Strimzi
0.45.1, Kafka 3.9.1): both entity-operator containers `OOMKilled` at
128Mi → CrashLoopBackOff → `KafkaTopic` never `READY`. After bumping to
512Mi the entity-operator rolls cleanly (both containers Ready, 0
restarts) and the `KafkaTopic` reconciles to `READY=True`.
### Release note
```release-note
fix(kafka): give entity-operator (topic/user-operator) explicit resources so they no longer inherit the 128Mi tenant LimitRange default and OOMKill, which left KafkaTopics/KafkaUsers unreconciled.
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Kafka topic and user operators now use explicit CPU and memory
requests and limits, avoiding reliance on namespace-level defaults.
* **Tests**
* Added regression coverage to verify the Kafka entity operator’s topic
and user operator memory limits are explicitly set (512Mi).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
a4c6a07b51 |
fix(kubernetes): harden Talos hook Jobs and correct Talos PKI cert
Add runAsNonRoot/runAsUser/seccompProfile to the bootstrap-token and talos-reconcile hook Jobs so they satisfy PodSecurity "restricted" (matching the csr-signer sidecar). Drop the keyEncipherment usage from the Ed25519 trustd Certificate (RFC 9295 - Ed25519 keys cannot do key encipherment). Correct the stale talos-pki comment that claimed the reconcile hook patches spec.ipAddresses: the cert carries DNS SANs only and workers route to trustd via the Service DNS name, as the reconcile job's own routing note explains. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
3478ed32b0 |
docs(kubernetes): note MHC remediation default flip and GitOps version-bump caveat
Document two behaviour changes from the Phase 1 Talos rollover in the package Breaking Changes: the worker MachineHealthCheck maxUnhealthy moved from a hard-coded 0 (remediation effectively off) to nodeHealthCheck.maxUnhealthy defaulting to "50%" (remediation enabled), and that GitOps-managed tenant Kubernetes CRs must bump spec.version to v1.31 in Git or migration 46's patch is reverted on the next source reconcile. Also fix the supported version range in the intro (1.31-1.35; v1.30 was removed from the support matrix). Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
d09d52e8dc |
chore(kubevirt-csi-driver): bump golang.org/x/net to v0.56.0
Clear the reachable GO-2026-5026 advisory: idna.ToASCII is reached through the KubeVirt client path, and the fix landed in x/net v0.55.0. Bump to v0.56.0, pulling x/sys, x/term and x/text forward as required; govulncheck then reports the module clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
9c5fbb69a5 |
chore(images): modernize Go builder images to 1.26
Several first-party images still built their Go binaries with EOL or behind-latest toolchains (golang 1.20, 1.20.6, 1.23.4, 1.24, 1.25), so a plain rebuild kept shipping standard-library versions that the current 1.26 toolchain fixes. Raise every Go builder image (FROM and the builder_image ARG defaults) to golang:1.26 across cluster-autoscaler, kubevirt-cloud-provider, redis-operator, multus-cni, keycloak-operator, objectstorage, kilo and linstor-csi. This is the build-time counterpart to raising the go directive in the first-party Go modules: the toolchain that compiles the binary, not the go.mod minimum, determines the standard-library version baked into the image. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |