mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 12:43:32 +09:00
adb307cbae0095bb30ca2c2e446f8d71042bca66
2919
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
adb307cbae |
[goldpinger] Set resources requests/limits on goldpinger DaemonSet
The Bloomberg goldpinger chart ships with resources: {}, which triggers
a 'no CPU limit' conformance warning on the goldpinger-daemon container
deployed on every node. Override the package values.yaml to set
limits 200m/256Mi and requests 50m/64Mi — comfortable margin for the
ping-mesh fan-out on larger clusters while keeping a small baseline
request.
Signed-off-by: Matthieu <matthieu@hidora.com>
|
||
|
|
1baf126b1a |
fix(tenant): inherit full ancestor label chain from parent namespace (#2912)
## Summary
`packages/apps/tenant/templates/namespace.yaml` labels every tenant
namespace
with `tenant.cozystack.io/<ancestor>` for each of its ancestors. These
labels
drive the `<ancestor>-egress` `CiliumClusterwideNetworkPolicy`
(`networkpolicy.yaml`): an ancestor tenant's pods may reach a namespace
only if
that namespace carries `tenant.cozystack.io/<ancestor>`.
The chain was derived by splitting `.Release.Namespace`, which only ever
reached
one level up. A direct child of `tenant-root` got `tenant-root` (its
parent
namespace *is* `tenant-root`), but a grandchild — `tenant-ktj-htdev`
created
under `tenant-ktj` — got only `tenant-ktj` and dropped `tenant-root`
entirely.
Without `tenant.cozystack.io/tenant-root` on a nested namespace, the
root
ingress controller cannot reach the proxy service in a nested
`kubernetes`
tenant, surfacing as `upstream timed out (110) while connecting to
upstream` in
the nginx logs.
## Fix
Derive the ancestor chain deterministically from the namespace name
instead of
from any cluster lookup. Every tenant descends from `tenant-root`, so
that label
is emitted unconditionally; the remaining ancestors are encoded in the
namespace
name itself, because `tenant.name` builds each child as
`<parent-namespace>-<word>` — so every progressive dash-prefix of the
name is a
real ancestor namespace (`tenant-ktj-htdev` yields `tenant-ktj` and
`tenant-ktj-htdev`). The logic lives in a new
`tenant.ancestorTenantLabels`
helper:
```yaml
labels:
{{- include "tenant.ancestorTenantLabels" $tenantName | nindent 4 }}
```
This needs no cluster state, renders identically offline (so it is
unit-testable), and converges on every reconcile regardless of the order
in
which parent and child `HelmRelease`s reconcile — unlike a
parent-namespace
`lookup`, which helm-controller does not re-render on and which is
ordering-dependent. The `ownerReference` `lookup` is kept but made
non-fatal:
during a real reconcile the parent namespace always exists (the tenant
`HelmRelease` and its `cozystack-values` Secret live in it), so the
`ownerReference` is always emitted; when the lookup is empty (offline /
unit
test) it is simply omitted so the labels still render.
Migration `49` backfills the labels on existing clusters so they heal on
upgrade
regardless of per-tenant `HelmRelease` state (e.g. suspended or failed
releases), and `migrations.targetVersion` is bumped to `50` so the
runner
iterates it. The migration derives the same chain from each namespace
name,
is idempotent (`kubectl label --overwrite`) and best-effort
(per-namespace
failures are tolerated so a transient apiserver error cannot abort the
upgrade).
## Test plan
- [x] `helm template packages/apps/tenant` at depth 1/2/3 — emits
exactly the
expected ancestor set at each depth: depth-1 `{tenant-root,
tenant-ktj}`,
depth-2 adds `tenant-ktj-htdev`, depth-3 adds `tenant-ktj-htdev-foo`. No
missing, no stray labels.
- [x] `helm unittest packages/apps/tenant` — new
`tests/namespace_ancestor_labels_test.yaml` pins the chain at depths
1/2/3
plus a negative assertion; whole suite green.
- [x] `bats hack/cozystack-version-stamp.bats` — migration `49` stamps
the
version through the shared `stamp_cozystack_version` helper (all 10
tests
pass, including the guard that migrations >= 42 must use the helper).
- [x] Migration `49` ancestor derivation verified under `/bin/sh` to
match the
helm output byte-for-byte, and dry-runs cleanly under `set -euo
pipefail`.
- [ ] Live cluster: create a 3-level tenant (`root -> ktj -> htdev`),
verify
`tenant-ktj-htdev` has all three `tenant.cozystack.io/*` labels.
- [ ] Live cluster: create a `kubernetes` tenant under `htdev` with
`addons.ingressNginx.hosts`, verify an external HTTP request reaches the
tenant ingress controller via the root-ingress proxy (no `upstream timed
out`).
## Closes
- Closes #2810
- Closes #2171
```release-note
fix(tenant): tenant namespaces now carry the full `tenant.cozystack.io/<ancestor>` label chain (including `tenant-root`) at any nesting depth, restoring root-ingress reachability to nested tenants; a migration backfills the labels on existing clusters
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved tenant namespace label inheritance across multi-level tenant
hierarchies, ensuring all ancestor tenant labels are consistently
applied.
* **New Features**
* Standardized ancestor label generation to reliably include the full
tenant chain for derived tenant namespaces.
* **Tests**
* Added regression coverage for ancestor label presence and absence at
multiple hierarchy depths.
* **Chores**
* Added a migration to backfill missing ancestor labels for existing
tenant namespaces during platform upgrades.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
1e5f53881f |
fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) (#3146)
## What Backports the upstream Kamaji fix [clastix/kamaji#1122](https://github.com/clastix/kamaji/pull/1122) (which fixes clastix/kamaji#1115) as a source patch on top of the pinned `26.3.6-edge` build. No version bump, no CRD/RBAC changes. This lands the fix on `main`; the `backport` / `backport-previous` labels fan it out to `release-1.5` and `release-1.4` via the backport automation. Supersedes the per-release PRs #3021 (release-1.5) and #3020 (release-1.4). ## Why Deleting a Tenant whose etcd `DataStore` has **0 referencing `TenantControlPlane`s** — e.g. a tenant deleted before any Kubernetes app / TCP is ever created in it — leaves the `DataStore` stuck `Terminating` with finalizer `kamaji.clastix.io/TenantControlPlane`. That hangs `helm uninstall <etcd>` (which waits by default), freezes the etcd HelmRelease at `status=uninstalling`, and wedges the tenant namespace in `Terminating` indefinitely. Tracked in #3018. Root cause: in `26.3.x-edge`, DataStore deletion drives an `AllowedDeletion` condition that only transitions `False → True`. With 0 referencing TCPs at deletion time the condition is never initialized, so the reconcile falls through to logging `DataStore can be safely deleted` and the deferred finalizer-removal (gated on `IsStatusConditionTrue`) never runs — with no requeue to retry. The pinned `26.3.6-edge` predates the upstream fix (merged 2026-04-14, first released in `26.4.4-edge`). ## Change Adds `packages/system/kamaji/images/kamaji/patches/fix-datastore-unused-deletion.diff`, applied by the existing `RUN git apply /patches/*.diff` step in the kamaji image build. The one-line guard adds the missing 0-TCP path: ```go - if meta.IsStatusConditionFalse(..., DataStoreConditionAllowedDeletionType) { + if meta.IsStatusConditionFalse(..., DataStoreConditionAllowedDeletionType) || len(tcpList.Items) == 0 { ``` ## Test - `git apply --check` confirms the patch applies cleanly to the `26.3.6-edge` source pinned on `main`; after applying, the guard is present at `controllers/datastore_controller.go`. - Reproduced the deadlock on a v1.4.x cluster (tenant deleted before any TCP existed) and confirmed the manual unstick; this patch removes the need for it. Workaround for an already-stuck DataStore (0 TCP refs): `kubectl patch datastore <name> --type=merge -p '{"metadata":{"finalizers":null}}'`. Relates to #2885 (full kamaji bump for CVEs — supersedes this patch once it lands). ```release-note Fix tenant namespace stuck `Terminating` when an etcd DataStore with no referencing TenantControlPlanes is deleted (Kamaji datastore unused-deletion deadlock). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Data stores are now recognized as unused even when no matching tenant control plane resources are found, improving deletion handling in that case. * This prevents inactive data stores from being left behind when there are no associated matches. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
47638f8569 |
fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122)
Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes left the DataStore stuck Terminating (finalizer kamaji.clastix.io/TenantControlPlane), hanging the etcd helm uninstall and wedging the tenant namespace. Backports clastix/kamaji#1122 (fixes clastix/kamaji#1115) as a source patch on the pinned 26.3.6-edge build. Refs #3018. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
59964043e3 |
fix(platform): migration 49 backfills tenant ancestor labels
The tenant chart fix makes new and re-reconciled tenant namespaces self-compute their full tenant.cozystack.io/<ancestor> chain, but existing namespaces whose HelmRelease does not promptly re-reconcile (suspended or failed releases) keep the incomplete labels — and with them the broken <ancestor>-egress reachability that surfaces as upstream timeouts to nested tenants. Backfill the labels directly so clusters heal on upgrade regardless of per-tenant HelmRelease state. Bump targetVersion to 50 so run-migrations.sh iterates the new script (seq CURRENT..49); the missing-file guard otherwise never fires. Best-effort and idempotent: kubectl label --overwrite adds only the listed labels, and per-namespace failures are tolerated so a transient apiserver error cannot abort the platform upgrade. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
1ac9398a4b |
fix(tenant): emit full ancestor label chain on tenant namespaces
The tenant.cozystack.io/<ancestor> namespace labels drive the <ancestor>-egress CiliumClusterwideNetworkPolicy, which lets an ancestor tenant reach a descendant namespace only when the descendant carries tenant.cozystack.io/<ancestor>. namespace.yaml derived that chain by splitting .Release.Namespace, which only reached one level up: a direct child of tenant-root got tenant-root, but a grandchild (tenant-ktj-htdev under tenant-ktj) got only tenant-ktj and dropped tenant-root entirely. The root ingress controller in tenant-root then could not reach nested tenants, surfacing as upstream timeouts. Derive the chain deterministically from the namespace name instead. Every tenant descends from tenant-root, and tenant.name builds each child as <parent-namespace>-<word>, so every progressive dash-prefix of the name is a real ancestor namespace. This needs no cluster lookup, renders offline (so it is unit-testable), and converges on every reconcile regardless of parent/child HelmRelease ordering. The ownerReference lookup is made non-fatal so the template still renders when the parent namespace lookup is empty; in a real reconcile the parent always exists and the ownerReference is always emitted. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
2b1d3e904d |
fix(kubernetes): gate vmop cert-manager-crds dependency on certManager.enabled
The victoria-metrics-operator HelmRelease is gated on
monitoringAgents.enabled but listed cert-manager-crds in dependsOn
unconditionally, while the cert-manager-crds HelmRelease is only
created when certManager.enabled is true. The valid combination
monitoringAgents.enabled=true + certManager.enabled=false therefore
left vmop blocked on a non-existent dependency
("...cert-manager-crds ... not found"), cascading to monitoring-agents
and vertical-pod-autoscaler. Gate the dependency on the same condition
that creates it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
71ca2b1bae |
fix(mariadb): re-guard replication so single-replica passes the operator webhook (#3180)
## What this PR does Fixes a regression where **single-replica MariaDB can no longer be created**. [PR #2279](https://github.com/cozystack/cozystack/pull/2279) made `replication.enabled: true` unconditional in the MariaDB CR so the operator would always create `<release>-primary` / `<release>-secondary` Services (expected by the dashboard RBAC and the ApplicationDefinition). But the vendored mariadb-operator's validating webhook (`vmariadb-v1alpha1.kb.io`) **rejects `replication.enabled: true` when `spec.replicas == 1`** — replication needs a primary + at least one replica. As a result, single-replica installs never became Ready (the HelmRelease failed at admission). The webhook rule lives in the operator's Go code (not in the vendored CRD — there is no CEL rule tying replication to replica count), so the failing path is reproduced by an e2e test rather than a schema assertion. ### Fix — re-guard replication, teach consumers about the bare service The operator provisions a bare `<release>` Service in **both** topologies; `-primary`/`-secondary` exist **only** when replication is on (`replicas>1`). Instead of forcing replication for everyone, this PR lets replication track the replica count and makes the service-name consumers handle the single-replica bare service: - **`mariadb.yaml`** — gate `replication` on `replicas>1`; restore single-replica external access via `service.type: LoadBalancer`, and keep `primaryService: LoadBalancer` for `replicas>1` only. (Effectively reverts #2279's two hunks.) - **`backup-cronjob.yaml`** — single-replica dumps target the bare service (no `-secondary` exists without replication); `replicas>1` still offloads to `-secondary`. - **`dashboard-resourcemap.yaml`** — always grant RBAC on the bare service, plus `-primary`/`-secondary` when `replicas>1`, matching what the ApplicationDefinition lists. - **`mariadb-rd`** — list the bare service in `services.include` so single-replica installs are visible in the dashboard. - **`hack/e2e-apps/mariadb.bats`** — new `Create single-replica MariaDB` test that reproduces the webhook rejection and guards against regression. Verified: `helm template` at `replicas=1`/`2` renders the intended CR/RBAC/backup host; `helm unittest` passes; `update-crd.sh` keeps the RD stable; `select-e2e.sh` maps the change to the mariadb suite. ### Release note ```release-note fix(mariadb): single-replica instances can be created again — replication is now enabled only for replicas>1 (matching the operator webhook), while the dashboard, RBAC, and backups correctly target the bare service for single-replica setups ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an end-to-end test covering MariaDB deployments with a single replica, including validation of the main service, endpoints, StatefulSet replica count, and metrics components. * **Bug Fixes** * Updated MariaDB chart behavior to correctly handle single-replica vs multi-replica configurations for external access, replication settings, and backup host selection. * Refined dashboard RBAC and secret inclusion so access applies to the main MariaDB service in addition to replica-specific services. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
41fd08e2ba |
feat(keycloak): enable UPDATE_EMAIL required action in cozy realm (#3175)
## What this PR does The portal user-settings page lets a user change their email via a Keycloak `kc_action=UPDATE_EMAIL` redirect. That flow only works when the **UPDATE_EMAIL required action is enabled at the realm level** — and on a fresh realm it is registered but **disabled by default**. `update-email` is a supported, default-on feature since Keycloak 26.4 (keycloak/keycloak#39722), so the provider is already loaded on the 26.x server this repo ships — **no `KC_FEATURES` change and no Keycloak restart are needed**. The `edp-keycloak-operator` has no CRD field for required actions, so this adds a small one-shot, idempotent `post-install`/`post-upgrade` Job to `keycloak-configure` that enables it via `kcadm.sh`, waiting for the operator to reconcile the realm first. Verified against a live Keycloak 26.5.2: the `UPDATE_EMAIL` action goes from `enabled=false` to `enabled=true`, and re-running the Job is a no-op. ### Release note ```release-note feat(keycloak): enable the UPDATE_EMAIL required action in the cozy realm so users can change their email from the portal ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Keycloak now automatically enables the email update action during install and upgrade, helping users change their email address in the `cozy` realm. * The configuration now includes the required Keycloak container image for this setup task. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
e4c6802863 |
fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry (#3171)
## What this PR does Fixes #3088. The post-delete cleanup Jobs in the **harbor** and **mariadb** charts hardcoded the cleanup image as `docker.io/clastix/kubectl:v1.32` — a moving Docker Hub tag. This: - **broke reproducibility / was a supply-chain risk** — `v1.32` is mutable and can be re-pushed to a different image; - **could not be satisfied on air-gapped / mirrored clusters** — the hardcoded `docker.io` bypassed the cluster images registry, and it failed at the worst moment: during **uninstall**, when the `post-delete` hook runs and a stuck Job stalls release teardown. ### Changes - **New canonical helper `cozy-lib.image`** (`packages/library/cozy-lib/templates/_cozyconfig.tpl`). Given a registry-relative image path and the global context, it prefixes `cozy-lib.images-registry` when set (`<registry>/<image>`) and returns the image unchanged when empty — **with no leading `/`**, so standard installs still resolve from the default registry. This establishes the previously-undefined "how to reference a cluster image" pattern (`images-registry` was defined but never consumed). - **Digest-pinned image** in both cleanup hooks: `clastix/kubectl:v1.32@sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c` (the immutable multi-arch OCI index for the `v1.32` tag; the same digest already pinned for `postgres-operator`'s webhook hook). - **Renovate keeps the pin fresh.** The repo disables the helm-values manager (`enabledManagers`), and the ref is assembled at render time by the helper rather than being a Dockerfile / go.mod dependency, so the built-in managers can't see it. A `custom.regex` manager over the hook templates tracks `clastix/kubectl` by tag+digest — mirroring the existing `extra/etcd` manager — so the supply-chain pin doesn't silently age. - **Hardened `cozy-lib.images-registry`** against a nil `.Values._cluster` so the helper never crashes the render — relevant precisely because this hook renders during teardown. - **Helm unit tests** in both charts assert the rendered image for three cases: empty registry (default), registry set, and the no-leading-slash edge case. Assertions match by **pattern** (routed + digest-pinned + no leading slash) rather than an exact digest, so a Renovate bump keeps them green. ### Verification - `helm unittest` passes for both charts (harbor: 16, mariadb: 10). - Real `helm template` renders confirmed for both charts: - empty registry → `clastix/kubectl:v1.32@sha256:b9ef…` - `images-registry=registry.internal:5000` → `registry.internal:5000/clastix/kubectl:v1.32@sha256:b9ef…` - The Renovate `matchStrings` regex verified to capture `currentValue=v1.32` and the digest from both hook templates. ### Note on the wider footprint The same `docker.io/clastix/kubectl:v1.32` hardcode is already merged on `main` in several other cleanup hooks (bucket, qdrant, gateway, seaweedfs, kubernetes, tenant, dashboard, keycloak-configure) and appears in the open PRs #3170 (etcd), #3094 (monitoring), #3072 (clickhouse). This PR fixes harbor + mariadb per the issue and establishes the `cozy-lib.image` + Renovate pattern; migrating the remaining hooks is a follow-up. ### Release note ```release-note fix(cozy-lib,harbor,mariadb): digest-pin the harbor and mariadb post-delete cleanup-hook kubectl image and route it through the cluster images registry (`cozy-lib.images-registry`) via the new `cozy-lib.image` helper, so mirrored / air-gapped installs can resolve it and uninstall no longer depends on a moving Docker Hub tag; a custom Renovate manager keeps the digest fresh ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Updated post-delete cleanup hook Jobs to use digest-pinned container images for more secure, repeatable execution. * Improved image reference rendering so cleanup hooks work correctly with or without an image registry configured, avoiding invalid leading/trailing slash issues. * **Tests** * Extended cleanup hook rendering tests to validate digest pinning and correct registry prefix behavior, including edge cases. * **Chores** * Enhanced automated dependency detection to treat digest-pinned hook image references as stable. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
6bf836db30 |
fix(dashboard): show LoadBalancer external IP on the app Services tab (#3172)
## What this PR does For an application exposed with `external: true` (which creates a `type: LoadBalancer` Service), the app detail → **Services** tab showed the service Type and Cluster IP but not the external IP assigned by MetalLB — so there was no way to learn the connection address from the per-app view. The pre-1.4.0 dashboard showed this, so it read as a regression (reported in #2789). Root cause: `ServicesTab.tsx` queried Services with a spec-only local type (`{ type, clusterIP, ports }`) and never read `status.loadBalancer.ingress[]`. The data was already reachable — `ExternalIpsPage.tsx` renders it — so this was a wiring gap in the per-app tab, not missing data. Changes: - Add a `ServiceStatus` type and extend the `useK8sList` query to `K8sResource<ServiceSpec, ServiceStatus>`. - Add an **External IP** column: `ip ?? hostname ?? "Pending"` for `LoadBalancer` services, `—` otherwise — mirroring `ExternalIpsPage`. - Add a co-located regression test (`ServicesTab.test.tsx`) covering the IP, hostname→Pending fallback, and the ClusterIP no-external case. Verified: `pnpm typecheck` clean, `pnpm test` green (320 tests, incl. 3 new), `eslint` clean on the changed files. Fixes #3111 ### Screenshots <img width="1220" height="572" alt="Снимок экрана 2026-07-02 в 14 12 11" src="https://github.com/user-attachments/assets/82cd1da8-ff81-4b95-a6bf-18729b6c85c7" /> ### Release note ```release-note fix(dashboard): show the external LoadBalancer IP on the per-app Services tab ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an **External IP** column to the service details view. * Load balancer services now show the external IP or hostname when available. * **Bug Fixes** * Fixed the external IP display for load balancer services so it no longer misses valid IPs. * Improved fallback behavior by showing **Pending** when no external endpoint is assigned. * Non-load balancer services now show an em dash instead of misleading external IP text. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a16268152d |
fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant module delete (#3090) (#3170)
## What this PR does When the `etcd` tenant module is disabled (`etcd: false`), the etcd HelmRelease is pruned but two classes of resources leak into the tenant namespace (reported in #3090): - **Data PVCs** — the etcd-operator StatefulSet `volumeClaimTemplate` PVCs `data-etcd-<ordinal>` are never garbage-collected on `EtcdCluster` deletion. - **TLS secrets** — the five `etcd-*-tls` secrets survive the uninstall: - `etcd-ca-tls` / `etcd-client-tls` are created by this chart as `pre-install` hooks with `helm.sh/resource-policy: keep`, so Helm deliberately keeps them; - `etcd-peer-ca-tls` / `etcd-server-tls` / `etcd-peer-tls` are populated by cert-manager from `Certificate` resources and carry no owner reference back to the chart. This adds a `post-delete` cleanup hook (`packages/extra/etcd/templates/hooks/cleanup.yaml`), modeled on the accepted seaweedfs (#3092) and monitoring (#3091) sibling fixes, that: - deletes the data PVCs by name, one per replica (`data-etcd-0 … data-etcd-<replicas-1>`); - deletes the five `etcd-*-tls` secrets by name. `check-release-name` pins the release name to `etcd`, so every target name is deterministic and no sibling workload shares the `data-etcd-*` prefix in a tenant namespace — deletion by name is safe. cert-manager TLS secrets carry only the cluster-wide `controller.cert-manager.io/fao` label, so the RBAC grants `delete` by explicit `resourceNames` rather than an unsafe label sweep. The cleanup Job is hardened: non-root, read-only root fs, dropped capabilities, `seccompProfile: RuntimeDefault`, resource requests/limits, `activeDeadlineSeconds`. Deletes use `--ignore-not-found` and failures are logged but never block teardown. A Helm unit-test suite (`tests/cleanup_test.yaml`) asserts the hook metadata/RBAC/hardening and that the rendered commands target exactly the reported PVCs and secrets, including scaling the PVC list to the configured replica count. > [!NOTE] > The **stuck HelmRelease finalizer** noted in #3090 is the platform-wide sharded-helm-controller side-bug (tracked separately); it affects every `chartRef`-based tenant module identically, not just etcd. This PR fixes the PVC and secret orphans the etcd chart itself owns, matching the scope of the merged seaweedfs/monitoring fixes. > [!WARNING] > **Disabling the etcd module now permanently deletes its data PVCs and the etcd data they hold.** This is intentional and fixes the storage leak. Back up before disabling if the data matters. Fixes #3090. ### Release note ```release-note fix(etcd): garbage-collect the orphaned etcd data PVCs (data-etcd-*) and the five etcd-*-tls secrets left behind when a tenant's etcd module is disabled. WARNING: disabling etcd now permanently removes its data PVCs and the data they hold. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a Helm post-delete cleanup hook for etcd that removes per-replica etcd PVCs and the related TLS secrets when a release is deleted. * Cleanup permissions and targets now scale with replica count, including safe handling when replicas are set to zero. * **Tests** * Added Helm unit tests validating hook annotations/order, scoped RBAC, secure job hardening, and the exact deletion behavior for PVCs/secrets (including higher-replica scenarios). * **Chores** * Marked the `test` make target as phony for the etcd package. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a76c5929fd |
fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry
The post-delete cleanup Jobs in the harbor and mariadb charts hardcoded `docker.io/clastix/kubectl:v1.32` — a moving tag pulled directly from Docker Hub. This broke build reproducibility (the tag can be re-pushed), was a supply-chain risk, and could not be satisfied on air-gapped / mirrored clusters, where it failed at the worst moment: during uninstall, when the post-delete hook runs and stalls release teardown. Add a canonical `cozy-lib.image` helper that prefixes a registry-relative image path with `cozy-lib.images-registry` when set and returns it unchanged (no leading "/") when empty, so standard installs resolve from the default registry and mirrored installs resolve internally. Consume it from both cleanup hooks with the image digest-pinned to the immutable multi-arch index sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c (the same digest already pinned for postgres-operator's webhook hook). Keep the pin fresh with a custom Renovate manager: the repo disables the helm-values manager (enabledManagers), and the ref is built at render time by the helper rather than being a Dockerfile/go.mod dependency, so the built-in managers cannot see it. A regex manager over the hook templates tracks clastix/kubectl by tag+digest — mirroring the existing extra/etcd manager — so the supply-chain pin does not silently age. Also harden `cozy-lib.images-registry` against a nil .Values._cluster so the helper never crashes the render — relevant precisely because the hook renders during teardown. Helm unit tests assert the rendered image (routed, digest-pinned, no leading slash) by pattern rather than an exact digest, so a Renovate bump keeps them green. Closes #3088 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
caa6607f4d |
feat(keycloak): add optional KMS-encrypting database proxy (#3003)
## What this PR does Adds an optional, flag-gated **column-level PII encryption** feature to the `keycloak` system package. When `encryption.enabled=true`, a `keycloak-kms-proxy` Deployment + Service is placed on the wire between the Keycloak StatefulSet and its CloudNativePG database, and Keycloak's `KC_DB_URL_HOST`/`KC_DB_URL_PORT` are repointed at the proxy. The proxy transparently encrypts Keycloak PII columns (username, email, first/last name, `pii-*` attributes, credential secrets) before they reach PostgreSQL and decrypts them on reads; the data-encryption key is wrapped by a KMS and never lives inside the database. The proxy lives in its own repository: https://github.com/cozystack/keycloak-kms-proxy (image `ghcr.io/cozystack/keycloak-kms-proxy:v0.1.0`). **KMS backend** is selectable via `encryption.kms.backend`: - `static` (default): a self-generated 32-byte KEK kept in a Secret — works on any cluster, suitable for evaluation. - `vault-transit`: the KEK lives in HashiCorp Vault (recommended for production; supports KEK rotation). The feature is **off by default**, so existing installs are unaffected — `KC_DB_URL_HOST` keeps sourcing from the CNPG `keycloak-db-app` secret. ### Changes - `values.yaml`: new `encryption` block. - `templates/proxy.yaml` (new): gated Deployment + Service (+ self-generated KEK Secret for the static backend), hardened pod (nonroot, read-only rootfs, dropped caps, seccomp). - `templates/sts.yaml`: conditional DB-endpoint override when encryption is enabled. - `tests/encryption_test.yaml` (new): helm-unittest covering on/off and both KMS backends. `helm unittest .` → 11/11 pass. ### Follow-ups (not in this draft) - Backend TLS re-origination to the CNPG endpoint + CA validation. - Liquibase bootstrap-window handling (`KKP_LENIENT`). - Offline backfill Job for migrating existing data. - HA via a shared DEK set (currently single replica). ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(keycloak): add optional KMS-encrypting database proxy for column-level PII encryption (off by default; static KEK or Vault Transit) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional transparent encryption for Keycloak database sensitive fields * Users can configure encryption scope and select key management backend options * **Tests** * Added comprehensive test coverage validating encryption feature across multiple configuration scenarios <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9472bb3412 |
fix(etcd): clamp cleanup PVC resourceNames so replicas:0 cannot widen RBAC
Address CodeRabbit review on #3170: the PVC cleanup Role generates its resourceNames from .Values.replicas. values.schema.json sets no minimum, so an explicit replicas: 0 rendered an empty resourceNames list — and an empty resourceNames in an RBAC rule matches every PVC in the namespace (a wildcard), silently undoing the least-privilege scoping the previous commit added. Clamp the count to max(1, replicas) via a shared $replicaCount used by both the Role resourceNames range and the delete command, so at least data-etcd-0 is always listed (it exists for any running cluster and the delete uses --ignore-not-found). Add a regression test asserting replicas: 0 still yields a name-scoped Role and a single-name delete rather than a namespace-wide grant. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
4a06df92e7 |
fix(etcd): scope cleanup Role to the data-etcd PVC names and assert hook metadata
Address CodeRabbit review on #3170: - Tighten the cleanup Role's PVC rule from a namespace-wide get/list/delete to get/delete by explicit resourceNames (data-etcd-<ordinal>, one per replica), mirroring the least-privilege pattern already used for the TLS secrets. The Job only ever deletes those deterministic names, so list is unnecessary and a regression can no longer reach a sibling workload's PVC. - Extend the Role and RoleBinding unit tests to assert helm.sh/hook and helm.sh/hook-delete-policy (not just the weight), so the RBAC cannot regress into normal release resources, and assert the PVC resourceNames. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
a92f060f63 |
fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant module delete (#3090)
When the etcd tenant module is disabled (etcd: false), the etcd-operator StatefulSet data PVCs (data-etcd-<ordinal>) are never garbage-collected, and the five etcd-*-tls secrets are left orphaned in the tenant namespace: etcd-ca-tls / etcd-client-tls are pre-install hooks this chart marks with helm.sh/resource-policy: keep, and etcd-peer-ca-tls / etcd-server-tls / etcd-peer-tls are cert-manager-issued secrets with no owner reference back to the chart. None are reclaimed by the Helm uninstall. Add a post-delete cleanup Job (modeled on the seaweedfs/monitoring sibling fixes) that deletes the data PVCs by name (one per replica) and the five TLS secrets by name. check-release-name pins the release to "etcd", so every target name is deterministic and no sibling workload shares the data-etcd-* prefix. The Job runs non-root with a read-only root fs, dropped capabilities and seccompProfile RuntimeDefault; its RBAC is scoped to PVC get/list/delete and the five secrets by resourceNames. Deletes use --ignore-not-found and failures are logged but never block teardown. The stuck-HelmRelease finalizer noted in the issue is the platform-wide sharded-helm-controller side-bug tracked separately; this change addresses the PVC and secret orphans the etcd chart owns. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
c7e8f4d975 |
fix(mariadb): re-guard replication so single-replica passes the operator webhook
PR #2279 made `replication.enabled: true` unconditional so the operator would always create -primary/-secondary services. But the mariadb-operator validating webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1, so single-replica MariaDB could no longer be created — the HelmRelease never became Ready. Re-guard replication on replicas>1 and teach the service-name consumers about the bare <release> service the operator provisions in both topologies: - mariadb.yaml: gate `replication` on replicas>1; restore single-replica external access via `service.type: LoadBalancer` and keep `primaryService` LoadBalancer for replicas>1 only. - backup-cronjob.yaml: route single-replica dumps to the bare service (no -secondary exists without replication). - dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus -primary/-secondary when replicas>1, matching the ApplicationDefinition. - mariadb-rd: list the bare service in services.include so single-replica installs are visible in the dashboard. Adds a single-replica e2e test that reproduces the webhook rejection and guards against regression. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
fdbe6b299c |
fix(clickhouse)!: clean up orphaned keeper and data PVCs on delete (#3057) (#3072)
## What this PR does After deleting a `ClickHouse` app CR, data PVCs were left orphaned in the tenant namespace and consumed storage permanently: - the keeper PVC `default-chk-<release>-keeper-cluster1-0-0-0`, backing a `ClickHouseKeeperInstallation` reconciled by the clickhouse-keeper-operator; - the main ClickHouse data / log PVCs. The keeper operator does not delete data PVCs when the CR is removed (data-safety default) and the CRD exposes no `persistentVolumeClaimRetentionPolicy` equivalent, so the volumes leaked. This adds a `post-delete` cleanup hook (modeled on the existing MariaDB hook at `packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml`) that deletes every PVC carrying the release label `app.kubernetes.io/instance=<release>`. It also adds that label to the keeper `volumeClaimTemplates` in `chkeeper.yaml`: the keeper PVC previously had **no** release label (the chart set labels only on the keeper pod template, not the PVC), so the operator never stamped one and a single selector would have missed it — leaving the reported leak unfixed. ### Existing releases (label backfill migration) The altinity clickhouse-keeper-operator only stamps `volumeClaimTemplates.metadata.labels` onto PVCs it creates **after** the chart change; it never re-labels PVCs that already exist. So for releases installed before this fix, the keeper PVC stays unlabeled and the cleanup selector would still miss it. To close #3057 for existing customers too, this PR adds migration `48` (`48 -> 49`, `packages/core/platform/images/migrations/migrations/48`) that back-fills `app.kubernetes.io/instance=<release>` onto pre-existing keeper PVCs, and bumps `migrations.targetVersion` to `49`. The migration: - routes its version stamp through the shared `lib/cozystack-version.sh` helper (`stamp_cozystack_version 49`), per the `cozystack-version-stamp.bats` convention; - no-ops when the keeper CRD is absent, and tolerates a transient apiserver error on the keeper list (`|| true`) so it cannot abort a platform-wide upgrade — the backfill is best-effort (worst case is the pre-existing leak, no worse than today); - is idempotent (skips PVCs already labeled). The main ClickHouse data/log PVCs have carried `app.kubernetes.io/instance` since 2025-06-03 (well before the migration era), so they need no backfill. ### Hook failure handling The cleanup hook does **not** mask failures: `kubectl delete` failing makes the Job exit non-zero. The **Job** uses `helm.sh/hook-delete-policy: before-hook-creation` only (no `hook-succeeded`) so a failed cleanup Job and its logs survive for debugging; the **ServiceAccount/Role/RoleBinding** keep `before-hook-creation,hook-succeeded` so the RBAC artifacts are reclaimed after a successful uninstall instead of lingering in the namespace. ### Scope The hook intentionally reclaims **only PVCs** of the release. Other release state (Secrets, ConfigMaps) is owned/garbage-collected through the normal Helm uninstall path and is out of scope here. > [!WARNING] > **This is destructive — deleting a ClickHouse app now permanently deletes its data.** > The hook removes **all** PVCs of the release: the keeper PVC **and** the main > ClickHouse data and log PVCs. After deletion nothing is left behind to recover > data from. This is intentional and matches the existing MariaDB cleanup-hook > behaviour and the general cozystack model (deleting a managed app reclaims its > storage). Back up before deleting. Fixes #3057. ### Release note ```release-note fix(clickhouse): garbage-collect the keeper and data/log PVCs when a ClickHouse app is deleted, so no orphaned volumes are left behind. A migration back-fills the release label onto keeper PVCs from releases installed before this fix so they are cleaned up too. WARNING: deleting a ClickHouse app now permanently removes all of its PVCs (keeper + data + logs) and the data they hold. ``` |
||
|
|
9e89ae84ac |
fix(kubernetes): default worker disk storageClass to the application storageClass (#3129)
## What this PR does
Worker node VMs in a tenant Kubernetes cluster set `evictionStrategy:
External`, so they live-migrate. Live migration requires the worker
system DataVolume to be ReadWriteMany, and linstor-csi only grants RWX
on a DRBD-backed StorageClass. When a node group leaves `storageClass`
empty, the worker DataVolume template previously omitted
`storageClassName` and fell through to the management cluster's default
StorageClass — which is operator-configured and not guaranteed to be
DRBD-backed. On any cluster whose default StorageClass is not a
replicated/DRBD class, the worker disk lands on a non-RWX class and live
migration breaks.
This changes the empty-`storageClass` fallback from the management
cluster default to the application-level `storageClass` (default
`replicated`, a DRBD/LINSTOR class): `{{- with .group.storageClass |
default $.Values.storageClass }}`. A per-node-group `storageClass` still
wins when set; when both the node group and the application
`storageClass` are empty, `storageClassName` is omitted so the cluster
default still applies (explicit opt-out preserved).
Doc and codegen surfaces (`types.go`, `values.yaml`,
`values.schema.json`, `README.md`, and the generated `kubernetes-rd`
ResourceDefinition) are regenerated in lockstep with the new field
description. Chart unit tests cover the full contract: fallback to the
application class, a custom application class, per-node-group override
winning, and the both-empty omit path.
**Upgrade impact — one-time worker-pool roll.** The
`KubevirtMachineTemplate` name embeds a hash of the rendered worker
template body, and the MachineDeployment references that name. On the
shipped default (`md0` leaves `storageClass` empty) the body changes
from "no `storageClassName`" to `storageClassName: replicated`, so on
upgrade the template is renamed and Cluster API performs a one-time
rolling replacement of the worker pool — for every existing tenant
cluster, including those whose default StorageClass already was
`replicated` (the roll is driven by the body hash, not by whether the
effective class differs). This is the same class of roll any
worker-template change triggers and is unavoidable with this approach.
Two test pins document and guard the hash so future body changes are
deliberate.
### Release note
```release-note
fix(kubernetes): worker node disks now fall back to the application-level storageClass (a replicated/DRBD class) instead of the management-cluster default when a node group leaves storageClass empty, so live migration of worker VMs gets the ReadWriteMany volume it requires. Upgrading rolls each tenant cluster's worker pool once.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved worker disk `storageClassName` fallback for empty
per-node-group values: it now reliably uses the app-level
replicated/DRBD StorageClass (rather than relying on the
management-cluster default).
* Ensured KubeVirt data volume templates and rendering behavior are
consistent across single- and multi-node-group setups.
* **Documentation**
* Refined `storageClass`/RWX/DRBD guidance across configuration docs and
schema descriptions, including the intended non-immutable behavior
(optional/undefaulted).
* **Tests**
* Updated template test expectations to cover the new fallback vs
omission cases, including naming assertions for rendered templates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
83bd83062e |
feat(keycloak): enable UPDATE_EMAIL required action in cozy realm
The portal user-settings page changes a user's email via a Keycloak kc_action=UPDATE_EMAIL redirect, which only works when the UPDATE_EMAIL required action is enabled at the realm level. On a fresh realm the action is registered (update-email is a supported, default-on feature since Keycloak 26.4) but disabled by default. The edp-keycloak-operator has no CRD field for required actions, so add a one-shot, idempotent post-install/post-upgrade Job to keycloak-configure that enables it via kcadm.sh, waiting for the operator to reconcile the realm first. Assisted-By: Claude Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
bf75d32a1a |
fix(dashboard): show LoadBalancer external IP on the app Services tab
The per-app Services tab queried Services with a spec-only type, so it never read status.loadBalancer.ingress and showed only the cluster-internal IP for external: true (type: LoadBalancer) services. This regressed the pre-1.4.0 behaviour (cozystack/cozystack#2789). Add a ServiceStatus type to the useK8sList query and surface an "External IP" column (ip / hostname / Pending for LoadBalancer, — otherwise), mirroring what ExternalIpsPage already does. Fixes #3111 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
7faa32508d |
fix(clickhouse): reclaim cleanup hook RBAC on successful uninstall
Address review [MINOR] on #3072: keep the cleanup Job on before-hook-creation (so a failed Job and its logs survive for post-mortem), but restore before-hook-creation,hook-succeeded on the ServiceAccount/Role/RoleBinding so those artifacts do not linger in the namespace after a successful uninstall. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
2e65ce108b |
fix(clickhouse): guard migration when keeper CRD absent and quote hook scalars
Address CodeRabbit review on #3072: - Migration 45 now skips the backfill when the clickhousekeeperinstallations CRD is not installed. The migration runs platform-wide via targetVersion, so on clusters that never deployed ClickHouse the kubectl get would otherwise fail under set -euo pipefail and abort the whole upgrade. - Quote the templated name/label/serviceAccountName scalars in the cleanup hook so the source parses under YAMLlint; rendered output is unchanged. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
a2ae1621af |
fix(clickhouse): surface cleanup hook failures and backfill legacy keeper PVC labels
Address review on #3072: - Cleanup hook no longer masks kubectl delete failures: the command now fails (exit 1) when PVC deletion fails instead of swallowing the error with || echo. The hook-delete-policy drops hook-succeeded (keeps only before-hook-creation) so a failed cleanup Job survives for inspection. - Add migration 45 (45 -> 46) that backfills app.kubernetes.io/instance onto keeper PVCs created before this change; the altinity keeper operator never re-stamps volumeClaimTemplate labels onto existing PVCs, so without this the cleanup selector misses pre-upgrade keeper PVCs and #3057 persists for existing releases. Bump migrations.targetVersion to 46. - Unify helm.sh/hook-weight key quoting across the four hook manifests. - Extend the unittest to assert the new hook-delete-policy and exit-on-failure. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
2c8f6b8ca1 |
fix(clickhouse)!: clean up orphaned keeper and data PVCs on delete (#3057)
After deleting a ClickHouse app CR, the keeper PVC and the main data/log PVCs were left orphaned in the tenant namespace, consuming storage permanently. The clickhouse-keeper-operator does not delete data PVCs on CR removal (data-safety default) and the CRD exposes no persistentVolumeClaimRetentionPolicy equivalent. Add a post-delete cleanup hook (modeled on the MariaDB hook) that deletes every PVC carrying the release label app.kubernetes.io/instance=<release>. Also stamp that label on the keeper volumeClaimTemplates in chkeeper.yaml: the keeper PVC previously had no release label (the chart labelled only the keeper pod template, not the PVC), so a single selector would have missed the very PVC the issue reports. BREAKING CHANGE: deleting a ClickHouse app now permanently removes all of its PVCs (keeper + data + logs) and the data they hold. Nothing is left behind to recover from. This matches the existing MariaDB cleanup-hook behaviour and the general cozystack model where deleting a managed app reclaims its storage. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
|
fff2e7716f |
fix(seaweedfs)!: clean up orphaned data PVCs and cert/db secrets on tenant module delete (#3092)
When the seaweedfs tenant module is disabled, the StatefulSet volume PVCs and a set of certificate/credential secrets were left orphaned. Add a post-delete cleanup hook that deletes the volume PVCs by their release-scoped labels and the secrets by name (all templated from .Release.Name). CNPG PVCs are not targeted (CNPG cleans them). The cleanup Job is hardened (non-root, read-only root fs, dropped caps, resource limits); RBAC scoped via resourceNames; failures are logged, never blocking teardown. The PVC selector matches on name + instance only, without an app.kubernetes.io/component clause, so volume PVCs from custom storage pools and MultiZone topologies (component volume-<pool>/volume-<zone>[-<pool>]) are cleaned up too, not just the default pool. Both deletes use --wait=false and the Job has activeDeadlineSeconds so a finalizer-stuck resource cannot hang the post-delete hook. NOTE: disabling seaweedfs now permanently removes its volume PVCs and the data they hold. BREAKING CHANGE: disabling the seaweedfs tenant module now permanently deletes its volume PVCs and the object data they hold. Back up before disabling if the data matters. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
|
5d36e13d06 |
refactor(keycloak): drop the generic db.* connection override
Remove the db.host/db.port/db.urlProperties override and the extraVolumes/extraVolumeMounts plumbing: they are no longer needed now that the database endpoint is handled natively. Keycloak's DB host/port come straight from the keycloak-db-app secret again, except when encryption is enabled, where the keycloak-kms-proxy owns the repointing. Theme and admin-console hostname handling are unaffected. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
d459ad800a |
test(keycloak): cover encryption misconfiguration failures
Add render-time negative-path coverage for the encryption proxy: an unsupported kms.backend and a vault-transit backend missing address, keyName, or tokenSecretName must all fail the template render rather than silently producing a broken Deployment. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
956759388f |
docs(keycloak): document encryption operational caveats and cap proxy memory
Address review feedback on the encryption values block: - Spell out the operational caveats an operator must understand before flipping encryption on: the single-replica proxy turns HA Keycloak into a single point of failure; existing data is not migrated automatically so enabling (or later disabling) on a populated database breaks logins until the offline backfill runs; and a lost static KEK is permanent data loss, so the KEK Secret must be backed up. - Add a memory limit to the proxy resources per the repository style guide. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
e97c918829 |
fix(keycloak): fail fast on invalid KMS backend and harden proxy template
Address review feedback on the encryption proxy template: - Validate encryption.kms.backend at render time and fail with a clear message on any value other than static/vault-transit. Both the KEK Secret and the env wiring now key off the same validated $backend, so a typo can no longer render a Deployment that mounts a KEK Secret that was never created (which would silently break the whole IdP). - Guard the static KEK lookup against a Secret that exists without a usable 'kek' key, preserving the freshly generated KEK instead of nil/empty. - Default kms.vault to an empty dict and require address/keyName at render time so an incomplete Vault config fails with an actionable message rather than a generic nil dereference. - Publish the metrics port (9090) on the proxy Service so it can be scraped. - Clarify the backend-TLS note: the proxy negotiates TLS in-band; CNPG CA pinning is the remaining follow-up. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
f5b078d8c7 |
feat(keycloak): add optional KMS-encrypting DB proxy
Add an opt-in `encryption` flag to the keycloak system package. When encryption.enabled=true, a keycloak-kms-proxy Deployment + Service is placed on the wire between Keycloak and its CloudNativePG database, and Keycloak's KC_DB_URL_HOST/PORT are repointed at the proxy. The proxy transparently encrypts PII columns (username, email, first/last name, PII attributes, credentials) before they reach PostgreSQL and decrypts them on reads; the data-encryption key is wrapped by a KMS and never lives inside the database. The KMS backend is selectable: a self-generated static KEK (default, works on any cluster, suitable for evaluation) or HashiCorp Vault Transit (recommended for production, supports KEK rotation). The feature is off by default, so existing installs are unaffected. Skeleton/draft scope: backend TLS re-origination to CNPG, the Liquibase bootstrap window (KKP_LENIENT), an offline backfill Job for pre-existing data, and HA via a shared DEK set are tracked as follow-ups. Proxy project: https://github.com/cozystack/keycloak-kms-proxy Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
a31a73cb87 |
fix(tenant): gate VM-resource HelmReleases on victoria-metrics-operator readiness (#3141)
## What this PR does The tenant `etcd`, `ingress` and `monitoring` HelmReleases create VictoriaMetrics custom resources — `VMPodScrape` directly in `etcd` and `ingress`, and `VMCluster` / `VMAlert` / `VLCluster` / `VMAlertmanager` / `VMServiceScrape` via the `monitoring-system` child that `monitoring` creates. All of these are intercepted by the `victoria-metrics-operator` validating webhook, which is configured with `failurePolicy: Fail`. On a cold install the operator pod can be briefly unavailable on `:9443` (for example a transient API-server blip during startup leaves the webhook backend not yet serving). In that window the tenant releases reconcile, the webhook rejects their VM* resources with `connection refused`, and the releases churn through failed-install → rollback → retry. They carry `install.remediation.retries: -1` and recover eventually, but the churn can outlast a cold-install readiness deadline and surfaces as spurious install failures. This adds a Flux `spec.dependsOn` from each of those three tenant releases to the `victoria-metrics-operator` HelmRelease in `cozy-victoria-metrics-operator`, so they wait for the operator to be Ready before creating any VM* resource. The system monitoring components already gate on the operator the same way (a variant-level `dependsOn` in the platform package sources); this brings the tenant releases to parity. `failurePolicy: Fail` is preserved — validation is unchanged. The tenant releases are sharded (`sharding.fluxcd.io/key`) while the operator release is not. helm-controller resolves `dependsOn` through an uncached API read that bypasses the shard's watch-label-selector cache (since v1.1.0, PR fluxcd/helm-controller#1070), so the cross-shard reference is visible to the tenant shard's controller; this repo ships helm-controller v1.5.0. A helm-unittest suite pins the gate on all three releases and asserts that the tenant releases creating no VM* resources (`seaweedfs`, `gateway`) do not carry it, documenting the intentional narrowness of the gate. ### Release note ```release-note fix(tenant): gate tenant etcd, ingress and monitoring HelmReleases on victoria-metrics-operator readiness so they no longer fail to install when the operator's admission webhook is briefly unavailable on a cold install ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved tenant installation reliability by adding readiness gating: etcd, ingress, and monitoring Helm releases now wait for the Victoria Metrics operator to become Ready before applying dependent resources. * Prevents install/rollback failures caused by temporary admission webhook unavailability during cold starts. * **Tests** * Added Helm-unittest coverage to confirm the dependency gate is present only for templates that create VM* resources (and omitted for templates that don’t). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
03435dcc50 |
fix(flux): tune source-watcher concurrency + CPU to unblock platform install under load (#3163)
## What this PR does `source-watcher` is the Flux controller that turns each `PackageSource` into an `ExternalArtifact` (one `ArtifactGenerator` per source, ~100 builds on the management cluster during install). The `cozystack-platform` HelmRelease blocks until all of those artifacts are Ready, so source-watcher's throughput sits on the install's critical path. It was the only Flux controller left without concurrency/CPU tuning. In the bootstrap all-in-one pod it ran on the binary default (`--concurrent=10`) with the shared 100m CPU request; in the production `FluxInstance` it was absent from the reconciler patch that source-, kustomize- and helm-controller already receive. Under load its per-artifact builds finish at staggered times, leaving a random tail of not-yet-Ready artifacts that times out the platform install. A failed run's diagnostics show the shape of this: the `cozystack-platform` HelmRelease timed out with `timeout waiting for: PackageSource/… status: 'InProgress'` (a run-to-run varying tail — e.g. `bucket-application`, `linstor-gui`, `capi-provider-core`), while its ~200 ExternalArtifacts reached Ready in a staggered trickle spread over ~60 seconds rather than together. That staggered completion of independent per-artifact builds is the signature of a throughput-limited builder, not a single all-or-nothing pull. This raises source-watcher to `--concurrent=20` in both install paths: - **Bootstrap all-in-one pod** (`internal/fluxinstall/manifests/fluxcd.yaml`, regenerated from `flux-aio.cue` + the `flux-aio` Makefile): `--concurrent=20` plus a 500m CPU request. The AIO pod sets no CPU limits, so the request is what drives source-watcher's proportional CPU share when the node is saturated. - **Production `FluxInstance`** (`packages/system/fluxcd/values.yaml`): source-watcher is added to the shared reconciler patch, so it now gets the same `--concurrent=20`, `--requeue-dependency=5s` and 2000m/2048Mi limits as the other tuned controllers. The `flux-aio` module exposes a per-controller resources override for the watcher (set via CUE) but no args override, so the bootstrap `--concurrent` bump is a guarded post-render patch in the Makefile; the embedded manifest is the product of that pipeline. source-watcher's support for both `--concurrent` and `--requeue-dependency` was confirmed against its `cmd/main.go`, and the production `op: replace` on `resources/limits` is safe because the distribution ships source-watcher with a `resources.limits` block (same shape as source-controller). Optional follow-up, intentionally left out to keep this focused on the root cause: if the artifact-build tail still brushes the 10-minute platform HelmRelease timeout under load, the installer's HelmRelease install/upgrade timeout values give a safety margin. ### Release note ```release-note fix(flux): tune source-watcher concurrency and CPU so the platform install does not time out under load ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Increased the source watcher’s concurrency to improve throughput and better handle larger workloads. * Added higher CPU and memory settings for the watcher to support more stable operation under load. * **Bug Fixes** * Updated deployment patching so the watcher receives the same configuration adjustments as other controllers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
6db5ae9c6f |
fix(tenant): gate VM-CR-creating HelmReleases on victoria-metrics-operator
The tenant etcd, ingress and monitoring HelmReleases create VM* custom resources (VMPodScrape directly in etcd/ingress; VMCluster, VMAlert, VLCluster, VMAlertmanager and VMServiceScrape via the monitoring-system child of monitoring) guarded by the victoria-metrics-operator validating webhook with failurePolicy: Fail. On a cold install the operator pod can be briefly unavailable on :9443, so these releases reconcile before the webhook backend is serving and are rejected with connection refused. They carry install.remediation.retries: -1 and would eventually recover, but the failed-install -> rollback -> retry churn can outlast a cold-install readiness deadline and surfaces as spurious install failures. Add a Flux spec.dependsOn from each of these releases to the victoria-metrics-operator HelmRelease in cozy-victoria-metrics-operator, so they wait for the operator to be Ready before creating VM* resources. The system monitoring components gate on the operator the same way, via a variant-level dependsOn. The cross-shard reference (tenant releases are sharded, the operator release is not) resolves via helm-controller's uncached dependency check (APIReader, v1.1.0+). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
41ff94eca9 |
fix(flux): tune source-watcher concurrency and CPU under load
source-watcher reconciles one ArtifactGenerator per PackageSource and builds the resulting ExternalArtifacts. On the management cluster that is ~100 artifact builds during install, and the platform HelmRelease blocks until all of them are Ready. It was the only Flux controller left untuned. In the bootstrap all-in-one pod it ran without an explicit --concurrent (binary default 10) and with the shared 100m CPU request; in the production FluxInstance it was missing from the --concurrent / CPU patch that source-, kustomize- and helm-controller already receive. Under load its per-artifact builds finish at staggered times, leaving a random tail of not-yet-Ready artifacts that times out the platform install. Raise source-watcher to --concurrent=20 in both deployments. In the bootstrap all-in-one pod also give it a 500m CPU request; the pod has no CPU limits, so the request is what drives its CPU share under contention. In the production FluxInstance add it to the shared reconciler patch, so it now receives the same --requeue-dependency=5s and 2000m/2048Mi limits the other tuned controllers already have. The flux-aio module exposes a resources override for the watcher (set via CUE) but no args override, so the bootstrap --concurrent bump is a post-render patch in the Makefile. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
f7271b8011 |
fix(velero): add startupProbe so slow startup does not crashloop the install gate
Velero binds its metrics/health endpoint (:8085, http-monitoring) only after the server loads its plugins and connects to the API server. Under a heavy parallel platform install that can take minutes, but the upstream liveness probe begins checking /metrics at initialDelaySeconds=10 and kills the container after failureThreshold=5 (~130s total) before :8085 is bound. The kubelet SIGTERMs velero (exitCode 0/Completed), it BackOff-restarts, and never escapes the loop inside the install-gate window. Because backupstrategy-controller hard-depends on velero, this blocks the gate. Add a startupProbe to the velero server container. While it is failing the kubelet holds off liveness and readiness entirely, granting failureThreshold * periodSeconds = 30 * 10s = 300s of startup grace, then hands over to the unchanged, tight upstream liveness (10s / 30s / 5) for steady-state crash detection. This is strictly better than loosening liveness, which would also blunt steady-state detection. The upstream velero chart (12.0.3) does not render a startupProbe, so the package patches the deployment template via patches/add-startup-probe.patch (re-applied after re-vendor by the Makefile update target, matching the seaweedfs/ingress-nginx convention) and supplies the startupProbe through the package values override. Pin the startupProbe budget and the unchanged liveness with a helm-unittest regression guard. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
9c18260f95 |
fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets) (#2957)
## What The e2e install pipeline still flakes on a loaded runner, failing at a different point each run. Root-caused via repeated runs and a live SSH breakpoint; the platform itself is fine (the components do converge), the failures are slowness + self-inflicted churn under resource pressure. ### Fixes 1. **VPA updater OOM** — the `vertical-pod-autoscaler-updater` memory limit was `110Mi`. The updater watches every pod in the cluster, so on a full install it OOM-kills → `ProgressDeadlineExceeded` → Flux fails the HelmRelease "early due to stalled resources", cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raised to `512Mi` (request `200Mi`). 2. **VPA-for-VPA eviction churn** — the two VPA-for-VPA `VerticalPodAutoscaler` objects ran `updateMode: Auto`, evicting the VPA recommender(s) during install and adding pod churn on an already busy cluster. Switched to `Initial`, matching the existing monitoring/etcd `VPA->Initial` fixes. 3. **Install wait budgets** — components reach readiness but overshoot tight waits on a busy runner: keycloak HR chain `10m→20m`, tenant aggregate wait `10m→20m`, per-app tenant readiness waits `5m→10m`. (The crust-gather failure-snapshot timeout that kept whole jobs hanging to the 3h cancel is already on `main`.) ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased several installation and readiness wait times to better handle slower environment startup and reduce premature failures. * Updated vertical autoscaling behavior to avoid unnecessary pod restarts during deployment. * Raised resource limits for the autoscaling updater to improve reliability under load. * **Tests** * Aligned automated checks with the new autoscaling behavior and longer readiness expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d6f66d2dcd |
fix(vertical-pod-autoscaler): stop the updater OOM + VPA-for-VPA eviction churn
Two root causes of e2e install flakiness, both in the VPA package: 1. The updater memory limit was 110Mi. The updater watches every pod in the cluster, so on a full cozystack install it OOM-kills, which trips the Deployment's ProgressDeadlineExceeded and makes Flux fail the HelmRelease 'early due to stalled resources' - cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raise the limit to 512Mi (request 200Mi); the recommender is already unlimited. 2. The two VPA-for-VPA VerticalPodAutoscaler objects ran updateMode: Auto, which evicts the VPA recommender(s) during install and adds pod churn on an already busy cluster. Switch to Initial, matching the existing monitoring/etcd VPA->Initial fixes. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
22c96d8dac |
fix(kubernetes): default worker disk storageClass to the application storageClass
Worker node VMs set evictionStrategy: External, so they live-migrate. Live migration requires the system DataVolume on a ReadWriteMany volume, and linstor-csi only grants RWX on a DRBD-backed StorageClass. When a node group leaves storageClass empty the template previously omitted the field and fell through to the management cluster default StorageClass, which may be a non-DRBD class the CSI driver rejects for RWX — breaking migration. Fall back to the application-level storageClass (the replicated/DRBD class) when the node group storageClass is empty, instead of the cluster default. When both are empty the field is still omitted, preserving the explicit cluster-default opt-out. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
cbc314a8c5 |
fix(kubevirt-cdi): order behind cozystack-basics so cdi-clone-dv RoleBinding finds cozy-public
The kubevirt-cdi default variant only depended on cozystack.networking, which reconciles long before cozystack-basics provisions the cozy-public namespace. kubevirt-cdi ships the cdi-clone-dv RoleBinding into cozy-public, so on a fresh install the RoleBinding can race namespace creation and fail with `namespaces "cozy-public" not found`. Flux retries normally mask it, but under slow reconcile it surfaces as a hard UpgradeFailed. Add a cozystack.cozystack-basics edge so the namespace exists first, mirroring the backupstrategy-controller ordering. No cycle: cozystack-basics' own dependencies (tenant-application, cozystack-engine, gateway-api-crds) do not reach kubevirt-cdi. Pin the edge with a helm-unittest regression guard. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
c725a41f3a |
feat(tenant): recursive (hierarchical) resource quotas for sub-tenants (#3086)
## What this PR does Adds **hierarchical (recursive) resource quotas** for tenants so a tenant can no longer escalate its allocation by creating sub-tenants. Today `tenant.spec.resourceQuotas` becomes a plain per-namespace `ResourceQuota`, which ignores the tenant tree: a super-admin inside a quota'd tenant can create a sub-tenant with a larger quota — or no quota at all — and consume more than was allocated. A tenant's declared quota is now treated as the budget for its **whole sub-tree**: - A child that declares its own quota **carves a fixed slice** out of the parent's remaining budget (parent 10 CPU → child 4 CPU → parent has 6 left). - A child **without** a quota **shares** the parent's remaining pool rather than being unbounded. - A child quota that exceeds the parent's remaining budget is **rejected**. Two cooperating layers, mirroring OpenShift's ClusterResourceQuota split (admission + reconciliation): 1. **Declaration-time gate** in the aggregated apiserver (`apps.cozystack.io` Tenant create/update): deterministically rejects a sub-tenant whose declared quota exceeds the parent's remaining budget. 2. **Runtime enforcer** in `cozystack-controller`: maintains a per-namespace `tenant-quota-allocated` ResourceQuota that clamps each pool member to its share of the parent budget, aggregating sub-tree usage so members collectively stay within the budget. Kubernetes enforces the most restrictive ResourceQuota in a namespace, so this binds without fighting Flux over the chart-rendered `tenant-quota`. Overcommitted pools (a parent quota lowered below the sum of its children) emit a `QuotaOvercommitted` event. A `--tenant-quota-buffer-percent` flag temporarily inflates pool budgets during rollout so workloads already over a freshly-introduced quota keep running. The model and the selector-spans-namespaces / aggregated-usage approach are adapted from OpenShift's ClusterResourceQuota controllers (`openshift/api`, `library-go`, `cluster-policy-controller`, `apiserver-library-go`), Apache-2.0, Copyright Red Hat, Inc. and the OpenShift Authors; attribution is preserved in the source. The per-namespace quota arithmetic reuses the upstream Kubernetes quota helpers (`k8s.io/apiserver/pkg/quota/v1`). No API types or the tenant chart change; the feature works with the existing `tenant.spec.resourceQuotas` field. ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(tenant): tenant resource quotas are now hierarchical — a sub-tenant's quota is carved out of its parent's remaining budget and can no longer exceed it; sub-tenants without a quota share the parent's pool. Set `--tenant-quota-buffer-percent` on cozystack-controller for a temporary grace buffer during upgrades. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added hierarchical tenant quota enforcement across tenant namespaces, including shared pool handling and configurable quota buffer support. * Tenant quota changes are now validated on create and update to prevent over-allocation. * **Bug Fixes** * Improved quota calculations so usage, carve-outs, and nested tenant limits are applied more consistently. * Stale quota allocations are now cleaned up when namespaces no longer belong to an active tenant pool. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
5b26ec3462 |
fix(mariadb): clean up orphaned operator-generated secrets (#3056) (#3071)
## What this PR does After deleting a `MariaDB` app CR with metrics and/or replication enabled, two Secrets created by the mariadb-operator were left orphaned in the tenant namespace: - `<release>-metrics-password` - `<release>-repl-password` These are generated by the operator (`k8s.mariadb.com/v1alpha1`) as a side effect of `metrics.enabled` / `replication.enabled`, so they are **not** Helm-owned and survive `helm uninstall`. The existing post-delete hook (`templates/hooks/cleanup-pvc.yaml`) only swept PVCs, leaving the Secrets behind. This PR extends that hook to also clean them up: - grants the cleanup `Role` RBAC on `secrets`; - deletes both password Secrets by name with `--ignore-not-found`; - adds a `helm-unittest` suite locking in the behaviour, plus a `test` Makefile target so the package's tests actually run in CI (`hack/helm-unit-tests.sh` only runs packages that define a `test` target). Fixes #3056. ### Release note ```release-note fix(mariadb): garbage-collect operator-generated metrics-password / repl-password Secrets when a MariaDB app is deleted ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved MariaDB uninstall cleanup to remove leftover operator-generated password Secrets in addition to persistent volumes/claims. * Hardened the cleanup hook Job with stricter pod/container security settings, least-privilege Secret permissions, idempotent deletions for missing resources, and correct non-zero exit on real failures. * **Tests** * Added a cleanup hook test suite validating Secret deletion behavior, failure propagation, and that cleanup RBAC grants least-privilege access only to the required Secrets. * Added a Helm unit test `test` target for the MariaDB chart. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d988b3e8c8 |
fix(apiserver): read tenant pool quota usage via direct client and grant RBAC
parentPoolUsage listed ResourceQuotas through the aggregated apiserver's cached client, which lazily starts a cluster-wide ResourceQuota informer. cozystack-api has no RBAC to list resourcequotas, so the informer never synced and tenant create/update of a bounded child hung until the request timed out (observed on a cluster). Read quota usage through the direct (uncached) watch client instead, scoped to the pool's member namespaces, so no cluster-wide informer is started, and grant cozystack-api get/watch/list on resourcequotas for those reads. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
844deb170f |
feat(controller): enforce recursive tenant quotas at runtime
Add a tenant quota reconciler to cozystack-controller that realises the hierarchical (sub-tree) quota model at runtime. Each tenant's declared quota is the budget for its whole sub-tree: a child that declares its own quota carves a fixed slice out of the parent budget, while a child without a quota shares the parent's remaining pool. The controller maintains a per-namespace 'tenant-quota-allocated' ResourceQuota (Kubernetes enforces the most restrictive quota, so it binds without fighting Flux over the chart-rendered 'tenant-quota'), aggregating sub-tree usage so a pool's members collectively stay within the budget. A --tenant-quota-buffer-percent flag temporarily inflates pool budgets so workloads already over a freshly-introduced quota keep running during rollout; overcommitted pools emit a QuotaOvercommitted event. The model and the selector-spans-namespaces/aggregated-usage approach are adapted from OpenShift's ClusterResourceQuota controllers (openshift/api, library-go, cluster-policy-controller, apiserver-library-go), Apache-2.0, Copyright Red Hat, Inc. and the OpenShift Authors. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
eab9b034f6 |
fix(seaweedfs): enforce read-only bucket access by bumping cosi-driver to v0.3.1 (#3136)
## What this PR does The SeaweedFS COSI driver shipped at `v0.1.2` hardcodes read-write S3 actions (`Read,Write,List,Tagging`) for every BucketAccess and never reads the `accessPolicy` BucketAccessClass parameter. The chart already defines a `-readonly` BucketAccessClass carrying `parameters.accessPolicy: readonly`, and the `bucket` app routes `readonly: true` users to it — but `v0.1.2` ignores the parameter, so a credential meant to be read-only is silently issued with full read-write access and can upload and delete objects. This is an access-control bug. Upstream `seaweedfs/seaweedfs-cosi-driver` added `accessPolicy` handling in `v0.3.0`: `readonly` maps to `Read,List`, while `readwrite` and the empty default stay `Read,Write,List,Tagging`. This bumps the driver to `v0.3.1` via a `seaweedfs.cosi.image` override in the package `values.yaml`, so the override survives `make update` and the vendored chart is left untouched. The default read-write action set is unchanged, so existing read-write credentials are unaffected; only read-only grants are tightened to actually be read-only. The `extra/seaweedfs` chart already runs `v0.3.1` with the same objectstorage-sidecar, so the driver/sidecar combination is already proven in this repo. A chart unit test pins the rendered driver image so this fix cannot silently regress on a future re-vendor. This does not close #1921. That issue bundles two features — read-only bucket access (fixed here) and Object Lock / WORM (the `-lock` BucketClass), which is independent and out of scope here. Relates to #1921. ### Release note ```release-note Fix SeaweedFS read-only bucket credentials being issued with read-write access; the COSI driver now honours the read-only access policy. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Pin the SeaweedFS COSI driver image to `ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1` to keep access behavior consistent and avoid relying on a vendored default. * Ensure the rendered SeaweedFS COSI deployment uses the expected container name and the pinned driver image. * **Tests** * Add a YAML test suite to validate the generated COSI deployment template matches the expected container name and pinned image. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
bef42229d4 |
fix(harbor): clean up orphaned jobservice PVC and ACME solver resources on delete (#3061) (#3077)
## What this PR does After deleting a `Harbor` app CR, two things were left orphaned in the tenant namespace: - the **jobservice / trivy PVCs** (`<release>-jobservice`, and `<release>-trivy` when Trivy is enabled), because `packages/apps/harbor/templates/harbor.yaml` sets `persistence.resourcePolicy: "keep"`, which annotates the upstream Harbor subchart's PVCs with `helm.sh/resource-policy: keep` so Helm/Flux deliberately never deletes them; - cert-manager's **ACME http01-solver** resources (`cm-acme-http-solver-*`), via the same stuck-challenge mechanism as the Bucket issue (#3060) when Harbor issues a non-wildcard cert. This adds a single `post-delete` cleanup hook that: - **deletes the upstream Harbor subchart's kept PVCs** — selecting by `app.kubernetes.io/instance=<release>-system` (the subchart runs as the `<release>-system` HelmRelease, so its PVCs carry that label, **not** `<release>`). A hook is required precisely because `resourcePolicy: keep` blocks normal reclaim. This intentionally covers both the jobservice and trivy PVCs: both hold only regenerable data (jobservice file logs; trivy's vulnerability-DB cache), and registry images live in S3. - **deletes this release's `Certificate`** (deterministic name `<release>-ingress-tls`); cert-manager then cascade-deletes the `Order`, `Challenge` and solver `Pod`/`Service`/`Ingress` objects it owns via owner references. - **host-scoped belt-and-braces sweep** for any ACME http01 solver objects cert-manager may have orphaned (e.g. the Certificate was already gone, so deleting it did not cascade). The sweep is scoped **strictly to this release's Harbor host** — Challenges filtered by `.spec.dnsName`, solver Ingresses by `.spec.rules[0].host` — **not** a namespace-wide `acme.cert-manager.io/http01-solver=true` label sweep, because a Cozystack tenant namespace can host many apps and a blanket label sweep would delete sibling apps' in-flight challenge solvers. The Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`, with CPU/memory requests and limits set. RBAC is scoped to exactly the resources it deletes (`persistentvolumeclaims`, `certificates`, `orders`, `challenges`, solver `ingresses`) and verified against the vendored cert-manager CRDs. Failures are logged but never block teardown. ## Cleanup scope — what is and isn't removed **Removed:** the upstream Harbor subchart's kept PVCs (jobservice + trivy) and this release's cert-manager ACME artifacts (Certificate, and host-scoped Order/Challenge/solver-Ingress). **Not touched (by design):** - **CNPG database PVCs** (`<release>-db-N`) — owned by the CNPG `Cluster` and governed by CNPG's own reclaim policy; they carry CNPG labels, not `app.kubernetes.io/instance=<release>-system`. Re-installing Harbor with a different DB size into the same namespace can therefore leak the old DB PVCs; this is left to CNPG's lifecycle on purpose (the data is the user's). - **Redis PVCs** — managed by the redis-operator with their own labels, likewise untouched. - **Solver `Pod`/`Service`** — not deleted directly; they are removed by cert-manager's owner-ref cascade when the Certificate/Challenge is deleted, so the hook grants no RBAC for them. > [!NOTE] > The PVC sweep is eventually-consistent: while the inner `<release>-system` > HelmRelease is still being uninstalled by Flux, a PVC may briefly stay in > `Terminating` until its consumer Pods (jobservice/trivy) exit. The Job's > deletion request always succeeds; reclaim completes once the pods are gone. > [!WARNING] > **This overrides the intentional `persistence.resourcePolicy: "keep"` setting > and force-deletes the jobservice/trivy PVCs on app deletion.** Those PVCs hold > only jobservice file logs and the trivy vuln-DB cache (registry images live in > S3), so data-loss risk is low — but if retaining them across re-installs is > desired, this is a conscious product decision to revisit. The ACME cleanup > affects only cert-manager's transient artifacts. Fixes #3061. > Follow-up tracked in #3088: digest-pin the cleanup `kubectl` image and route it > through `cozy-lib.images-registry` (pre-existing debt shared with the MariaDB > cleanup hook), out of scope for this PR. ### Release note ```release-note fix(harbor): clean up the orphaned jobservice/trivy PVCs and cert-manager ACME http01-solver resources left behind after a Harbor app is deleted. NOTE: this force-deletes the jobservice/trivy PVCs, overriding persistence.resourcePolicy=keep (jobservice file logs and trivy vuln-DB cache only; registry images are in S3). CNPG database PVCs are not touched. ``` |
||
|
|
3e20a0cd41 |
fix(kubernetes): make node group resources optional when instanceType is set (#3121)
## What this PR does
A tenant Kubernetes node group can be sized two ways: by `instanceType`
or by an explicit `resources` block. The generated values schema,
however, marked `resources` as required on every node group, so a node
group sized purely by `instanceType` was rejected at install time:
```text
at '/nodeGroups/<name>': missing property 'resources'
```
The same failure occurs through the typed `Kubernetes` CR: an empty
`resources: {}` is dropped during the CR-to-HelmRelease translation, so
by validation time the key is absent and the required check fails — even
though sizing was already fully specified by `instanceType`.
This marks `resources` optional in the node group definition and
regenerates the dependent artifacts (`values.schema.json`, API types,
README, and the `ApplicationDefinition`). No change is needed to emit
the VM: the chart already renders `domain.cpu`/`domain.memory` only when
`resources` is present and `instancetype` only when set, so a node group
sized purely by `instanceType` now validates and installs. Node groups
that specify `resources` are unaffected.
`instanceType` and `resources` are alternative, mutually exclusive
sizing sources — KubeVirt rejects a VirtualMachine that references an
instancetype and also sets explicit `domain.cpu`/`domain.memory`, since
instancetype CPU/memory cannot be overridden. To avoid emitting a VM
that the instancetype admission webhook would later reject, the chart
now fails fast at render time with a clear message when a node group
sets both a non-empty `instanceType` and explicit `resources`.
Tests: helm-unittest coverage proves a node group with `instanceType`
and no `resources` passes schema validation and renders without explicit
`domain.cpu`/`memory`; a complementary case confirms explicit
`resources` are still honoured; and a `failedTemplate` case pins the
render-time rejection of the conflicting both-set combination. The
omit-resources case fails against the pre-fix schema with the exact
error above, so it is a real regression guard.
Addresses #3046.
### Screenshots
N/A — schema and chart change, no UI impact.
### Release note
```release-note
fix(kubernetes): allow a node group to omit `resources` when it is sized by `instanceType` (previously failed schema validation with "missing property 'resources'"); reject the unsupported combination of `instanceType` and explicit `resources` at render time
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Worker node groups can optionally size CPU/memory via `resources`, and
can omit `resources` when using `instanceType`.
* **Bug Fixes**
* Render now validates CPU/memory rules: `instanceType` with only one of
`resources.cpu`/`resources.memory` fails, and `instancetype` is omitted
when both `resources.cpu` and `resources.memory` are set (with
`instanceType` ignored).
* **Documentation**
* Updated API/Helm docs and JSON schema to clarify `resources` vs
`instanceType` precedence and the “set both CPU and memory (or neither)”
requirement.
* **Tests**
* Added helm-unittest scenarios covering omission, precedence, and
render-time failure cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
0892c5a38c |
fix(qdrant)!: clean up orphaned data PVC on delete (#3059) (#3075)
## What this PR does After deleting a `Qdrant` app CR, the StatefulSet data PVC (`qdrant-storage-<release>-0`) was left orphaned in the tenant namespace, consuming storage permanently. StatefulSet-templated PVCs are never garbage-collected on StatefulSet deletion by default. The vendored upstream Qdrant chart does **not** support `persistentVolumeClaimRetentionPolicy` (verified: its `statefulset.yaml` never renders that field), so the OpenBAO-style value passthrough is not possible here. Instead this adds a `post-delete` PVC cleanup hook (modeled on the MariaDB hook) that deletes the release's PVCs by label. The chart's volumeClaimTemplate only stamps `app: qdrant` (shared by every Qdrant in a namespace), so to select PVCs safely per release this also passes `persistence.additionalLabels.app.kubernetes.io/instance: <release>` through the HelmRelease — giving the data PVC a release-unique label the hook selects on. The cleanup Job is hardened: runs as non-root (uid 65534), read-only root fs, all capabilities dropped, `seccompProfile: RuntimeDefault`; RBAC scoped to `persistentvolumeclaims`. A real cleanup failure is logged (not silently swallowed) but never blocks namespace teardown. > [!WARNING] > **Deleting a Qdrant app now permanently removes its data PVC and the vector data it holds.** > This is intentional and fixes the storage leak; the hook only fires on full > release deletion (not on scale-down). Back up before deleting. Fixes #3059. ### Release note ```release-note fix(qdrant): garbage-collect the StatefulSet data PVC when a Qdrant app is deleted, instead of leaving it orphaned. WARNING: deleting a Qdrant app now permanently removes its data PVC and the data it holds. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a post-delete cleanup hook for Qdrant to remove leftover StatefulSet data PVCs after deletion. * Included hardened Kubernetes settings for the cleanup job to improve runtime safety. * **Tests** * Added Helm chart tests that validate the post-delete cleanup job and its RBAC permissions. * Enhanced end-to-end verification to confirm the data PVC is fully removed after deleting a Qdrant instance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
bf14a73b39 |
feat(api): add SecurityGroup network policy resource (sdn.cozystack.io) (#2922)
## What this PR does
Introduces a tenant-facing `SecurityGroup` resource in a new
`sdn.cozystack.io/v1alpha1` API group, served by the Cozystack
aggregated API server as a 1:1 projection of a `CiliumNetworkPolicy`. A
SecurityGroup attaches to one of the tenant's managed applications by
reference (`spec.targetRef: {apiGroup, kind, name}`) and declares
allow-list ingress/egress for that application. Tenants manage network
policy for their own apps entirely through
`securitygroups.sdn.cozystack.io`, without being granted any access to
the `cilium.io` API group.
The translation is synchronous and stateless — the API server converts
each SecurityGroup into a CiliumNetworkPolicy in the same namespace on
write and back on read; there is no controller. The backing policy's
`endpointSelector` is **derived** from `targetRef` (matched on the
application's lineage labels
`apps.cozystack.io/application.{group,kind,name}`), never copied from
tenant input, so the selector is machine-generated and can only ever
match the referenced application's own pods in the tenant's namespace.
Policies owned by this API carry an `sdn.cozystack.io/securitygroup`
marker label that the storage always re-asserts and hides from the view,
so it only ever surfaces or mutates its own policies and never touches
platform-managed network policies.
The value is self-service network policy on a Cozystack-native,
RBAC-scoped surface, without exposing Cilium's version-coupled CRDs to
tenants. Because the selector is derived from an authorized application
reference rather than free-form tenant input, the RBAC boundary is
structural, not selector-validated: a tenant cannot express network
policy against arbitrary or platform-owned pods. The full design —
selector derivation from lineage labels, the structural boundary, the
additive-policy reality (a SecurityGroup widens a managed application's
allowed traffic but cannot sever its platform-managed management plane,
and equally cannot tighten below the platform baseline), the in-tree
CiliumNetworkPolicy mirror that avoids a Cilium Go-module dependency,
RBAC, and deferred extensions (raw-pod targeting, cluster scope,
reusable groups) — is in `pkg/apis/sdn/DESIGN.md`.
Implements the attachment model from #1614.
### Release note
```release-note
feat(api): add the SecurityGroup resource (sdn.cozystack.io/v1alpha1) — a tenant-facing, namespace-scoped firewall that attaches to a managed application by reference and projects 1:1 onto a CiliumNetworkPolicy with a machine-derived endpoint selector, letting tenants manage their application's network policy without cilium.io access.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added the `SecurityGroup` API resource (`sdn.cozystack.io` v1alpha1)
with a namespaced projection backed by Cilium network policies.
* Enabled the SDN `APIService` and wired SDN routing, scheme, and REST
support.
* Expanded RBAC to include SecurityGroup access and required Cilium
policy permissions.
* **Documentation**
* Added a design document describing projection behavior and rule
semantics.
* **Bug Fixes**
* Improved API rule validation coverage for missing list-type fields.
* **Tests**
* Added SDN round-trip, fuzzing, RBAC, and comprehensive REST
create/get/list/update/delete/watch/resource-version tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|