Commit Graph
2919 Commits
Author SHA1 Message Date
Matthieu adb307cbae [goldpinger] Set resources requests/limits on goldpinger DaemonSet
The Bloomberg goldpinger chart ships with resources: {}, which triggers
a 'no CPU limit' conformance warning on the goldpinger-daemon container
deployed on every node. Override the package values.yaml to set
limits 200m/256Mi and requests 50m/64Mi — comfortable margin for the
ping-mesh fan-out on larger clusters while keeping a small baseline
request.

Signed-off-by: Matthieu <matthieu@hidora.com>
2026-07-03 19:14:27 +03:00
IvanHunters 1baf126b1a fix(tenant): inherit full ancestor label chain from parent namespace (#2912)
## Summary

`packages/apps/tenant/templates/namespace.yaml` labels every tenant
namespace
with `tenant.cozystack.io/<ancestor>` for each of its ancestors. These
labels
drive the `<ancestor>-egress` `CiliumClusterwideNetworkPolicy`
(`networkpolicy.yaml`): an ancestor tenant's pods may reach a namespace
only if
that namespace carries `tenant.cozystack.io/<ancestor>`.

The chain was derived by splitting `.Release.Namespace`, which only ever
reached
one level up. A direct child of `tenant-root` got `tenant-root` (its
parent
namespace *is* `tenant-root`), but a grandchild — `tenant-ktj-htdev`
created
under `tenant-ktj` — got only `tenant-ktj` and dropped `tenant-root`
entirely.
Without `tenant.cozystack.io/tenant-root` on a nested namespace, the
root
ingress controller cannot reach the proxy service in a nested
`kubernetes`
tenant, surfacing as `upstream timed out (110) while connecting to
upstream` in
the nginx logs.

## Fix

Derive the ancestor chain deterministically from the namespace name
instead of
from any cluster lookup. Every tenant descends from `tenant-root`, so
that label
is emitted unconditionally; the remaining ancestors are encoded in the
namespace
name itself, because `tenant.name` builds each child as
`<parent-namespace>-<word>` — so every progressive dash-prefix of the
name is a
real ancestor namespace (`tenant-ktj-htdev` yields `tenant-ktj` and
`tenant-ktj-htdev`). The logic lives in a new
`tenant.ancestorTenantLabels`
helper:

```yaml
labels:
  {{- include "tenant.ancestorTenantLabels" $tenantName | nindent 4 }}
```

This needs no cluster state, renders identically offline (so it is
unit-testable), and converges on every reconcile regardless of the order
in
which parent and child `HelmRelease`s reconcile — unlike a
parent-namespace
`lookup`, which helm-controller does not re-render on and which is
ordering-dependent. The `ownerReference` `lookup` is kept but made
non-fatal:
during a real reconcile the parent namespace always exists (the tenant
`HelmRelease` and its `cozystack-values` Secret live in it), so the
`ownerReference` is always emitted; when the lookup is empty (offline /
unit
test) it is simply omitted so the labels still render.

Migration `49` backfills the labels on existing clusters so they heal on
upgrade
regardless of per-tenant `HelmRelease` state (e.g. suspended or failed
releases), and `migrations.targetVersion` is bumped to `50` so the
runner
iterates it. The migration derives the same chain from each namespace
name,
is idempotent (`kubectl label --overwrite`) and best-effort
(per-namespace
failures are tolerated so a transient apiserver error cannot abort the
upgrade).

## Test plan

- [x] `helm template packages/apps/tenant` at depth 1/2/3 — emits
exactly the
expected ancestor set at each depth: depth-1 `{tenant-root,
tenant-ktj}`,
depth-2 adds `tenant-ktj-htdev`, depth-3 adds `tenant-ktj-htdev-foo`. No
      missing, no stray labels.
- [x] `helm unittest packages/apps/tenant` — new
`tests/namespace_ancestor_labels_test.yaml` pins the chain at depths
1/2/3
      plus a negative assertion; whole suite green.
- [x] `bats hack/cozystack-version-stamp.bats` — migration `49` stamps
the
version through the shared `stamp_cozystack_version` helper (all 10
tests
pass, including the guard that migrations >= 42 must use the helper).
- [x] Migration `49` ancestor derivation verified under `/bin/sh` to
match the
helm output byte-for-byte, and dry-runs cleanly under `set -euo
pipefail`.
- [ ] Live cluster: create a 3-level tenant (`root -> ktj -> htdev`),
verify
      `tenant-ktj-htdev` has all three `tenant.cozystack.io/*` labels.
- [ ] Live cluster: create a `kubernetes` tenant under `htdev` with
`addons.ingressNginx.hosts`, verify an external HTTP request reaches the
tenant ingress controller via the root-ingress proxy (no `upstream timed
      out`).

## Closes

- Closes #2810
- Closes #2171

```release-note
fix(tenant): tenant namespaces now carry the full `tenant.cozystack.io/<ancestor>` label chain (including `tenant-root`) at any nesting depth, restoring root-ingress reachability to nested tenants; a migration backfills the labels on existing clusters
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved tenant namespace label inheritance across multi-level tenant
hierarchies, ensuring all ancestor tenant labels are consistently
applied.
* **New Features**
* Standardized ancestor label generation to reliably include the full
tenant chain for derived tenant namespaces.
* **Tests**
* Added regression coverage for ancestor label presence and absence at
multiple hierarchy depths.
* **Chores**
* Added a migration to backfill missing ancestor labels for existing
tenant namespaces during platform upgrades.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-03 16:12:24 +03:00
myasnikovdaniil 1e5f53881f fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) (#3146)
## What

Backports the upstream Kamaji fix
[clastix/kamaji#1122](https://github.com/clastix/kamaji/pull/1122)
(which fixes clastix/kamaji#1115) as a source patch on top of the pinned
`26.3.6-edge` build. No version bump, no CRD/RBAC changes.

This lands the fix on `main`; the `backport` / `backport-previous`
labels fan it out to `release-1.5` and `release-1.4` via the backport
automation. Supersedes the per-release PRs #3021 (release-1.5) and #3020
(release-1.4).

## Why

Deleting a Tenant whose etcd `DataStore` has **0 referencing
`TenantControlPlane`s** — e.g. a tenant deleted before any Kubernetes
app / TCP is ever created in it — leaves the `DataStore` stuck
`Terminating` with finalizer `kamaji.clastix.io/TenantControlPlane`.
That hangs `helm uninstall <etcd>` (which waits by default), freezes the
etcd HelmRelease at `status=uninstalling`, and wedges the tenant
namespace in `Terminating` indefinitely. Tracked in #3018.

Root cause: in `26.3.x-edge`, DataStore deletion drives an
`AllowedDeletion` condition that only transitions `False → True`. With 0
referencing TCPs at deletion time the condition is never initialized, so
the reconcile falls through to logging `DataStore can be safely deleted`
and the deferred finalizer-removal (gated on `IsStatusConditionTrue`)
never runs — with no requeue to retry. The pinned `26.3.6-edge` predates
the upstream fix (merged 2026-04-14, first released in `26.4.4-edge`).

## Change

Adds
`packages/system/kamaji/images/kamaji/patches/fix-datastore-unused-deletion.diff`,
applied by the existing `RUN git apply /patches/*.diff` step in the
kamaji image build. The one-line guard adds the missing 0-TCP path:

```go
- if meta.IsStatusConditionFalse(..., DataStoreConditionAllowedDeletionType) {
+ if meta.IsStatusConditionFalse(..., DataStoreConditionAllowedDeletionType) || len(tcpList.Items) == 0 {
```

## Test

- `git apply --check` confirms the patch applies cleanly to the
`26.3.6-edge` source pinned on `main`; after applying, the guard is
present at `controllers/datastore_controller.go`.
- Reproduced the deadlock on a v1.4.x cluster (tenant deleted before any
TCP existed) and confirmed the manual unstick; this patch removes the
need for it.

Workaround for an already-stuck DataStore (0 TCP refs): `kubectl patch
datastore <name> --type=merge -p '{"metadata":{"finalizers":null}}'`.

Relates to #2885 (full kamaji bump for CVEs — supersedes this patch once
it lands).

```release-note
Fix tenant namespace stuck `Terminating` when an etcd DataStore with no referencing TenantControlPlanes is deleted (Kamaji datastore unused-deletion deadlock).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Data stores are now recognized as unused even when no matching tenant
control plane resources are found, improving deletion handling in that
case.
* This prevents inactive data stores from being left behind when there
are no associated matches.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-03 14:52:11 +05:00
Myasnikov Daniil 47638f8569 fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122)
Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes left the DataStore stuck Terminating (finalizer kamaji.clastix.io/TenantControlPlane), hanging the etcd helm uninstall and wedging the tenant namespace. Backports clastix/kamaji#1122 (fixes clastix/kamaji#1115) as a source patch on the pinned 26.3.6-edge build. Refs #3018.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-03 12:59:38 +05:00
IvanHunters 59964043e3 fix(platform): migration 49 backfills tenant ancestor labels
The tenant chart fix makes new and re-reconciled tenant namespaces
self-compute their full tenant.cozystack.io/<ancestor> chain, but existing
namespaces whose HelmRelease does not promptly re-reconcile (suspended or
failed releases) keep the incomplete labels — and with them the broken
<ancestor>-egress reachability that surfaces as upstream timeouts to nested
tenants. Backfill the labels directly so clusters heal on upgrade regardless
of per-tenant HelmRelease state.

Bump targetVersion to 50 so run-migrations.sh iterates the new script
(seq CURRENT..49); the missing-file guard otherwise never fires. Best-effort
and idempotent: kubectl label --overwrite adds only the listed labels, and
per-namespace failures are tolerated so a transient apiserver error cannot
abort the platform upgrade.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-03 09:54:12 +03:00
IvanHunters 1ac9398a4b fix(tenant): emit full ancestor label chain on tenant namespaces
The tenant.cozystack.io/<ancestor> namespace labels drive the
<ancestor>-egress CiliumClusterwideNetworkPolicy, which lets an ancestor
tenant reach a descendant namespace only when the descendant carries
tenant.cozystack.io/<ancestor>. namespace.yaml derived that chain by
splitting .Release.Namespace, which only reached one level up: a direct
child of tenant-root got tenant-root, but a grandchild (tenant-ktj-htdev
under tenant-ktj) got only tenant-ktj and dropped tenant-root entirely.
The root ingress controller in tenant-root then could not reach nested
tenants, surfacing as upstream timeouts.

Derive the chain deterministically from the namespace name instead. Every
tenant descends from tenant-root, and tenant.name builds each child as
<parent-namespace>-<word>, so every progressive dash-prefix of the name is
a real ancestor namespace. This needs no cluster lookup, renders offline
(so it is unit-testable), and converges on every reconcile regardless of
parent/child HelmRelease ordering. The ownerReference lookup is made
non-fatal so the template still renders when the parent namespace lookup
is empty; in a real reconcile the parent always exists and the
ownerReference is always emitted.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-03 09:51:49 +03:00
Myasnikov DaniilandClaude Opus 4.8 2b1d3e904d fix(kubernetes): gate vmop cert-manager-crds dependency on certManager.enabled
The victoria-metrics-operator HelmRelease is gated on
monitoringAgents.enabled but listed cert-manager-crds in dependsOn
unconditionally, while the cert-manager-crds HelmRelease is only
created when certManager.enabled is true. The valid combination
monitoringAgents.enabled=true + certManager.enabled=false therefore
left vmop blocked on a non-existent dependency
("...cert-manager-crds ... not found"), cascading to monitoring-agents
and vertical-pod-autoscaler. Gate the dependency on the same condition
that creates it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-03 11:18:46 +05:00
scooby87 71ca2b1bae fix(mariadb): re-guard replication so single-replica passes the operator webhook (#3180)
## What this PR does

Fixes a regression where **single-replica MariaDB can no longer be
created**.

[PR #2279](https://github.com/cozystack/cozystack/pull/2279) made
`replication.enabled: true` unconditional in the MariaDB CR so the
operator would always create `<release>-primary` / `<release>-secondary`
Services (expected by the dashboard RBAC and the ApplicationDefinition).
But the vendored mariadb-operator's validating webhook
(`vmariadb-v1alpha1.kb.io`) **rejects `replication.enabled: true` when
`spec.replicas == 1`** — replication needs a primary + at least one
replica. As a result, single-replica installs never became Ready (the
HelmRelease failed at admission).

The webhook rule lives in the operator's Go code (not in the vendored
CRD — there is no CEL rule tying replication to replica count), so the
failing path is reproduced by an e2e test rather than a schema
assertion.

### Fix — re-guard replication, teach consumers about the bare service

The operator provisions a bare `<release>` Service in **both**
topologies; `-primary`/`-secondary` exist **only** when replication is
on (`replicas>1`). Instead of forcing replication for everyone, this PR
lets replication track the replica count and makes the service-name
consumers handle the single-replica bare service:

- **`mariadb.yaml`** — gate `replication` on `replicas>1`; restore
single-replica external access via `service.type: LoadBalancer`, and
keep `primaryService: LoadBalancer` for `replicas>1` only. (Effectively
reverts #2279's two hunks.)
- **`backup-cronjob.yaml`** — single-replica dumps target the bare
service (no `-secondary` exists without replication); `replicas>1` still
offloads to `-secondary`.
- **`dashboard-resourcemap.yaml`** — always grant RBAC on the bare
service, plus `-primary`/`-secondary` when `replicas>1`, matching what
the ApplicationDefinition lists.
- **`mariadb-rd`** — list the bare service in `services.include` so
single-replica installs are visible in the dashboard.
- **`hack/e2e-apps/mariadb.bats`** — new `Create single-replica MariaDB`
test that reproduces the webhook rejection and guards against
regression.

Verified: `helm template` at `replicas=1`/`2` renders the intended
CR/RBAC/backup host; `helm unittest` passes; `update-crd.sh` keeps the
RD stable; `select-e2e.sh` maps the change to the mariadb suite.

### Release note

```release-note
fix(mariadb): single-replica instances can be created again — replication is now enabled only for replicas>1 (matching the operator webhook), while the dashboard, RBAC, and backups correctly target the bare service for single-replica setups
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an end-to-end test covering MariaDB deployments with a single
replica, including validation of the main service, endpoints,
StatefulSet replica count, and metrics components.

* **Bug Fixes**
* Updated MariaDB chart behavior to correctly handle single-replica vs
multi-replica configurations for external access, replication settings,
and backup host selection.
* Refined dashboard RBAC and secret inclusion so access applies to the
main MariaDB service in addition to replica-specific services.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 22:54:14 +03:00
Andrei Kvapil 41fd08e2ba feat(keycloak): enable UPDATE_EMAIL required action in cozy realm (#3175)
## What this PR does

The portal user-settings page lets a user change their email via a
Keycloak `kc_action=UPDATE_EMAIL` redirect. That flow only works when
the **UPDATE_EMAIL required action is enabled at the realm level** — and
on a fresh realm it is registered but **disabled by default**.

`update-email` is a supported, default-on feature since Keycloak 26.4
(keycloak/keycloak#39722), so the provider is already loaded on the 26.x
server this repo ships — **no `KC_FEATURES` change and no Keycloak
restart are needed**. The `edp-keycloak-operator` has no CRD field for
required actions, so this adds a small one-shot, idempotent
`post-install`/`post-upgrade` Job to `keycloak-configure` that enables
it via `kcadm.sh`, waiting for the operator to reconcile the realm
first.

Verified against a live Keycloak 26.5.2: the `UPDATE_EMAIL` action goes
from `enabled=false` to `enabled=true`, and re-running the Job is a
no-op.

### Release note

```release-note
feat(keycloak): enable the UPDATE_EMAIL required action in the cozy realm so users can change their email from the portal
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Keycloak now automatically enables the email update action during
install and upgrade, helping users change their email address in the
`cozy` realm.
* The configuration now includes the required Keycloak container image
for this setup task.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 21:02:18 +02:00
scooby87 e4c6802863 fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry (#3171)
## What this PR does

Fixes #3088.

The post-delete cleanup Jobs in the **harbor** and **mariadb** charts
hardcoded the cleanup image as `docker.io/clastix/kubectl:v1.32` — a
moving Docker Hub tag. This:

- **broke reproducibility / was a supply-chain risk** — `v1.32` is
mutable and can be re-pushed to a different image;
- **could not be satisfied on air-gapped / mirrored clusters** — the
hardcoded `docker.io` bypassed the cluster images registry, and it
failed at the worst moment: during **uninstall**, when the `post-delete`
hook runs and a stuck Job stalls release teardown.

### Changes

- **New canonical helper `cozy-lib.image`**
(`packages/library/cozy-lib/templates/_cozyconfig.tpl`). Given a
registry-relative image path and the global context, it prefixes
`cozy-lib.images-registry` when set (`<registry>/<image>`) and returns
the image unchanged when empty — **with no leading `/`**, so standard
installs still resolve from the default registry. This establishes the
previously-undefined "how to reference a cluster image" pattern
(`images-registry` was defined but never consumed).
- **Digest-pinned image** in both cleanup hooks:
`clastix/kubectl:v1.32@sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c`
(the immutable multi-arch OCI index for the `v1.32` tag; the same digest
already pinned for `postgres-operator`'s webhook hook).
- **Renovate keeps the pin fresh.** The repo disables the helm-values
manager (`enabledManagers`), and the ref is assembled at render time by
the helper rather than being a Dockerfile / go.mod dependency, so the
built-in managers can't see it. A `custom.regex` manager over the hook
templates tracks `clastix/kubectl` by tag+digest — mirroring the
existing `extra/etcd` manager — so the supply-chain pin doesn't silently
age.
- **Hardened `cozy-lib.images-registry`** against a nil
`.Values._cluster` so the helper never crashes the render — relevant
precisely because this hook renders during teardown.
- **Helm unit tests** in both charts assert the rendered image for three
cases: empty registry (default), registry set, and the no-leading-slash
edge case. Assertions match by **pattern** (routed + digest-pinned + no
leading slash) rather than an exact digest, so a Renovate bump keeps
them green.

### Verification

- `helm unittest` passes for both charts (harbor: 16, mariadb: 10).
- Real `helm template` renders confirmed for both charts:
  - empty registry → `clastix/kubectl:v1.32@sha256:b9ef…`
- `images-registry=registry.internal:5000` →
`registry.internal:5000/clastix/kubectl:v1.32@sha256:b9ef…`
- The Renovate `matchStrings` regex verified to capture
`currentValue=v1.32` and the digest from both hook templates.

### Note on the wider footprint

The same `docker.io/clastix/kubectl:v1.32` hardcode is already merged on
`main` in several other cleanup hooks (bucket, qdrant, gateway,
seaweedfs, kubernetes, tenant, dashboard, keycloak-configure) and
appears in the open PRs #3170 (etcd), #3094 (monitoring), #3072
(clickhouse). This PR fixes harbor + mariadb per the issue and
establishes the `cozy-lib.image` + Renovate pattern; migrating the
remaining hooks is a follow-up.

### Release note

```release-note
fix(cozy-lib,harbor,mariadb): digest-pin the harbor and mariadb post-delete cleanup-hook kubectl image and route it through the cluster images registry (`cozy-lib.images-registry`) via the new `cozy-lib.image` helper, so mirrored / air-gapped installs can resolve it and uninstall no longer depends on a moving Docker Hub tag; a custom Renovate manager keeps the digest fresh
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Updated post-delete cleanup hook Jobs to use digest-pinned container
images for more secure, repeatable execution.
* Improved image reference rendering so cleanup hooks work correctly
with or without an image registry configured, avoiding invalid
leading/trailing slash issues.
* **Tests**
* Extended cleanup hook rendering tests to validate digest pinning and
correct registry prefix behavior, including edge cases.
* **Chores**
* Enhanced automated dependency detection to treat digest-pinned hook
image references as stable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 21:38:43 +03:00
scooby87 6bf836db30 fix(dashboard): show LoadBalancer external IP on the app Services tab (#3172)
## What this PR does

For an application exposed with `external: true` (which creates a `type:
LoadBalancer` Service), the app detail → **Services** tab showed the
service Type and Cluster IP but not the external IP assigned by MetalLB
— so there was no way to learn the connection address from the per-app
view. The pre-1.4.0 dashboard showed this, so it read as a regression
(reported in #2789).

Root cause: `ServicesTab.tsx` queried Services with a spec-only local
type (`{ type, clusterIP, ports }`) and never read
`status.loadBalancer.ingress[]`. The data was already reachable —
`ExternalIpsPage.tsx` renders it — so this was a wiring gap in the
per-app tab, not missing data.

Changes:
- Add a `ServiceStatus` type and extend the `useK8sList` query to
`K8sResource<ServiceSpec, ServiceStatus>`.
- Add an **External IP** column: `ip ?? hostname ?? "Pending"` for
`LoadBalancer` services, `—` otherwise — mirroring `ExternalIpsPage`.
- Add a co-located regression test (`ServicesTab.test.tsx`) covering the
IP, hostname→Pending fallback, and the ClusterIP no-external case.

Verified: `pnpm typecheck` clean, `pnpm test` green (320 tests, incl. 3
new), `eslint` clean on the changed files.

Fixes #3111

### Screenshots
<img width="1220" height="572" alt="Снимок экрана 2026-07-02 в 14 12 11"
src="https://github.com/user-attachments/assets/82cd1da8-ff81-4b95-a6bf-18729b6c85c7"
/>


### Release note

```release-note
fix(dashboard): show the external LoadBalancer IP on the per-app Services tab
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Added an **External IP** column to the service details view.
* Load balancer services now show the external IP or hostname when
available.

* **Bug Fixes**
* Fixed the external IP display for load balancer services so it no
longer misses valid IPs.
* Improved fallback behavior by showing **Pending** when no external
endpoint is assigned.
* Non-load balancer services now show an em dash instead of misleading
external IP text.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 20:06:26 +03:00
scooby87 a16268152d fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant module delete (#3090) (#3170)
## What this PR does

When the `etcd` tenant module is disabled (`etcd: false`), the etcd
HelmRelease is pruned but two classes of resources leak into the tenant
namespace (reported in #3090):

- **Data PVCs** — the etcd-operator StatefulSet `volumeClaimTemplate`
PVCs `data-etcd-<ordinal>` are never garbage-collected on `EtcdCluster`
deletion.
- **TLS secrets** — the five `etcd-*-tls` secrets survive the uninstall:
- `etcd-ca-tls` / `etcd-client-tls` are created by this chart as
`pre-install` hooks with `helm.sh/resource-policy: keep`, so Helm
deliberately keeps them;
- `etcd-peer-ca-tls` / `etcd-server-tls` / `etcd-peer-tls` are populated
by cert-manager from `Certificate` resources and carry no owner
reference back to the chart.

This adds a `post-delete` cleanup hook
(`packages/extra/etcd/templates/hooks/cleanup.yaml`), modeled on the
accepted seaweedfs (#3092) and monitoring (#3091) sibling fixes, that:

- deletes the data PVCs by name, one per replica (`data-etcd-0 …
data-etcd-<replicas-1>`);
- deletes the five `etcd-*-tls` secrets by name.

`check-release-name` pins the release name to `etcd`, so every target
name is deterministic and no sibling workload shares the `data-etcd-*`
prefix in a tenant namespace — deletion by name is safe. cert-manager
TLS secrets carry only the cluster-wide `controller.cert-manager.io/fao`
label, so the RBAC grants `delete` by explicit `resourceNames` rather
than an unsafe label sweep.

The cleanup Job is hardened: non-root, read-only root fs, dropped
capabilities, `seccompProfile: RuntimeDefault`, resource
requests/limits, `activeDeadlineSeconds`. Deletes use
`--ignore-not-found` and failures are logged but never block teardown.

A Helm unit-test suite (`tests/cleanup_test.yaml`) asserts the hook
metadata/RBAC/hardening and that the rendered commands target exactly
the reported PVCs and secrets, including scaling the PVC list to the
configured replica count.

> [!NOTE]
> The **stuck HelmRelease finalizer** noted in #3090 is the
platform-wide sharded-helm-controller side-bug (tracked separately); it
affects every `chartRef`-based tenant module identically, not just etcd.
This PR fixes the PVC and secret orphans the etcd chart itself owns,
matching the scope of the merged seaweedfs/monitoring fixes.

> [!WARNING]
> **Disabling the etcd module now permanently deletes its data PVCs and
the etcd data they hold.** This is intentional and fixes the storage
leak. Back up before disabling if the data matters.

Fixes #3090.

### Release note

```release-note
fix(etcd): garbage-collect the orphaned etcd data PVCs (data-etcd-*) and the five etcd-*-tls secrets left behind when a tenant's etcd module is disabled. WARNING: disabling etcd now permanently removes its data PVCs and the data they hold.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a Helm post-delete cleanup hook for etcd that removes
per-replica etcd PVCs and the related TLS secrets when a release is
deleted.
* Cleanup permissions and targets now scale with replica count,
including safe handling when replicas are set to zero.
* **Tests**
* Added Helm unit tests validating hook annotations/order, scoped RBAC,
secure job hardening, and the exact deletion behavior for PVCs/secrets
(including higher-replica scenarios).
* **Chores**
  * Marked the `test` make target as phony for the etcd package.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 18:21:36 +03:00
Алексей Артамонов a76c5929fd fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry
The post-delete cleanup Jobs in the harbor and mariadb charts hardcoded
`docker.io/clastix/kubectl:v1.32` — a moving tag pulled directly from
Docker Hub. This broke build reproducibility (the tag can be re-pushed),
was a supply-chain risk, and could not be satisfied on air-gapped /
mirrored clusters, where it failed at the worst moment: during uninstall,
when the post-delete hook runs and stalls release teardown.

Add a canonical `cozy-lib.image` helper that prefixes a registry-relative
image path with `cozy-lib.images-registry` when set and returns it
unchanged (no leading "/") when empty, so standard installs resolve from
the default registry and mirrored installs resolve internally. Consume it
from both cleanup hooks with the image digest-pinned to the immutable
multi-arch index sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c
(the same digest already pinned for postgres-operator's webhook hook).

Keep the pin fresh with a custom Renovate manager: the repo disables the
helm-values manager (enabledManagers), and the ref is built at render time
by the helper rather than being a Dockerfile/go.mod dependency, so the
built-in managers cannot see it. A regex manager over the hook templates
tracks clastix/kubectl by tag+digest — mirroring the existing extra/etcd
manager — so the supply-chain pin does not silently age.

Also harden `cozy-lib.images-registry` against a nil .Values._cluster so
the helper never crashes the render — relevant precisely because the hook
renders during teardown. Helm unit tests assert the rendered image (routed,
digest-pinned, no leading slash) by pattern rather than an exact digest, so
a Renovate bump keeps them green.

Closes #3088

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-02 17:44:59 +03:00
Andrei Kvapil caa6607f4d feat(keycloak): add optional KMS-encrypting database proxy (#3003)
## What this PR does

Adds an optional, flag-gated **column-level PII encryption** feature to
the `keycloak` system package. When `encryption.enabled=true`, a
`keycloak-kms-proxy` Deployment + Service is placed on the wire between
the Keycloak StatefulSet and its CloudNativePG database, and Keycloak's
`KC_DB_URL_HOST`/`KC_DB_URL_PORT` are repointed at the proxy. The proxy
transparently encrypts Keycloak PII columns (username, email, first/last
name, `pii-*` attributes, credential secrets) before they reach
PostgreSQL and decrypts them on reads; the data-encryption key is
wrapped by a KMS and never lives inside the database.

The proxy lives in its own repository:
https://github.com/cozystack/keycloak-kms-proxy (image
`ghcr.io/cozystack/keycloak-kms-proxy:v0.1.0`).

**KMS backend** is selectable via `encryption.kms.backend`:
- `static` (default): a self-generated 32-byte KEK kept in a Secret —
works on any cluster, suitable for evaluation.
- `vault-transit`: the KEK lives in HashiCorp Vault (recommended for
production; supports KEK rotation).

The feature is **off by default**, so existing installs are unaffected —
`KC_DB_URL_HOST` keeps sourcing from the CNPG `keycloak-db-app` secret.

### Changes
- `values.yaml`: new `encryption` block.
- `templates/proxy.yaml` (new): gated Deployment + Service (+
self-generated KEK Secret for the static backend), hardened pod
(nonroot, read-only rootfs, dropped caps, seccomp).
- `templates/sts.yaml`: conditional DB-endpoint override when encryption
is enabled.
- `tests/encryption_test.yaml` (new): helm-unittest covering on/off and
both KMS backends.

`helm unittest .` → 11/11 pass.

### Follow-ups (not in this draft)
- Backend TLS re-origination to the CNPG endpoint + CA validation.
- Liquibase bootstrap-window handling (`KKP_LENIENT`).
- Offline backfill Job for migrating existing data.
- HA via a shared DEK set (currently single replica).

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(keycloak): add optional KMS-encrypting database proxy for column-level PII encryption (off by default; static KEK or Vault Transit)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added optional transparent encryption for Keycloak database sensitive
fields
* Users can configure encryption scope and select key management backend
options

* **Tests**
* Added comprehensive test coverage validating encryption feature across
multiple configuration scenarios

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 16:41:35 +02:00
Алексей АртамоновandClaude Opus 4.8 9472bb3412 fix(etcd): clamp cleanup PVC resourceNames so replicas:0 cannot widen RBAC
Address CodeRabbit review on #3170: the PVC cleanup Role generates its
resourceNames from .Values.replicas. values.schema.json sets no minimum, so an
explicit replicas: 0 rendered an empty resourceNames list — and an empty
resourceNames in an RBAC rule matches every PVC in the namespace (a wildcard),
silently undoing the least-privilege scoping the previous commit added.

Clamp the count to max(1, replicas) via a shared $replicaCount used by both the
Role resourceNames range and the delete command, so at least data-etcd-0 is
always listed (it exists for any running cluster and the delete uses
--ignore-not-found). Add a regression test asserting replicas: 0 still yields a
name-scoped Role and a single-name delete rather than a namespace-wide grant.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-02 17:36:03 +03:00
Алексей АртамоновandClaude Opus 4.8 4a06df92e7 fix(etcd): scope cleanup Role to the data-etcd PVC names and assert hook metadata
Address CodeRabbit review on #3170:
- Tighten the cleanup Role's PVC rule from a namespace-wide get/list/delete to
  get/delete by explicit resourceNames (data-etcd-<ordinal>, one per replica),
  mirroring the least-privilege pattern already used for the TLS secrets. The
  Job only ever deletes those deterministic names, so list is unnecessary and a
  regression can no longer reach a sibling workload's PVC.
- Extend the Role and RoleBinding unit tests to assert helm.sh/hook and
  helm.sh/hook-delete-policy (not just the weight), so the RBAC cannot regress
  into normal release resources, and assert the PVC resourceNames.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-02 15:58:09 +03:00
Алексей АртамоновandClaude Opus 4.8 a92f060f63 fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant module delete (#3090)
When the etcd tenant module is disabled (etcd: false), the etcd-operator
StatefulSet data PVCs (data-etcd-<ordinal>) are never garbage-collected, and
the five etcd-*-tls secrets are left orphaned in the tenant namespace:
etcd-ca-tls / etcd-client-tls are pre-install hooks this chart marks with
helm.sh/resource-policy: keep, and etcd-peer-ca-tls / etcd-server-tls /
etcd-peer-tls are cert-manager-issued secrets with no owner reference back to
the chart. None are reclaimed by the Helm uninstall.

Add a post-delete cleanup Job (modeled on the seaweedfs/monitoring sibling
fixes) that deletes the data PVCs by name (one per replica) and the five TLS
secrets by name. check-release-name pins the release to "etcd", so every
target name is deterministic and no sibling workload shares the data-etcd-*
prefix. The Job runs non-root with a read-only root fs, dropped capabilities
and seccompProfile RuntimeDefault; its RBAC is scoped to PVC get/list/delete
and the five secrets by resourceNames. Deletes use --ignore-not-found and
failures are logged but never block teardown.

The stuck-HelmRelease finalizer noted in the issue is the platform-wide
sharded-helm-controller side-bug tracked separately; this change addresses the
PVC and secret orphans the etcd chart owns.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-02 15:58:08 +03:00
Алексей Артамонов c7e8f4d975 fix(mariadb): re-guard replication so single-replica passes the operator webhook
PR #2279 made `replication.enabled: true` unconditional so the operator would
always create -primary/-secondary services. But the mariadb-operator validating
webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1,
so single-replica MariaDB could no longer be created — the HelmRelease never
became Ready.

Re-guard replication on replicas>1 and teach the service-name consumers about
the bare <release> service the operator provisions in both topologies:

- mariadb.yaml: gate `replication` on replicas>1; restore single-replica
  external access via `service.type: LoadBalancer` and keep `primaryService`
  LoadBalancer for replicas>1 only.
- backup-cronjob.yaml: route single-replica dumps to the bare service
  (no -secondary exists without replication).
- dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus
  -primary/-secondary when replicas>1, matching the ApplicationDefinition.
- mariadb-rd: list the bare service in services.include so single-replica
  installs are visible in the dashboard.

Adds a single-replica e2e test that reproduces the webhook rejection and
guards against regression.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-02 15:48:29 +03:00
scooby87 fdbe6b299c fix(clickhouse)!: clean up orphaned keeper and data PVCs on delete (#3057) (#3072)
## What this PR does

After deleting a `ClickHouse` app CR, data PVCs were left orphaned in
the tenant
namespace and consumed storage permanently:

- the keeper PVC `default-chk-<release>-keeper-cluster1-0-0-0`, backing
a
`ClickHouseKeeperInstallation` reconciled by the
clickhouse-keeper-operator;
- the main ClickHouse data / log PVCs.

The keeper operator does not delete data PVCs when the CR is removed
(data-safety
default) and the CRD exposes no `persistentVolumeClaimRetentionPolicy`
equivalent,
so the volumes leaked.

This adds a `post-delete` cleanup hook (modeled on the existing MariaDB
hook at
`packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml`) that deletes
every PVC
carrying the release label `app.kubernetes.io/instance=<release>`. It
also adds
that label to the keeper `volumeClaimTemplates` in `chkeeper.yaml`: the
keeper
PVC previously had **no** release label (the chart set labels only on
the keeper
pod template, not the PVC), so the operator never stamped one and a
single
selector would have missed it — leaving the reported leak unfixed.

### Existing releases (label backfill migration)

The altinity clickhouse-keeper-operator only stamps
`volumeClaimTemplates.metadata.labels`
onto PVCs it creates **after** the chart change; it never re-labels PVCs
that
already exist. So for releases installed before this fix, the keeper PVC
stays
unlabeled and the cleanup selector would still miss it. To close #3057
for
existing customers too, this PR adds migration `48` (`48 -> 49`,
`packages/core/platform/images/migrations/migrations/48`) that
back-fills
`app.kubernetes.io/instance=<release>` onto pre-existing keeper PVCs,
and bumps
`migrations.targetVersion` to `49`. The migration:

- routes its version stamp through the shared `lib/cozystack-version.sh`
helper
(`stamp_cozystack_version 49`), per the `cozystack-version-stamp.bats`
convention;
- no-ops when the keeper CRD is absent, and tolerates a transient
apiserver error
on the keeper list (`|| true`) so it cannot abort a platform-wide
upgrade — the
backfill is best-effort (worst case is the pre-existing leak, no worse
than today);
- is idempotent (skips PVCs already labeled).

The main ClickHouse data/log PVCs have carried
`app.kubernetes.io/instance` since
2025-06-03 (well before the migration era), so they need no backfill.

### Hook failure handling

The cleanup hook does **not** mask failures: `kubectl delete` failing
makes the
Job exit non-zero. The **Job** uses `helm.sh/hook-delete-policy:
before-hook-creation`
only (no `hook-succeeded`) so a failed cleanup Job and its logs survive
for
debugging; the **ServiceAccount/Role/RoleBinding** keep
`before-hook-creation,hook-succeeded` so the RBAC artifacts are
reclaimed after a
successful uninstall instead of lingering in the namespace.

### Scope

The hook intentionally reclaims **only PVCs** of the release. Other
release
state (Secrets, ConfigMaps) is owned/garbage-collected through the
normal Helm
uninstall path and is out of scope here.

> [!WARNING]
> **This is destructive — deleting a ClickHouse app now permanently
deletes its data.**
> The hook removes **all** PVCs of the release: the keeper PVC **and**
the main
> ClickHouse data and log PVCs. After deletion nothing is left behind to
recover
> data from. This is intentional and matches the existing MariaDB
cleanup-hook
> behaviour and the general cozystack model (deleting a managed app
reclaims its
> storage). Back up before deleting.

Fixes #3057.

### Release note

```release-note
fix(clickhouse): garbage-collect the keeper and data/log PVCs when a ClickHouse app is deleted, so no orphaned volumes are left behind. A migration back-fills the release label onto keeper PVCs from releases installed before this fix so they are cleaned up too. WARNING: deleting a ClickHouse app now permanently removes all of its PVCs (keeper + data + logs) and the data they hold.
```
2026-07-02 15:31:01 +03:00
Aleksei Sviridkin 9e89ae84ac fix(kubernetes): default worker disk storageClass to the application storageClass (#3129)
## What this PR does

Worker node VMs in a tenant Kubernetes cluster set `evictionStrategy:
External`, so they live-migrate. Live migration requires the worker
system DataVolume to be ReadWriteMany, and linstor-csi only grants RWX
on a DRBD-backed StorageClass. When a node group leaves `storageClass`
empty, the worker DataVolume template previously omitted
`storageClassName` and fell through to the management cluster's default
StorageClass — which is operator-configured and not guaranteed to be
DRBD-backed. On any cluster whose default StorageClass is not a
replicated/DRBD class, the worker disk lands on a non-RWX class and live
migration breaks.

This changes the empty-`storageClass` fallback from the management
cluster default to the application-level `storageClass` (default
`replicated`, a DRBD/LINSTOR class): `{{- with .group.storageClass |
default $.Values.storageClass }}`. A per-node-group `storageClass` still
wins when set; when both the node group and the application
`storageClass` are empty, `storageClassName` is omitted so the cluster
default still applies (explicit opt-out preserved).

Doc and codegen surfaces (`types.go`, `values.yaml`,
`values.schema.json`, `README.md`, and the generated `kubernetes-rd`
ResourceDefinition) are regenerated in lockstep with the new field
description. Chart unit tests cover the full contract: fallback to the
application class, a custom application class, per-node-group override
winning, and the both-empty omit path.

**Upgrade impact — one-time worker-pool roll.** The
`KubevirtMachineTemplate` name embeds a hash of the rendered worker
template body, and the MachineDeployment references that name. On the
shipped default (`md0` leaves `storageClass` empty) the body changes
from "no `storageClassName`" to `storageClassName: replicated`, so on
upgrade the template is renamed and Cluster API performs a one-time
rolling replacement of the worker pool — for every existing tenant
cluster, including those whose default StorageClass already was
`replicated` (the roll is driven by the body hash, not by whether the
effective class differs). This is the same class of roll any
worker-template change triggers and is unavoidable with this approach.
Two test pins document and guard the hash so future body changes are
deliberate.

### Release note

```release-note
fix(kubernetes): worker node disks now fall back to the application-level storageClass (a replicated/DRBD class) instead of the management-cluster default when a node group leaves storageClass empty, so live migration of worker VMs gets the ReadWriteMany volume it requires. Upgrading rolls each tenant cluster's worker pool once.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved worker disk `storageClassName` fallback for empty
per-node-group values: it now reliably uses the app-level
replicated/DRBD StorageClass (rather than relying on the
management-cluster default).
* Ensured KubeVirt data volume templates and rendering behavior are
consistent across single- and multi-node-group setups.

* **Documentation**
* Refined `storageClass`/RWX/DRBD guidance across configuration docs and
schema descriptions, including the intended non-immutable behavior
(optional/undefaulted).

* **Tests**
* Updated template test expectations to cover the new fallback vs
omission cases, including naming assertions for rendered templates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-02 12:00:00 +03:00
Andrei Kvapil 83bd83062e feat(keycloak): enable UPDATE_EMAIL required action in cozy realm
The portal user-settings page changes a user's email via a Keycloak
kc_action=UPDATE_EMAIL redirect, which only works when the UPDATE_EMAIL
required action is enabled at the realm level. On a fresh realm the
action is registered (update-email is a supported, default-on feature
since Keycloak 26.4) but disabled by default.

The edp-keycloak-operator has no CRD field for required actions, so add a
one-shot, idempotent post-install/post-upgrade Job to keycloak-configure
that enables it via kcadm.sh, waiting for the operator to reconcile the
realm first.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-07-01 20:25:15 +02:00
Алексей Артамонов bf75d32a1a fix(dashboard): show LoadBalancer external IP on the app Services tab
The per-app Services tab queried Services with a spec-only type, so it
never read status.loadBalancer.ingress and showed only the cluster-internal
IP for external: true (type: LoadBalancer) services. This regressed the
pre-1.4.0 behaviour (cozystack/cozystack#2789).

Add a ServiceStatus type to the useK8sList query and surface an
"External IP" column (ip / hostname / Pending for LoadBalancer, — otherwise),
mirroring what ExternalIpsPage already does.

Fixes #3111

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-01 19:12:36 +03:00
Алексей Артамонов 7faa32508d fix(clickhouse): reclaim cleanup hook RBAC on successful uninstall
Address review [MINOR] on #3072: keep the cleanup Job on
before-hook-creation (so a failed Job and its logs survive for
post-mortem), but restore before-hook-creation,hook-succeeded on the
ServiceAccount/Role/RoleBinding so those artifacts do not linger in the
namespace after a successful uninstall.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-01 18:31:05 +03:00
Алексей Артамонов 2e65ce108b fix(clickhouse): guard migration when keeper CRD absent and quote hook scalars
Address CodeRabbit review on #3072:

- Migration 45 now skips the backfill when the
  clickhousekeeperinstallations CRD is not installed. The migration runs
  platform-wide via targetVersion, so on clusters that never deployed
  ClickHouse the kubectl get would otherwise fail under set -euo pipefail
  and abort the whole upgrade.
- Quote the templated name/label/serviceAccountName scalars in the cleanup
  hook so the source parses under YAMLlint; rendered output is unchanged.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-01 18:31:05 +03:00
Алексей Артамонов a2ae1621af fix(clickhouse): surface cleanup hook failures and backfill legacy keeper PVC labels
Address review on #3072:

- Cleanup hook no longer masks kubectl delete failures: the command now
  fails (exit 1) when PVC deletion fails instead of swallowing the error
  with || echo. The hook-delete-policy drops hook-succeeded (keeps only
  before-hook-creation) so a failed cleanup Job survives for inspection.
- Add migration 45 (45 -> 46) that backfills app.kubernetes.io/instance
  onto keeper PVCs created before this change; the altinity keeper operator
  never re-stamps volumeClaimTemplate labels onto existing PVCs, so without
  this the cleanup selector misses pre-upgrade keeper PVCs and #3057
  persists for existing releases. Bump migrations.targetVersion to 46.
- Unify helm.sh/hook-weight key quoting across the four hook manifests.
- Extend the unittest to assert the new hook-delete-policy and exit-on-failure.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-01 18:31:05 +03:00
Алексей Артамонов 2c8f6b8ca1 fix(clickhouse)!: clean up orphaned keeper and data PVCs on delete (#3057)
After deleting a ClickHouse app CR, the keeper PVC and the main data/log
PVCs were left orphaned in the tenant namespace, consuming storage
permanently. The clickhouse-keeper-operator does not delete data PVCs on CR
removal (data-safety default) and the CRD exposes no
persistentVolumeClaimRetentionPolicy equivalent.

Add a post-delete cleanup hook (modeled on the MariaDB hook) that deletes
every PVC carrying the release label app.kubernetes.io/instance=<release>.
Also stamp that label on the keeper volumeClaimTemplates in chkeeper.yaml:
the keeper PVC previously had no release label (the chart labelled only the
keeper pod template, not the PVC), so a single selector would have missed
the very PVC the issue reports.

BREAKING CHANGE: deleting a ClickHouse app now permanently removes all of its
PVCs (keeper + data + logs) and the data they hold. Nothing is left behind to
recover from. This matches the existing MariaDB cleanup-hook behaviour and the
general cozystack model where deleting a managed app reclaims its storage.

Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
2026-07-01 18:31:05 +03:00
Алексей Артамонов fff2e7716f fix(seaweedfs)!: clean up orphaned data PVCs and cert/db secrets on tenant module delete (#3092)
When the seaweedfs tenant module is disabled, the StatefulSet volume PVCs and
a set of certificate/credential secrets were left orphaned. Add a post-delete
cleanup hook that deletes the volume PVCs by their release-scoped labels and
the secrets by name (all templated from .Release.Name). CNPG PVCs are not
targeted (CNPG cleans them). The cleanup Job is hardened (non-root, read-only
root fs, dropped caps, resource limits); RBAC scoped via resourceNames;
failures are logged, never blocking teardown.

The PVC selector matches on name + instance only, without an
app.kubernetes.io/component clause, so volume PVCs from custom storage pools
and MultiZone topologies (component volume-<pool>/volume-<zone>[-<pool>]) are
cleaned up too, not just the default pool. Both deletes use --wait=false and
the Job has activeDeadlineSeconds so a finalizer-stuck resource cannot hang
the post-delete hook.

NOTE: disabling seaweedfs now permanently removes its volume PVCs and the data
they hold.

BREAKING CHANGE: disabling the seaweedfs tenant module now permanently deletes
its volume PVCs and the object data they hold. Back up before disabling if the
data matters.

Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
2026-07-01 16:42:43 +03:00
Andrei KvapilandClaude 5d36e13d06 refactor(keycloak): drop the generic db.* connection override
Remove the db.host/db.port/db.urlProperties override and the
extraVolumes/extraVolumeMounts plumbing: they are no longer needed now that
the database endpoint is handled natively. Keycloak's DB host/port come
straight from the keycloak-db-app secret again, except when encryption is
enabled, where the keycloak-kms-proxy owns the repointing. Theme and
admin-console hostname handling are unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-01 13:46:32 +02:00
Andrei KvapilandClaude d459ad800a test(keycloak): cover encryption misconfiguration failures
Add render-time negative-path coverage for the encryption proxy: an
unsupported kms.backend and a vault-transit backend missing address,
keyName, or tokenSecretName must all fail the template render rather than
silently producing a broken Deployment.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-01 13:46:32 +02:00
Andrei KvapilandClaude 956759388f docs(keycloak): document encryption operational caveats and cap proxy memory
Address review feedback on the encryption values block:

- Spell out the operational caveats an operator must understand before
  flipping encryption on: the single-replica proxy turns HA Keycloak into a
  single point of failure; existing data is not migrated automatically so
  enabling (or later disabling) on a populated database breaks logins until
  the offline backfill runs; and a lost static KEK is permanent data loss, so
  the KEK Secret must be backed up.
- Add a memory limit to the proxy resources per the repository style guide.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-01 13:46:31 +02:00
Andrei KvapilandClaude e97c918829 fix(keycloak): fail fast on invalid KMS backend and harden proxy template
Address review feedback on the encryption proxy template:

- Validate encryption.kms.backend at render time and fail with a clear
  message on any value other than static/vault-transit. Both the KEK Secret
  and the env wiring now key off the same validated $backend, so a typo can
  no longer render a Deployment that mounts a KEK Secret that was never
  created (which would silently break the whole IdP).
- Guard the static KEK lookup against a Secret that exists without a usable
  'kek' key, preserving the freshly generated KEK instead of nil/empty.
- Default kms.vault to an empty dict and require address/keyName at render
  time so an incomplete Vault config fails with an actionable message rather
  than a generic nil dereference.
- Publish the metrics port (9090) on the proxy Service so it can be scraped.
- Clarify the backend-TLS note: the proxy negotiates TLS in-band; CNPG CA
  pinning is the remaining follow-up.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-01 13:46:31 +02:00
Andrei Kvapil f5b078d8c7 feat(keycloak): add optional KMS-encrypting DB proxy
Add an opt-in `encryption` flag to the keycloak system package. When
encryption.enabled=true, a keycloak-kms-proxy Deployment + Service is
placed on the wire between Keycloak and its CloudNativePG database, and
Keycloak's KC_DB_URL_HOST/PORT are repointed at the proxy. The proxy
transparently encrypts PII columns (username, email, first/last name,
PII attributes, credentials) before they reach PostgreSQL and decrypts
them on reads; the data-encryption key is wrapped by a KMS and never
lives inside the database.

The KMS backend is selectable: a self-generated static KEK (default,
works on any cluster, suitable for evaluation) or HashiCorp Vault
Transit (recommended for production, supports KEK rotation). The
feature is off by default, so existing installs are unaffected.

Skeleton/draft scope: backend TLS re-origination to CNPG, the Liquibase
bootstrap window (KKP_LENIENT), an offline backfill Job for pre-existing
data, and HA via a shared DEK set are tracked as follow-ups.

Proxy project: https://github.com/cozystack/keycloak-kms-proxy

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-01 13:46:30 +02:00
Aleksei Sviridkin a31a73cb87 fix(tenant): gate VM-resource HelmReleases on victoria-metrics-operator readiness (#3141)
## What this PR does

The tenant `etcd`, `ingress` and `monitoring` HelmReleases create
VictoriaMetrics custom resources — `VMPodScrape` directly in `etcd` and
`ingress`, and `VMCluster` / `VMAlert` / `VLCluster` / `VMAlertmanager`
/ `VMServiceScrape` via the `monitoring-system` child that `monitoring`
creates. All of these are intercepted by the `victoria-metrics-operator`
validating webhook, which is configured with `failurePolicy: Fail`.

On a cold install the operator pod can be briefly unavailable on `:9443`
(for example a transient API-server blip during startup leaves the
webhook backend not yet serving). In that window the tenant releases
reconcile, the webhook rejects their VM* resources with `connection
refused`, and the releases churn through failed-install → rollback →
retry. They carry `install.remediation.retries: -1` and recover
eventually, but the churn can outlast a cold-install readiness deadline
and surfaces as spurious install failures.

This adds a Flux `spec.dependsOn` from each of those three tenant
releases to the `victoria-metrics-operator` HelmRelease in
`cozy-victoria-metrics-operator`, so they wait for the operator to be
Ready before creating any VM* resource. The system monitoring components
already gate on the operator the same way (a variant-level `dependsOn`
in the platform package sources); this brings the tenant releases to
parity. `failurePolicy: Fail` is preserved — validation is unchanged.

The tenant releases are sharded (`sharding.fluxcd.io/key`) while the
operator release is not. helm-controller resolves `dependsOn` through an
uncached API read that bypasses the shard's watch-label-selector cache
(since v1.1.0, PR fluxcd/helm-controller#1070), so the cross-shard
reference is visible to the tenant shard's controller; this repo ships
helm-controller v1.5.0.

A helm-unittest suite pins the gate on all three releases and asserts
that the tenant releases creating no VM* resources (`seaweedfs`,
`gateway`) do not carry it, documenting the intentional narrowness of
the gate.

### Release note

```release-note
fix(tenant): gate tenant etcd, ingress and monitoring HelmReleases on victoria-metrics-operator readiness so they no longer fail to install when the operator's admission webhook is briefly unavailable on a cold install
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved tenant installation reliability by adding readiness gating:
etcd, ingress, and monitoring Helm releases now wait for the Victoria
Metrics operator to become Ready before applying dependent resources.
* Prevents install/rollback failures caused by temporary admission
webhook unavailability during cold starts.
* **Tests**
* Added Helm-unittest coverage to confirm the dependency gate is present
only for templates that create VM* resources (and omitted for templates
that don’t).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-01 13:51:32 +03:00
Aleksei Sviridkin 03435dcc50 fix(flux): tune source-watcher concurrency + CPU to unblock platform install under load (#3163)
## What this PR does

`source-watcher` is the Flux controller that turns each `PackageSource`
into an `ExternalArtifact` (one `ArtifactGenerator` per source, ~100
builds on the management cluster during install). The
`cozystack-platform` HelmRelease blocks until all of those artifacts are
Ready, so source-watcher's throughput sits on the install's critical
path.

It was the only Flux controller left without concurrency/CPU tuning. In
the bootstrap all-in-one pod it ran on the binary default
(`--concurrent=10`) with the shared 100m CPU request; in the production
`FluxInstance` it was absent from the reconciler patch that source-,
kustomize- and helm-controller already receive. Under load its
per-artifact builds finish at staggered times, leaving a random tail of
not-yet-Ready artifacts that times out the platform install.

A failed run's diagnostics show the shape of this: the
`cozystack-platform` HelmRelease timed out with `timeout waiting for:
PackageSource/… status: 'InProgress'` (a run-to-run varying tail — e.g.
`bucket-application`, `linstor-gui`, `capi-provider-core`), while its
~200 ExternalArtifacts reached Ready in a staggered trickle spread over
~60 seconds rather than together. That staggered completion of
independent per-artifact builds is the signature of a throughput-limited
builder, not a single all-or-nothing pull.

This raises source-watcher to `--concurrent=20` in both install paths:

- **Bootstrap all-in-one pod**
(`internal/fluxinstall/manifests/fluxcd.yaml`, regenerated from
`flux-aio.cue` + the `flux-aio` Makefile): `--concurrent=20` plus a 500m
CPU request. The AIO pod sets no CPU limits, so the request is what
drives source-watcher's proportional CPU share when the node is
saturated.
- **Production `FluxInstance`** (`packages/system/fluxcd/values.yaml`):
source-watcher is added to the shared reconciler patch, so it now gets
the same `--concurrent=20`, `--requeue-dependency=5s` and 2000m/2048Mi
limits as the other tuned controllers.

The `flux-aio` module exposes a per-controller resources override for
the watcher (set via CUE) but no args override, so the bootstrap
`--concurrent` bump is a guarded post-render patch in the Makefile; the
embedded manifest is the product of that pipeline. source-watcher's
support for both `--concurrent` and `--requeue-dependency` was confirmed
against its `cmd/main.go`, and the production `op: replace` on
`resources/limits` is safe because the distribution ships source-watcher
with a `resources.limits` block (same shape as source-controller).

Optional follow-up, intentionally left out to keep this focused on the
root cause: if the artifact-build tail still brushes the 10-minute
platform HelmRelease timeout under load, the installer's HelmRelease
install/upgrade timeout values give a safety margin.

### Release note

```release-note
fix(flux): tune source-watcher concurrency and CPU so the platform install does not time out under load
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Increased the source watcher’s concurrency to improve throughput and
better handle larger workloads.
* Added higher CPU and memory settings for the watcher to support more
stable operation under load.

* **Bug Fixes**
* Updated deployment patching so the watcher receives the same
configuration adjustments as other controllers.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-01 13:20:55 +03:00
Aleksei Sviridkin 6db5ae9c6f fix(tenant): gate VM-CR-creating HelmReleases on victoria-metrics-operator
The tenant etcd, ingress and monitoring HelmReleases create VM* custom
resources (VMPodScrape directly in etcd/ingress; VMCluster, VMAlert,
VLCluster, VMAlertmanager and VMServiceScrape via the monitoring-system
child of monitoring) guarded by the victoria-metrics-operator validating
webhook with failurePolicy: Fail. On a cold install the operator pod can
be briefly unavailable on :9443, so these releases reconcile before the
webhook backend is serving and are rejected with connection refused. They
carry install.remediation.retries: -1 and would eventually recover, but
the failed-install -> rollback -> retry churn can outlast a cold-install
readiness deadline and surfaces as spurious install failures.

Add a Flux spec.dependsOn from each of these releases to the
victoria-metrics-operator HelmRelease in cozy-victoria-metrics-operator,
so they wait for the operator to be Ready before creating VM* resources.
The system monitoring components gate on the operator the same way, via a
variant-level dependsOn. The cross-shard reference (tenant releases are
sharded, the operator release is not) resolves via helm-controller's
uncached dependency check (APIReader, v1.1.0+).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-01 12:05:23 +03:00
Aleksei Sviridkin 41ff94eca9 fix(flux): tune source-watcher concurrency and CPU under load
source-watcher reconciles one ArtifactGenerator per PackageSource and
builds the resulting ExternalArtifacts. On the management cluster that is
~100 artifact builds during install, and the platform HelmRelease blocks
until all of them are Ready.

It was the only Flux controller left untuned. In the bootstrap all-in-one
pod it ran without an explicit --concurrent (binary default 10) and with
the shared 100m CPU request; in the production FluxInstance it was missing
from the --concurrent / CPU patch that source-, kustomize- and
helm-controller already receive. Under load its per-artifact builds finish
at staggered times, leaving a random tail of not-yet-Ready artifacts that
times out the platform install.

Raise source-watcher to --concurrent=20 in both deployments. In the
bootstrap all-in-one pod also give it a 500m CPU request; the pod has no
CPU limits, so the request is what drives its CPU share under contention.
In the production FluxInstance add it to the shared reconciler patch, so it
now receives the same --requeue-dependency=5s and 2000m/2048Mi limits the
other tuned controllers already have.

The flux-aio module exposes a resources override for the watcher (set via
CUE) but no args override, so the bootstrap --concurrent bump is a
post-render patch in the Makefile.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-01 07:42:33 +03:00
Aleksei Sviridkin f7271b8011 fix(velero): add startupProbe so slow startup does not crashloop the install gate
Velero binds its metrics/health endpoint (:8085, http-monitoring) only
after the server loads its plugins and connects to the API server. Under
a heavy parallel platform install that can take minutes, but the upstream
liveness probe begins checking /metrics at initialDelaySeconds=10 and
kills the container after failureThreshold=5 (~130s total) before :8085
is bound. The kubelet SIGTERMs velero (exitCode 0/Completed), it
BackOff-restarts, and never escapes the loop inside the install-gate
window. Because backupstrategy-controller hard-depends on velero, this
blocks the gate.

Add a startupProbe to the velero server container. While it is failing
the kubelet holds off liveness and readiness entirely, granting
failureThreshold * periodSeconds = 30 * 10s = 300s of startup grace, then
hands over to the unchanged, tight upstream liveness (10s / 30s / 5) for
steady-state crash detection. This is strictly better than loosening
liveness, which would also blunt steady-state detection.

The upstream velero chart (12.0.3) does not render a startupProbe, so the
package patches the deployment template via patches/add-startup-probe.patch
(re-applied after re-vendor by the Makefile update target, matching the
seaweedfs/ingress-nginx convention) and supplies the startupProbe through
the package values override. Pin the startupProbe budget and the unchanged
liveness with a helm-unittest regression guard.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-01 06:04:22 +03:00
Aleksei Sviridkin 9c18260f95 fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets) (#2957)
## What

The e2e install pipeline still flakes on a loaded runner, failing at a
different point each run. Root-caused via repeated runs and a live SSH
breakpoint; the platform itself is fine (the components do converge),
the failures are slowness + self-inflicted churn under resource
pressure.

### Fixes

1. **VPA updater OOM** — the `vertical-pod-autoscaler-updater` memory
limit was `110Mi`. The updater watches every pod in the cluster, so on a
full install it OOM-kills → `ProgressDeadlineExceeded` → Flux fails the
HelmRelease "early due to stalled resources", cascading NotReady to
monitoring, velero, etcd-operator and everything downstream. Raised to
`512Mi` (request `200Mi`).

2. **VPA-for-VPA eviction churn** — the two VPA-for-VPA
`VerticalPodAutoscaler` objects ran `updateMode: Auto`, evicting the VPA
recommender(s) during install and adding pod churn on an already busy
cluster. Switched to `Initial`, matching the existing monitoring/etcd
`VPA->Initial` fixes.

3. **Install wait budgets** — components reach readiness but overshoot
tight waits on a busy runner: keycloak HR chain `10m→20m`, tenant
aggregate wait `10m→20m`, per-app tenant readiness waits `5m→10m`.

(The crust-gather failure-snapshot timeout that kept whole jobs hanging
to the 3h cancel is already on `main`.)

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Increased several installation and readiness wait times to better
handle slower environment startup and reduce premature failures.
* Updated vertical autoscaling behavior to avoid unnecessary pod
restarts during deployment.
* Raised resource limits for the autoscaling updater to improve
reliability under load.

* **Tests**
* Aligned automated checks with the new autoscaling behavior and longer
readiness expectations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-01 06:00:43 +03:00
Andrei KvapilandClaude d6f66d2dcd fix(vertical-pod-autoscaler): stop the updater OOM + VPA-for-VPA eviction churn
Two root causes of e2e install flakiness, both in the VPA package:

1. The updater memory limit was 110Mi. The updater watches every pod in
   the cluster, so on a full cozystack install it OOM-kills, which trips
   the Deployment's ProgressDeadlineExceeded and makes Flux fail the
   HelmRelease 'early due to stalled resources' - cascading NotReady to
   monitoring, velero, etcd-operator and everything downstream. Raise the
   limit to 512Mi (request 200Mi); the recommender is already unlimited.

2. The two VPA-for-VPA VerticalPodAutoscaler objects ran updateMode: Auto,
   which evicts the VPA recommender(s) during install and adds pod churn on
   an already busy cluster. Switch to Initial, matching the existing
   monitoring/etcd VPA->Initial fixes.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-01 03:27:15 +03:00
Aleksei Sviridkin 22c96d8dac fix(kubernetes): default worker disk storageClass to the application storageClass
Worker node VMs set evictionStrategy: External, so they live-migrate.
Live migration requires the system DataVolume on a ReadWriteMany volume,
and linstor-csi only grants RWX on a DRBD-backed StorageClass. When a node
group leaves storageClass empty the template previously omitted the field
and fell through to the management cluster default StorageClass, which may
be a non-DRBD class the CSI driver rejects for RWX — breaking migration.

Fall back to the application-level storageClass (the replicated/DRBD class)
when the node group storageClass is empty, instead of the cluster default.
When both are empty the field is still omitted, preserving the explicit
cluster-default opt-out.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-01 00:56:29 +03:00
Aleksei Sviridkin cbc314a8c5 fix(kubevirt-cdi): order behind cozystack-basics so cdi-clone-dv RoleBinding finds cozy-public
The kubevirt-cdi default variant only depended on cozystack.networking,
which reconciles long before cozystack-basics provisions the cozy-public
namespace. kubevirt-cdi ships the cdi-clone-dv RoleBinding into cozy-public,
so on a fresh install the RoleBinding can race namespace creation and fail
with `namespaces "cozy-public" not found`. Flux retries normally mask it,
but under slow reconcile it surfaces as a hard UpgradeFailed.

Add a cozystack.cozystack-basics edge so the namespace exists first,
mirroring the backupstrategy-controller ordering. No cycle: cozystack-basics'
own dependencies (tenant-application, cozystack-engine, gateway-api-crds) do
not reach kubevirt-cdi. Pin the edge with a helm-unittest regression guard.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-30 19:51:46 +03:00
Andrei Kvapil c725a41f3a feat(tenant): recursive (hierarchical) resource quotas for sub-tenants (#3086)
## What this PR does

Adds **hierarchical (recursive) resource quotas** for tenants so a
tenant can no longer escalate its allocation by creating sub-tenants.
Today `tenant.spec.resourceQuotas` becomes a plain per-namespace
`ResourceQuota`, which ignores the tenant tree: a super-admin inside a
quota'd tenant can create a sub-tenant with a larger quota — or no quota
at all — and consume more than was allocated.

A tenant's declared quota is now treated as the budget for its **whole
sub-tree**:

- A child that declares its own quota **carves a fixed slice** out of
the parent's remaining budget (parent 10 CPU → child 4 CPU → parent has
6 left).
- A child **without** a quota **shares** the parent's remaining pool
rather than being unbounded.
- A child quota that exceeds the parent's remaining budget is
**rejected**.

Two cooperating layers, mirroring OpenShift's ClusterResourceQuota split
(admission + reconciliation):

1. **Declaration-time gate** in the aggregated apiserver
(`apps.cozystack.io` Tenant create/update): deterministically rejects a
sub-tenant whose declared quota exceeds the parent's remaining budget.
2. **Runtime enforcer** in `cozystack-controller`: maintains a
per-namespace `tenant-quota-allocated` ResourceQuota that clamps each
pool member to its share of the parent budget, aggregating sub-tree
usage so members collectively stay within the budget. Kubernetes
enforces the most restrictive ResourceQuota in a namespace, so this
binds without fighting Flux over the chart-rendered `tenant-quota`.
Overcommitted pools (a parent quota lowered below the sum of its
children) emit a `QuotaOvercommitted` event. A
`--tenant-quota-buffer-percent` flag temporarily inflates pool budgets
during rollout so workloads already over a freshly-introduced quota keep
running.

The model and the selector-spans-namespaces / aggregated-usage approach
are adapted from OpenShift's ClusterResourceQuota controllers
(`openshift/api`, `library-go`, `cluster-policy-controller`,
`apiserver-library-go`), Apache-2.0, Copyright Red Hat, Inc. and the
OpenShift Authors; attribution is preserved in the source. The
per-namespace quota arithmetic reuses the upstream Kubernetes quota
helpers (`k8s.io/apiserver/pkg/quota/v1`).

No API types or the tenant chart change; the feature works with the
existing `tenant.spec.resourceQuotas` field.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(tenant): tenant resource quotas are now hierarchical — a sub-tenant's quota is carved out of its parent's remaining budget and can no longer exceed it; sub-tenants without a quota share the parent's pool. Set `--tenant-quota-buffer-percent` on cozystack-controller for a temporary grace buffer during upgrades.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added hierarchical tenant quota enforcement across tenant namespaces,
including shared pool handling and configurable quota buffer support.
* Tenant quota changes are now validated on create and update to prevent
over-allocation.

* **Bug Fixes**
* Improved quota calculations so usage, carve-outs, and nested tenant
limits are applied more consistently.
* Stale quota allocations are now cleaned up when namespaces no longer
belong to an active tenant pool.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-30 17:56:24 +02:00
scooby87 5b26ec3462 fix(mariadb): clean up orphaned operator-generated secrets (#3056) (#3071)
## What this PR does

After deleting a `MariaDB` app CR with metrics and/or replication
enabled, two
Secrets created by the mariadb-operator were left orphaned in the tenant
namespace:

- `<release>-metrics-password`
- `<release>-repl-password`

These are generated by the operator (`k8s.mariadb.com/v1alpha1`) as a
side
effect of `metrics.enabled` / `replication.enabled`, so they are **not**
Helm-owned and survive `helm uninstall`. The existing post-delete hook
(`templates/hooks/cleanup-pvc.yaml`) only swept PVCs, leaving the
Secrets behind.

This PR extends that hook to also clean them up:

- grants the cleanup `Role` RBAC on `secrets`;
- deletes both password Secrets by name with `--ignore-not-found`;
- adds a `helm-unittest` suite locking in the behaviour, plus a `test`
Makefile
target so the package's tests actually run in CI
(`hack/helm-unit-tests.sh`
  only runs packages that define a `test` target).

Fixes #3056.

### Release note

```release-note
fix(mariadb): garbage-collect operator-generated metrics-password / repl-password Secrets when a MariaDB app is deleted
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved MariaDB uninstall cleanup to remove leftover
operator-generated password Secrets in addition to persistent
volumes/claims.
* Hardened the cleanup hook Job with stricter pod/container security
settings, least-privilege Secret permissions, idempotent deletions for
missing resources, and correct non-zero exit on real failures.
* **Tests**
* Added a cleanup hook test suite validating Secret deletion behavior,
failure propagation, and that cleanup RBAC grants least-privilege access
only to the required Secrets.
  * Added a Helm unit test `test` target for the MariaDB chart.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-30 18:05:55 +03:00
Алексей Артамонов d988b3e8c8 fix(apiserver): read tenant pool quota usage via direct client and grant RBAC
parentPoolUsage listed ResourceQuotas through the aggregated apiserver's cached
client, which lazily starts a cluster-wide ResourceQuota informer. cozystack-api
has no RBAC to list resourcequotas, so the informer never synced and tenant
create/update of a bounded child hung until the request timed out (observed on a
cluster). Read quota usage through the direct (uncached) watch client instead,
scoped to the pool's member namespaces, so no cluster-wide informer is started,
and grant cozystack-api get/watch/list on resourcequotas for those reads.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-06-30 16:12:17 +02:00
Andrei KvapilandClaude 844deb170f feat(controller): enforce recursive tenant quotas at runtime
Add a tenant quota reconciler to cozystack-controller that realises the
hierarchical (sub-tree) quota model at runtime. Each tenant's declared
quota is the budget for its whole sub-tree: a child that declares its own
quota carves a fixed slice out of the parent budget, while a child without
a quota shares the parent's remaining pool.

The controller maintains a per-namespace 'tenant-quota-allocated'
ResourceQuota (Kubernetes enforces the most restrictive quota, so it binds
without fighting Flux over the chart-rendered 'tenant-quota'), aggregating
sub-tree usage so a pool's members collectively stay within the budget. A
--tenant-quota-buffer-percent flag temporarily inflates pool budgets so
workloads already over a freshly-introduced quota keep running during
rollout; overcommitted pools emit a QuotaOvercommitted event.

The model and the selector-spans-namespaces/aggregated-usage approach are
adapted from OpenShift's ClusterResourceQuota controllers (openshift/api,
library-go, cluster-policy-controller, apiserver-library-go), Apache-2.0,
Copyright Red Hat, Inc. and the OpenShift Authors.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-06-30 16:12:17 +02:00
Aleksei Sviridkin eab9b034f6 fix(seaweedfs): enforce read-only bucket access by bumping cosi-driver to v0.3.1 (#3136)
## What this PR does

The SeaweedFS COSI driver shipped at `v0.1.2` hardcodes read-write S3
actions (`Read,Write,List,Tagging`) for every BucketAccess and never
reads the `accessPolicy` BucketAccessClass parameter. The chart already
defines a `-readonly` BucketAccessClass carrying
`parameters.accessPolicy: readonly`, and the `bucket` app routes
`readonly: true` users to it — but `v0.1.2` ignores the parameter, so a
credential meant to be read-only is silently issued with full read-write
access and can upload and delete objects. This is an access-control bug.

Upstream `seaweedfs/seaweedfs-cosi-driver` added `accessPolicy` handling
in `v0.3.0`: `readonly` maps to `Read,List`, while `readwrite` and the
empty default stay `Read,Write,List,Tagging`. This bumps the driver to
`v0.3.1` via a `seaweedfs.cosi.image` override in the package
`values.yaml`, so the override survives `make update` and the vendored
chart is left untouched. The default read-write action set is unchanged,
so existing read-write credentials are unaffected; only read-only grants
are tightened to actually be read-only.

The `extra/seaweedfs` chart already runs `v0.3.1` with the same
objectstorage-sidecar, so the driver/sidecar combination is already
proven in this repo. A chart unit test pins the rendered driver image so
this fix cannot silently regress on a future re-vendor.

This does not close #1921. That issue bundles two features — read-only
bucket access (fixed here) and Object Lock / WORM (the `-lock`
BucketClass), which is independent and out of scope here. Relates to
#1921.

### Release note

```release-note
Fix SeaweedFS read-only bucket credentials being issued with read-write access; the COSI driver now honours the read-only access policy.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Pin the SeaweedFS COSI driver image to
`ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1` to keep access behavior
consistent and avoid relying on a vendored default.
* Ensure the rendered SeaweedFS COSI deployment uses the expected
container name and the pinned driver image.
* **Tests**
* Add a YAML test suite to validate the generated COSI deployment
template matches the expected container name and pinned image.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-30 16:04:00 +03:00
scooby87 bef42229d4 fix(harbor): clean up orphaned jobservice PVC and ACME solver resources on delete (#3061) (#3077)
## What this PR does

After deleting a `Harbor` app CR, two things were left orphaned in the
tenant
namespace:

- the **jobservice / trivy PVCs** (`<release>-jobservice`, and
`<release>-trivy`
when Trivy is enabled), because
`packages/apps/harbor/templates/harbor.yaml`
sets `persistence.resourcePolicy: "keep"`, which annotates the upstream
Harbor
subchart's PVCs with `helm.sh/resource-policy: keep` so Helm/Flux
deliberately
  never deletes them;
- cert-manager's **ACME http01-solver** resources
(`cm-acme-http-solver-*`), via
the same stuck-challenge mechanism as the Bucket issue (#3060) when
Harbor
  issues a non-wildcard cert.

This adds a single `post-delete` cleanup hook that:

- **deletes the upstream Harbor subchart's kept PVCs** — selecting by
`app.kubernetes.io/instance=<release>-system` (the subchart runs as the
  `<release>-system` HelmRelease, so its PVCs carry that label, **not**
`<release>`). A hook is required precisely because `resourcePolicy:
keep`
blocks normal reclaim. This intentionally covers both the jobservice and
trivy
  PVCs: both hold only regenerable data (jobservice file logs; trivy's
  vulnerability-DB cache), and registry images live in S3.
- **deletes this release's `Certificate`** (deterministic name
`<release>-ingress-tls`); cert-manager then cascade-deletes the `Order`,
`Challenge` and solver `Pod`/`Service`/`Ingress` objects it owns via
owner
  references.
- **host-scoped belt-and-braces sweep** for any ACME http01 solver
objects
cert-manager may have orphaned (e.g. the Certificate was already gone,
so
deleting it did not cascade). The sweep is scoped **strictly to this
release's
Harbor host** — Challenges filtered by `.spec.dnsName`, solver Ingresses
by
  `.spec.rules[0].host` — **not** a namespace-wide
`acme.cert-manager.io/http01-solver=true` label sweep, because a
Cozystack
tenant namespace can host many apps and a blanket label sweep would
delete
  sibling apps' in-flight challenge solvers.

The Job is hardened: non-root (uid 65534), read-only root fs, all caps
dropped,
`seccompProfile: RuntimeDefault`, with CPU/memory requests and limits
set. RBAC
is scoped to exactly the resources it deletes (`persistentvolumeclaims`,
`certificates`, `orders`, `challenges`, solver `ingresses`) and verified
against
the vendored cert-manager CRDs. Failures are logged but never block
teardown.

## Cleanup scope — what is and isn't removed

**Removed:** the upstream Harbor subchart's kept PVCs (jobservice +
trivy) and
this release's cert-manager ACME artifacts (Certificate, and host-scoped
Order/Challenge/solver-Ingress).

**Not touched (by design):**

- **CNPG database PVCs** (`<release>-db-N`) — owned by the CNPG
`Cluster` and
  governed by CNPG's own reclaim policy; they carry CNPG labels, not
`app.kubernetes.io/instance=<release>-system`. Re-installing Harbor with
a
different DB size into the same namespace can therefore leak the old DB
PVCs;
  this is left to CNPG's lifecycle on purpose (the data is the user's).
- **Redis PVCs** — managed by the redis-operator with their own labels,
likewise
  untouched.
- **Solver `Pod`/`Service`** — not deleted directly; they are removed by
cert-manager's owner-ref cascade when the Certificate/Challenge is
deleted, so
  the hook grants no RBAC for them.

> [!NOTE]
> The PVC sweep is eventually-consistent: while the inner
`<release>-system`
> HelmRelease is still being uninstalled by Flux, a PVC may briefly stay
in
> `Terminating` until its consumer Pods (jobservice/trivy) exit. The
Job's
> deletion request always succeeds; reclaim completes once the pods are
gone.

> [!WARNING]
> **This overrides the intentional `persistence.resourcePolicy: "keep"`
setting
> and force-deletes the jobservice/trivy PVCs on app deletion.** Those
PVCs hold
> only jobservice file logs and the trivy vuln-DB cache (registry images
live in
> S3), so data-loss risk is low — but if retaining them across
re-installs is
> desired, this is a conscious product decision to revisit. The ACME
cleanup
> affects only cert-manager's transient artifacts.

Fixes #3061.

> Follow-up tracked in #3088: digest-pin the cleanup `kubectl` image and
route it
> through `cozy-lib.images-registry` (pre-existing debt shared with the
MariaDB
> cleanup hook), out of scope for this PR.

### Release note

```release-note
fix(harbor): clean up the orphaned jobservice/trivy PVCs and cert-manager ACME http01-solver resources left behind after a Harbor app is deleted. NOTE: this force-deletes the jobservice/trivy PVCs, overriding persistence.resourcePolicy=keep (jobservice file logs and trivy vuln-DB cache only; registry images are in S3). CNPG database PVCs are not touched.
```
2026-06-30 15:55:52 +03:00
myasnikovdaniil 3e20a0cd41 fix(kubernetes): make node group resources optional when instanceType is set (#3121)
## What this PR does

A tenant Kubernetes node group can be sized two ways: by `instanceType`
or by an explicit `resources` block. The generated values schema,
however, marked `resources` as required on every node group, so a node
group sized purely by `instanceType` was rejected at install time:

```text
at '/nodeGroups/<name>': missing property 'resources'
```

The same failure occurs through the typed `Kubernetes` CR: an empty
`resources: {}` is dropped during the CR-to-HelmRelease translation, so
by validation time the key is absent and the required check fails — even
though sizing was already fully specified by `instanceType`.

This marks `resources` optional in the node group definition and
regenerates the dependent artifacts (`values.schema.json`, API types,
README, and the `ApplicationDefinition`). No change is needed to emit
the VM: the chart already renders `domain.cpu`/`domain.memory` only when
`resources` is present and `instancetype` only when set, so a node group
sized purely by `instanceType` now validates and installs. Node groups
that specify `resources` are unaffected.

`instanceType` and `resources` are alternative, mutually exclusive
sizing sources — KubeVirt rejects a VirtualMachine that references an
instancetype and also sets explicit `domain.cpu`/`domain.memory`, since
instancetype CPU/memory cannot be overridden. To avoid emitting a VM
that the instancetype admission webhook would later reject, the chart
now fails fast at render time with a clear message when a node group
sets both a non-empty `instanceType` and explicit `resources`.

Tests: helm-unittest coverage proves a node group with `instanceType`
and no `resources` passes schema validation and renders without explicit
`domain.cpu`/`memory`; a complementary case confirms explicit
`resources` are still honoured; and a `failedTemplate` case pins the
render-time rejection of the conflicting both-set combination. The
omit-resources case fails against the pre-fix schema with the exact
error above, so it is a real regression guard.

Addresses #3046.

### Screenshots

N/A — schema and chart change, no UI impact.

### Release note

```release-note
fix(kubernetes): allow a node group to omit `resources` when it is sized by `instanceType` (previously failed schema validation with "missing property 'resources'"); reject the unsupported combination of `instanceType` and explicit `resources` at render time
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Worker node groups can optionally size CPU/memory via `resources`, and
can omit `resources` when using `instanceType`.
* **Bug Fixes**
* Render now validates CPU/memory rules: `instanceType` with only one of
`resources.cpu`/`resources.memory` fails, and `instancetype` is omitted
when both `resources.cpu` and `resources.memory` are set (with
`instanceType` ignored).
* **Documentation**
* Updated API/Helm docs and JSON schema to clarify `resources` vs
`instanceType` precedence and the “set both CPU and memory (or neither)”
requirement.
* **Tests**
* Added helm-unittest scenarios covering omission, precedence, and
render-time failure cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-30 17:46:54 +05:00
Aleksei Sviridkin 0892c5a38c fix(qdrant)!: clean up orphaned data PVC on delete (#3059) (#3075)
## What this PR does

After deleting a `Qdrant` app CR, the StatefulSet data PVC
(`qdrant-storage-<release>-0`) was left orphaned in the tenant
namespace,
consuming storage permanently. StatefulSet-templated PVCs are never
garbage-collected on StatefulSet deletion by default.

The vendored upstream Qdrant chart does **not** support
`persistentVolumeClaimRetentionPolicy` (verified: its `statefulset.yaml`
never
renders that field), so the OpenBAO-style value passthrough is not
possible
here. Instead this adds a `post-delete` PVC cleanup hook (modeled on the
MariaDB
hook) that deletes the release's PVCs by label.

The chart's volumeClaimTemplate only stamps `app: qdrant` (shared by
every
Qdrant in a namespace), so to select PVCs safely per release this also
passes
`persistence.additionalLabels.app.kubernetes.io/instance: <release>`
through the
HelmRelease — giving the data PVC a release-unique label the hook
selects on.

The cleanup Job is hardened: runs as non-root (uid 65534), read-only
root fs,
all capabilities dropped, `seccompProfile: RuntimeDefault`; RBAC scoped
to
`persistentvolumeclaims`. A real cleanup failure is logged (not silently
swallowed) but never blocks namespace teardown.

> [!WARNING]
> **Deleting a Qdrant app now permanently removes its data PVC and the
vector data it holds.**
> This is intentional and fixes the storage leak; the hook only fires on
full
> release deletion (not on scale-down). Back up before deleting.

Fixes #3059.

### Release note

```release-note
fix(qdrant): garbage-collect the StatefulSet data PVC when a Qdrant app is deleted, instead of leaving it orphaned. WARNING: deleting a Qdrant app now permanently removes its data PVC and the data it holds.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a post-delete cleanup hook for Qdrant to remove leftover
StatefulSet data PVCs after deletion.
* Included hardened Kubernetes settings for the cleanup job to improve
runtime safety.
* **Tests**
* Added Helm chart tests that validate the post-delete cleanup job and
its RBAC permissions.
* Enhanced end-to-end verification to confirm the data PVC is fully
removed after deleting a Qdrant instance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-30 14:53:11 +03:00
Aleksei Sviridkin bf14a73b39 feat(api): add SecurityGroup network policy resource (sdn.cozystack.io) (#2922)
## What this PR does

Introduces a tenant-facing `SecurityGroup` resource in a new
`sdn.cozystack.io/v1alpha1` API group, served by the Cozystack
aggregated API server as a 1:1 projection of a `CiliumNetworkPolicy`. A
SecurityGroup attaches to one of the tenant's managed applications by
reference (`spec.targetRef: {apiGroup, kind, name}`) and declares
allow-list ingress/egress for that application. Tenants manage network
policy for their own apps entirely through
`securitygroups.sdn.cozystack.io`, without being granted any access to
the `cilium.io` API group.

The translation is synchronous and stateless — the API server converts
each SecurityGroup into a CiliumNetworkPolicy in the same namespace on
write and back on read; there is no controller. The backing policy's
`endpointSelector` is **derived** from `targetRef` (matched on the
application's lineage labels
`apps.cozystack.io/application.{group,kind,name}`), never copied from
tenant input, so the selector is machine-generated and can only ever
match the referenced application's own pods in the tenant's namespace.
Policies owned by this API carry an `sdn.cozystack.io/securitygroup`
marker label that the storage always re-asserts and hides from the view,
so it only ever surfaces or mutates its own policies and never touches
platform-managed network policies.

The value is self-service network policy on a Cozystack-native,
RBAC-scoped surface, without exposing Cilium's version-coupled CRDs to
tenants. Because the selector is derived from an authorized application
reference rather than free-form tenant input, the RBAC boundary is
structural, not selector-validated: a tenant cannot express network
policy against arbitrary or platform-owned pods. The full design —
selector derivation from lineage labels, the structural boundary, the
additive-policy reality (a SecurityGroup widens a managed application's
allowed traffic but cannot sever its platform-managed management plane,
and equally cannot tighten below the platform baseline), the in-tree
CiliumNetworkPolicy mirror that avoids a Cilium Go-module dependency,
RBAC, and deferred extensions (raw-pod targeting, cluster scope,
reusable groups) — is in `pkg/apis/sdn/DESIGN.md`.

Implements the attachment model from #1614.

### Release note

```release-note
feat(api): add the SecurityGroup resource (sdn.cozystack.io/v1alpha1) — a tenant-facing, namespace-scoped firewall that attaches to a managed application by reference and projects 1:1 onto a CiliumNetworkPolicy with a machine-derived endpoint selector, letting tenants manage their application's network policy without cilium.io access.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added the `SecurityGroup` API resource (`sdn.cozystack.io` v1alpha1)
with a namespaced projection backed by Cilium network policies.
* Enabled the SDN `APIService` and wired SDN routing, scheme, and REST
support.
* Expanded RBAC to include SecurityGroup access and required Cilium
policy permissions.
* **Documentation**
* Added a design document describing projection behavior and rule
semantics.
* **Bug Fixes**
  * Improved API rule validation coverage for missing list-type fields.
* **Tests**
* Added SDN round-trip, fuzzing, RBAC, and comprehensive REST
create/get/list/update/delete/watch/resource-version tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-30 14:30:23 +03:00