admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-08 04:54:52 +09:00
Code Issues Packages Projects Releases Wiki Activity
4,598 Commits 569 Branches 294 Tags
2cfcdc265c8aa21033defedfced38e288f498a29
Commit Graph
4598 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Aleksei Sviridkin 2cfcdc265c fix(cert-manager): raise cainjector memory limit to unblock caBundle injection (#3199) 
## What this PR does

cert-manager's cainjector loads every ValidatingWebhookConfiguration,
MutatingWebhookConfiguration, APIService and CRD into its informer
caches at
startup. On a full cozystack install that working set exceeds the 128Mi
memory
limit the package sets today, so the single leader-elected cainjector
Pod is
OOMKilled (exit 137) during cache population and enters CrashLoopBackOff
before
it ever runs a reconcile.

Because cainjector is the component that injects the issuing CA into
webhook
`clientConfig.caBundle` (via `cert-manager.io/inject-ca-from`), a
cainjector
that never completes a reconcile leaves those caBundles empty. With
`failurePolicy: Fail`, the apiserver then rejects every call to an
affected
webhook with `x509: certificate signed by unknown authority` — even
though the
cert-manager PKI chain is healthy and the webhook Pods serve a valid
cert. In
practice this intermittently blocks `Ingress` creation in tenant
namespaces
(the ingress-nginx admission webhook), which stalls dependent
HelmReleases and
fails installs.

It behaves as a flake because it is a memory race: on a lighter run
cainjector
fits under 128Mi, injects the caBundle within seconds and everything
works; on
a heavier run (more CRDs/webhooks resident, slower leader election) peak
startup
memory crosses 128Mi and it is OOMKilled before the first injection,
stalling CA
injection cluster-wide for as long as it CrashLoops.

Fix: raise the cainjector memory limit to 512Mi (matching the
cert-manager
controller in the same values file) and lift the request from 32Mi to
128Mi so
injection completes deterministically instead of racing the OOM floor. A
helm-unittest pins the sizing so a blanket resource change or a `make
update`
cannot silently lower it again. The webhook and controller resources are
left
unchanged — only cainjector exhibited the OOM.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
fix(cert-manager): raise cainjector memory limit so CA injection into webhook caBundles no longer fails under memory pressure (previously surfaced as intermittent "x509: certificate signed by unknown authority" admission errors)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Increased cainjector memory settings for cert-manager, helping it run
more reliably during startup and cache population.
* Added automated chart tests to verify the cainjector resource settings
stay within the expected range.

* **Bug Fixes**
* Reduced the risk of cainjector being OOM-killed, improving stability
and consistency of CA injection.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-04 22:36:14 +03:00
Ivan Okhotnikov 223c0ef4fb fix(cert-manager): raise cainjector memory limit to unblock caBundle injection 
cainjector loads all ValidatingWebhookConfigurations, APIServices and CRDs
into its informer caches at startup. On a full cozystack install that working
set exceeds the 128Mi limit, so the single leader-elected cainjector Pod is
OOMKilled during cache population and CrashLoops before it injects any
caBundle. Webhook ValidatingWebhookConfigurations then keep an empty caBundle
and every admission call fails with "x509: certificate signed by unknown
authority" (failurePolicy: Fail), intermittently blocking Ingress creation
and stalling tenant installs.

Raise the cainjector memory limit to 512Mi (matching the controller) and the
request to 128Mi, and add a helm-unittest that pins the sizing so a blanket
resource change or a `make update` cannot silently lower it again.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-04 15:57:49 +03:00
Aleksei Sviridkin 10f8e2de1f feat(kamaji): default to 2 replicas, drop telemetry webhook (#2671) 
## What this PR does

`kamaji-controller-manager` runs **9 sub-controllers per
TenantControlPlane** (bootstraptoken, coredns, konnectivity, kubeadm*,
kubelet, kubeproxy, …) inside a single Go process, **and** serves the
admission webhooks for `tenantcontrolplanes` + `datastores` from the
same pod via `kamaji-webhook-service`. With the upstream default
`replicaCount=1`, every webhook round-trip from the apiserver shares CPU
with the reconciler goroutines. On a multi-tenant cluster this pushes
the apiserver-observed p99 of `mtenantcontrolplane.kb.io` and
`vtenantcontrolplane.kb.io` from the kamaji-side `~10ms` (per
`controller_runtime_webhook_latency_seconds_count/sum`) up to `85–95ms`
— purely due to CPU contention.

Three safe defaults are changed:

- **`kamaji.replicaCount: 2`** — HA-safe baseline. `--leader-elect` is
already enabled upstream, so only one replica reconciles tenants; the
follower's CPU is fully available for webhook handlers. On a single-node
cluster both pods still schedule (just colocated).
- **`kamaji.telemetry.disabled: true`** — drops the
`--disable-telemetry` handler **and** removes the
`telemetry.kamaji.clastix.io` entry from the
`ValidatingWebhookConfiguration`. The latter requires `patches/1.diff`
to wrap the telemetry block in `{{- if not .Values.telemetry.disabled }}
… {{- end }}` inside the vendored
`controller-gen/validating-webhook.yaml`. Without the webhook entry, the
apiserver no longer round-trips to a no-op handler on every
TenantControlPlane CREATE/UPDATE/DELETE.
- **`kamaji.affinity` (soft `podAntiAffinity`)** — spread the two
replicas across nodes when possible, while keeping single-node clusters
working (`preferredDuringScheduling`, not required).

The `patches/1.diff` approach (re-applied by `make update`) mirrors the
pattern already used in `packages/system/fluxcd-operator/Makefile` and
`packages/system/ingress-nginx/Makefile`, and survives the next
`clastix/kamaji` chart bump.

## Measured impact

Hikube prod, 22 TenantControlPlanes, before/after a manual rollout of
the same configuration:

| webhook | p99 before | p99 after | Δ |
| --- | --- | --- | --- |
| `mtenantcontrolplane.kb.io` | 85 ms | 25 ms | **-71%** |
| `vtenantcontrolplane.kb.io` | 92 ms | 25 ms | **-73%** |
| `telemetry.kamaji.clastix.io` | 94 ms | _webhook removed_ | **-100%**
|

Cumulative Kamaji admission **impact** (`p99 × calls/s`) dropped from
**830 → 140 (-83%)**. The Kamaji webhooks moved from the top three of
the `apiserver_admission_webhook_admission_duration_seconds_*` impact
ranking down to positions #3-#4.

## Release note

```release-note
feat(kamaji): default to 2 replicas with soft pod anti-affinity and disable telemetry (handler + webhook entry). Reduces apiserver admission p99 for tenantcontrolplane mutating/validating webhooks by ~70% on multi-tenant clusters.
```

## Test plan

- [x] `helm template packages/system/kamaji` renders `replicas: 2`,
`--disable-telemetry`, the soft anti-affinity, and **no**
`telemetry.kamaji.clastix.io` entry in the
`ValidatingWebhookConfiguration`.
- [x] `patches/1.diff` applies cleanly against the freshly extracted
upstream chart (`patch --dry-run -p1 < patches/1.diff`).
- [x] `make update` re-applies the patch automatically.
- [x] Live deploy on a 22-TCP cluster: 3 pods Running (when overriden to
`replicaCount: 3` via cozystack-values), zero pod restarts, leader
election visible, `kamaji-webhook-service` Endpoints serving N IPs.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Telemetry can be disabled via configuration.
  * Controller configured for high availability with 2 replicas.
* Pod anti-affinity added to spread replicas across nodes for better
resilience.

* **Bug Fixes**
* Validating webhook is now omitted when telemetry is disabled to avoid
unintended behavior.

* **Chores**
* Update process improved to re-apply chart patches after extraction for
reproducible updates.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2671?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 19:31:51 +03:00
Aleksei Sviridkin 809a04a220 feat(kubevirt-cdi): make cloneStrategyOverride configurable (#2926) 
## What this PR does

- Exposes the CDI `cloneStrategyOverride` as a Helm value in
`packages/system/kubevirt-cdi` instead of hardcoding it in the CR
template.
- Default stays `csi-clone`, so **behavior is unchanged** for everyone.
- It can now be overridden via a cozystack `Package` (e.g.
`cloneStrategyOverride: copy`), the same way `uploadProxyURL` already is
— no fork needed.

### Why

The value was hardcoded, so the only way to change the clone strategy
was to fork the package. On some storage backends `csi-clone` is
problematic: with LINSTOR/DRBD a CSI clone is created co-located with
the source golden image and then relocated to optimal nodes
asynchronously; that async relocation (`migrate-disk`) can race the
post-clone resize CDI performs, occasionally leaving a volume whose DRBD
device is smaller than its PVC capacity. Being able to select `snapshot`
or `copy` per cluster, without maintaining a fork, is a useful escape
hatch.

This only adds a knob; it does not change the default or recommend a
strategy.

## Release note

```release-note
feat(kubevirt-cdi): make CDI cloneStrategyOverride configurable via values (default csi-clone)
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* The KubeVirt CDI clone strategy is now configurable via Helm chart
values, allowing selection of `csi-clone`, `snapshot`, or `copy` for
DataVolume cloning.
* The default remains `csi-clone` to preserve existing behavior and
ensure backward compatibility.
* **Documentation**
* Added the `cloneStrategyOverride` Helm value description and allowed
options to the chart’s configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 19:31:47 +03:00
Aleksei Sviridkin 901bb981ae fix(seaweedfs): configure postgres2 connection pool for the filer (#2906) 
## What this PR does

- Adds `WEED_POSTGRES2_CONNECTION_MAX_IDLE=20`,
`WEED_POSTGRES2_CONNECTION_MAX_OPEN=40` and
`WEED_POSTGRES2_CONNECTION_MAX_LIFETIME_SECONDS=600` to the filer
`extraEnvironmentVars` in `packages/system/seaweedfs/values.yaml`.

The vendored seaweedfs chart only ships connection pool settings for the
**mysql** store (`WEED_MYSQL_CONNECTION_MAX_*`), which cozystack does
not use. The **postgres2** store, which cozystack enables, has no pool
settings at all. As a result the Go `database/sql` pool runs with
`SetMaxIdleConns(0)`: every filer metadata lookup opens a fresh
PostgreSQL connection (TCP + TLS + SCRAM, ~300 ms each). With ~5
path-component lookups per S3 operation, this adds roughly **2 seconds
of latency to every S3 request**, even on an idle cluster.

### Measurements (production cluster, 12 nodes, 2 filers, CNPG backend)

Before / after applying these three variables, same test (sequential ops
through the S3 gateway):

| Operation | Before | After |
|---|---|---|
| PUT 4 KiB (p50) | 10.0 s | 0.28 s |
| GET 4 KiB (p50) | 3.0 s | 0.14 s |
| HEAD (p50) | 2.6 s | 0.16 s |
| Bounded LIST, 100 keys (p50) | 3.8 s | 0.37 s |

`pg_stat_activity` confirmed the root cause: before the fix every
backend was younger than ~200 ms (permanent connection churn); after the
fix connections are reused for minutes. Real-traffic p95 of
`SeaweedFS_filerStore_request_seconds` dropped from 1.3–6.5 s to ~0.18
s.

`max_open` is capped at 40 so that the default two filer replicas stay
below the CNPG default `max_connections` of 100.

## Release note

```release-note
fix(seaweedfs): configure postgres2 connection pool for the filer, removing ~2s of per-request S3 latency caused by PostgreSQL connection churn
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Optimized PostgreSQL connection pool configuration in SeaweedFS to
reduce latency and improve performance during metadata operations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 19:31:44 +03:00
Aleksei Sviridkin ec9c29948b [reloader] Set resources requests/limits on reloader container (#2631) 
## Summary

The upstream Stakater reloader chart ships with `resources: {}`, which
triggers a `no CPU limit` conformance warning on the `reloader`
Deployment.

Override `packages/system/reloader/values.yaml` at
`reloader.reloader.deployment.resources` with the exact values from the
chart's own commented example:

- `limits` : `cpu: 100m`, `memory: 512Mi`
- `requests` : `cpu: 10m`, `memory: 128Mi`

Reloader is a light controller that watches ConfigMap / Secret changes
and triggers workload restarts — small CPU envelope with enough memory
to track many objects in clusters with lots of namespaces.

## Test plan

- [ ] `helm template . --namespace cozy-reloader` from
`packages/system/reloader/` renders a `resources` block on the
`reloader-reloader` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `reloader-reloader has no
CPU limit` warning should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Set CPU/memory requests and limits on the reloader Deployment to clear
the "no CPU limit" conformance warning. Values follow the upstream
chart's commented example (limits 100m/512Mi, requests 10m/128Mi).
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated resource allocation configuration to define CPU and memory
parameters for system component deployments, enhancing infrastructure
stability and resource optimization.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2631)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 19:31:39 +03:00
Aleksei Sviridkin f2d66579f1 [metrics-server] Set resources requests/limits on metrics-server container (#2630) 
## Summary

The upstream metrics-server chart ships with `limits` commented out
(only `requests cpu 100m / memory 200Mi` are set), which triggers a `no
CPU limit` conformance warning on the `metrics-server` Deployment.

Override `packages/system/metrics-server/values.yaml` to keep the
upstream-recommended baseline `requests` and add `limits`:

- `limits` : `cpu: 500m`, `memory: 500Mi`
- `requests` : `cpu: 100m`, `memory: 200Mi`

Comfortable headroom on top of the [official sizing
formula](https://github.com/kubernetes-sigs/metrics-server#configuration)
(`100m + 1m/node`, `200Mi + 2Mi/node`), covering clusters up to ~150
nodes without needing to revisit. Larger clusters can override these
values further.

## Test plan

- [ ] `helm template . --namespace cozy-monitoring` from
`packages/system/metrics-server/` renders a `resources` block on the
`metrics-server` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `metrics-server has no CPU
limit` warning should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Set CPU/memory requests and limits on the metrics-server Deployment to
clear the "no CPU limit" conformance warning. Values follow the upstream
sizing recommendation (requests 100m/200Mi) with a comfortable limits
cap (500m/500Mi).
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Configured explicit resource requests and limits for the
metrics-server service (CPU: 100m-500m, Memory: 200Mi-500Mi) to ensure
consistent resource allocation.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2630)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 19:31:36 +03:00
Aleksei Sviridkin 4952909e4d [external-dns] Set resources requests/limits on external-dns container (#2629) 
## Summary

The upstream external-dns chart ships with `resources: {}`, which
triggers a `no CPU limit` conformance warning on the `external-dns`
Deployment.

Override `packages/system/external-dns/values.yaml` to set:

- `limits` : `cpu: 100m`, `memory: 128Mi`
- `requests` : `cpu: 10m`, `memory: 64Mi`

Modest sizing for a controller that just watches Service / Ingress
objects and pushes DNS records to the provider.

## Test plan

- [ ] `helm template . --namespace cozy-external-dns` from
`packages/system/external-dns/` renders a `resources` block on the
`external-dns` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `external-dns has no CPU
limit` warning should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Set CPU/memory requests and limits on the external-dns Deployment to
clear the "no CPU limit" conformance warning.
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated ExternalDNS resource allocation with optimized CPU and memory
limits and requests for improved resource efficiency.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2629)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 19:31:32 +03:00
Aleksei Sviridkin 24e3bf98b5 [bootbox] Add resources requests/limits to matchbox container (#2625) 
## Summary

The `matchbox` container in
`packages/extra/bootbox/templates/matchbox/deployment.yaml` had no
`resources` block, triggering a `no CPU limit` conformance warning on
the `bootbox-matchbox` Deployment (PXE/iPXE provisioning service).

Bootbox is a standalone chart (no vendored upstream), so the values are
set directly in the template — same pattern as #2614 (csi-resizer) and
#2624 (s3manager).

- `limits` : `cpu: 200m`, `memory: 256Mi`
- `requests` : `cpu: 50m`, `memory: 128Mi`

Modest sizing since matchbox is a lightweight HTTP server serving iPXE
profiles and boot assets from ConfigMaps.

## Test plan

- [ ] `helm template bootbox . --namespace tenant-root --show-only
templates/matchbox/deployment.yaml --set _namespace.host=example.com
--set _cluster.solver=letsencrypt` from `packages/extra/bootbox/`
renders the `resources` block on the `matchbox` container (verified
locally)
- [ ] Re-run the conformance/lint tool — the `matchbox has no CPU limit`
warning should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Add CPU and memory requests/limits to the matchbox container of the
bootbox Deployment to clear the "no CPU limit" conformance warning.
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Added Kubernetes resource limits and requests configuration to
optimize container resource allocation and improve deployment stability.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2625)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 19:31:29 +03:00
Aleksei Sviridkin 04a05f623c [bucket] Add resources requests/limits to s3manager container (#2624) 
## Summary

The `s3manager` container in
`packages/system/bucket/templates/deployment.yaml` had no `resources`
block, triggering a `no CPU limit` conformance warning on the bucket UI
Deployment.

The bucket package is a standalone chart (no vendored upstream), so the
values are set directly in the template — same pattern as the
csi-resizer fix in #2614 and the existing `capi-providers-cpprovider`
template.

- `limits` : `cpu: 500m`, `memory: 512Mi`
- `requests` : `cpu: 100m`, `memory: 128Mi`

Comfortable headroom for the lightweight S3 browser UI while keeping a
small baseline request.

## Test plan

- [ ] `helm template . --show-only templates/deployment.yaml` from
`packages/system/bucket/` renders the `resources` block on the
`s3manager` container (verified locally)
- [ ] Re-run the conformance/lint tool — the `s3manager has no CPU
limit` warning should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Add CPU and memory requests/limits to the s3manager container of the
bucket UI Deployment to clear the "no CPU limit" conformance warning.
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Improved deployment configuration with explicit resource allocation
settings to ensure optimal service stability and resource management.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2624)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 19:31:25 +03:00
Aleksei Sviridkin 6762d2d95c [goldpinger] Set resources requests/limits on goldpinger DaemonSet (#2623) 
## Summary

The upstream Bloomberg goldpinger chart ships with `resources: {}`,
which triggers a `no CPU limit` conformance warning on the
`goldpinger-daemon` container running on every node.

Override `packages/system/goldpinger/values.yaml` to set:

- `limits` : `cpu: 200m`, `memory: 256Mi`
- `requests` : `cpu: 50m`, `memory: 64Mi`

Comfortable margin for the ping-mesh fan-out on larger clusters while
keeping a small baseline request so the DaemonSet schedules everywhere.

## Test plan

- [ ] `helm template . --namespace cozy-goldpinger` from
`packages/system/goldpinger/` renders the `resources` block on the
`goldpinger-daemon` container of the DaemonSet (verified locally)
- [ ] Re-run the conformance/lint tool — the `goldpinger has no CPU
limit` warning should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Set CPU/memory requests and limits on the goldpinger DaemonSet to clear
the "no CPU limit" conformance warning.
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated Kubernetes resource requirements for the goldpinger component,
defining CPU limits (200m), memory limits (256Mi), CPU requests (50m),
and memory requests (64Mi).

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2623)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 19:31:22 +03:00
Aleksei Sviridkin 1aa6436922 [cert-manager] Set resources requests/limits on cert-manager pods (#2616) 
## Summary

The upstream cert-manager chart ships with `resources: {}` for all four
workloads, which triggers `no CPU limit` (and `no memory limit`)
conformance warnings on every pod.

Override `packages/system/cert-manager/values.yaml` (previously empty)
to set sane requests/limits sized to each component's role:

| Component | requests cpu/mem | limits cpu/mem | Notes |
|---|---|---|---|
| controller | 50m / 128Mi | 500m / 512Mi | Main workload — issues and
renews certs |
| webhook | 10m / 32Mi | 200m / 128Mi | Admission webhook, short-lived
requests |
| cainjector | 10m / 32Mi | 200m / 128Mi | CA bundle injection, light
watch loop |
| startupapicheck | 10m / 16Mi | 100m / 64Mi | One-shot Job at install
time |

Requests for `cainjector` (`10m` cpu / `32Mi` mem) match the example in
the upstream values.yaml comment.

## Test plan

- [ ] `helm template . --namespace cert-manager` from
`packages/system/cert-manager/` renders a `resources` block on all four
pods (verified locally for controller, webhook, cainjector and the
startupapicheck Job)
- [ ] Re-run the conformance/lint tool that originally flagged the
warnings on `cert-manager-cainjector` — should be clean on the other
three pods too
- [ ] CI passes

### Release note

\`\`\`release-note
Set CPU/memory requests and limits on cert-manager controller, webhook,
cainjector and startupapicheck pods to clear "no CPU limit" conformance
warnings.
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Added resource allocation configuration for cert-manager deployment,
specifying CPU and memory requests and limits for improved system
stability and resource management across core components.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2616)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 19:31:19 +03:00
Aleksei Sviridkin b2d00af495 [kubernetes] Add CPU/memory limits to csi-resizer sidecar (#2614) 
## Summary

- The `csi-resizer` container in
`packages/apps/kubernetes/templates/csi/deploy.yaml` was the only
sidecar in the `kcsi-controller` Deployment without `resources.limits`,
which triggers a "no CPU limit" conformance warning.
- Aligned it with the other six CSI sidecars in the same Deployment:
`limits` `cpu: 512m` / `memory: 512Mi` and `requests` `cpu: 125m` /
`memory: 128Mi`.
- No other files touched; vendored charts and other CSI drivers
(`nfs-driver`, `piraeus-operator`) are out of scope for this PR.

## Test plan

- [ ] `make -C packages/apps/kubernetes show NAME=test NAMESPACE=test`
renders the Deployment with the new `limits` block
- [ ] Re-run the conformance/lint tool that originally flagged the
warning — `csi-resizer has no CPU limit` should be gone
- [ ] CI passes

### Release note

\`\`\`release-note
Set CPU and memory limits on the csi-resizer sidecar of the kubevirt-csi
controller to align with the other CSI sidecars and clear the "no CPU
limit" conformance warning.
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated resource allocation for the CSI resizer component, increasing
CPU and memory requests to improve performance and reliability.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2614)

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 19:31:15 +03:00
mattia-eleuteri 8bc62cca1c feat(kamaji): default to 2 replicas, drop telemetry webhook 
The kamaji-controller-manager runs ~9 sub-controllers per
TenantControlPlane (bootstraptoken, coredns, konnectivity, kubeadm*,
kubelet, kubeproxy, etc.) inside a single Go process, AND serves the
admission webhooks for tenantcontrolplanes/datastores via the
kamaji-webhook-service ClusterIP. With the upstream default
replicaCount=1, every webhook RT shares CPU with the reconciler
goroutines. On a multi-tenant cluster (~20 TCPs), this pushes
apiserver-observed p99 for mtenantcontrolplane.kb.io and
vtenantcontrolplane.kb.io from the kamaji-side ~10ms to 85-95ms.

Three changes, each safe on small clusters and high-impact on large ones:

* replicaCount=2 — HA-safe baseline. --leader-elect is already enabled
  upstream, so only one replica reconciles tenants; the follower's CPU
  is fully available for webhook handlers, cutting p99 ~70% on busy
  clusters. On a single-node cluster both pods schedule normally.

* telemetry.disabled=true — drops both the --disable-telemetry handler
  and the telemetry.kamaji.clastix.io admission webhook entry from the
  ValidatingWebhookConfiguration. The latter requires patches/1.diff to
  wrap the telemetry block in a {{ if not .Values.telemetry.disabled }}
  conditional inside the vendored controller-gen file. Without the
  webhook entry, the apiserver no longer issues a TCP round-trip to a
  no-op handler on every TenantControlPlane CREATE/UPDATE/DELETE.

* podAntiAffinity (preferred) — spread the two replicas across nodes
  when possible. preferredDuringScheduling so single-node clusters
  still schedule (both pods colocated, which is acceptable).

Measured on a 22-TCP production cluster (Hikube, before/after):
mtenantcontrolplane p99 85 → 25 ms (-71%), vtenantcontrolplane p99
92 → 25 ms (-73%), telemetry webhook impact 291 → 0. Total Kamaji
admission impact -83% (830 → 140).

The patches/1.diff approach (re-applied by `make update`) mirrors the
pattern already used in fluxcd-operator and ingress-nginx and survives
the next clastix/kamaji chart bump.

Signed-off-by: mattia-eleuteri <mattia@hidora.io>
 2026-07-03 19:14:36 +03:00
mattia-eleuteriandClaude Opus 4.8 c47bb6edb0 feat(kubevirt-cdi): make cloneStrategyOverride configurable 
The CDI cloneStrategyOverride was hardcoded to csi-clone in the CR
template, so operators could only change it by forking the package.
Expose it as a Helm value (default csi-clone, behavior unchanged) so it
can be set via a cozystack Package override, the same way uploadProxyURL
already is.

This is useful where the csi-clone path is problematic on a given
storage backend (e.g. LINSTOR/DRBD, where a clone is co-located with the
source and relocated afterwards; the async relocation can race the
post-clone resize), letting operators select snapshot or copy without
maintaining a fork.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
 2026-07-03 19:14:33 +03:00
mattia-eleuteri 49cbea8c73 fix(seaweedfs): configure postgres2 connection pool for the filer 
The vendored seaweedfs chart ships connection pool settings only for the
mysql store, which cozystack does not use. The postgres2 store, which is
enabled, has no pool settings, so the Go sql pool keeps zero idle
connections and every filer metadata lookup opens a fresh PostgreSQL
connection (TCP + TLS + SCRAM, ~300ms each). With ~5 path lookups per S3
operation this adds roughly 2 seconds of latency to every S3 request,
regardless of cluster load.

Measured on a production cluster (12 nodes, 2 filers): PUT of a 4KiB
object went from 10s (p50) to 0.28s, HEAD from 2.6s to 0.16s after
setting these values; pg_stat_activity confirmed connections were
previously recycled every ~200ms and are now reused.

max_open is capped at 40 so that two filer replicas stay below the CNPG
default max_connections of 100.

Signed-off-by: mattia-eleuteri <mattia@hidora.io>
 2026-07-03 19:14:30 +03:00
Matthieu adb307cbae [goldpinger] Set resources requests/limits on goldpinger DaemonSet 
The Bloomberg goldpinger chart ships with resources: {}, which triggers
a 'no CPU limit' conformance warning on the goldpinger-daemon container
deployed on every node. Override the package values.yaml to set
limits 200m/256Mi and requests 50m/64Mi — comfortable margin for the
ping-mesh fan-out on larger clusters while keeping a small baseline
request.

Signed-off-by: Matthieu <matthieu@hidora.com>
 2026-07-03 19:14:27 +03:00
Matthieu bf54945c08 [reloader] Set resources requests/limits on reloader container 
The upstream Stakater reloader chart ships with resources: {}, which
triggers a 'no CPU limit' conformance warning on the reloader
Deployment. Apply the exact values from the chart's own commented
example (limits 100m/512Mi, requests 10m/128Mi) — reloader is a light
controller that watches ConfigMap/Secret changes and triggers restarts,
so a small CPU envelope with enough memory to track many objects fits.

Signed-off-by: Matthieu <matthieu@hidora.com>
 2026-07-03 19:14:21 +03:00
Matthieu 3c2133ce4b [metrics-server] Set resources requests/limits on metrics-server container 
The upstream metrics-server chart ships with limits commented out
(requests cpu 100m / memory 200Mi only), which triggers a 'no CPU
limit' conformance warning. Override the package values.yaml to keep
the upstream-recommended baseline requests and add limits 500m/500Mi
— comfortable headroom on top of the official sizing formula
(100m + 1m/node, 200Mi + 2Mi/node), covering clusters up to ~150 nodes
without needing to revisit.

Signed-off-by: Matthieu <matthieu@hidora.com>
 2026-07-03 19:14:18 +03:00
Matthieu 7f72e7bbe8 [external-dns] Set resources requests/limits on external-dns container 
The upstream external-dns chart ships with resources: {}, which triggers
a 'no CPU limit' conformance warning on the external-dns Deployment.
Override the package values.yaml to set limits 100m/128Mi and requests
10m/64Mi — modest sizing for a controller that just watches Service /
Ingress objects and pushes DNS records to the provider.

Signed-off-by: Matthieu <matthieu@hidora.com>
 2026-07-03 19:14:15 +03:00
Matthieu 0ea94e7d2f [bootbox] Add resources requests/limits to matchbox container 
The matchbox Deployment in the bootbox package had no resources block,
triggering a 'no CPU limit' conformance warning. Set limits 200m/256Mi
and requests 50m/128Mi — modest sizing since matchbox is a lightweight
HTTP server serving iPXE profiles and boot assets from ConfigMaps.
Bootbox is a standalone chart (no vendored upstream), so the values
are set directly in the template, matching the existing pattern in
csi-resizer and s3manager.

Signed-off-by: Matthieu <matthieu@hidora.com>
 2026-07-03 19:14:12 +03:00
Matthieu c8b180d91a [bucket] Add resources requests/limits to s3manager container 
The s3manager Deployment in the bucket package had no resources block,
triggering a 'no CPU limit' conformance warning. Set limits 500m/512Mi
and requests 100m/128Mi — comfortable headroom for the lightweight S3
browser UI. Bucket is a standalone chart (no vendored upstream), so
the values are set directly in the template, matching the existing
pattern in csi-resizer and capi-providers.

Signed-off-by: Matthieu <matthieu@hidora.com>
 2026-07-03 19:14:09 +03:00
Matthieu cd95040c58 [cert-manager] Set resources requests/limits on cert-manager pods 
The upstream cert-manager chart ships with resources: {} for all four
workloads (controller, webhook, cainjector, startupapicheck), which
triggers 'no CPU limit' conformance warnings on every pod. Override the
package values.yaml to set sane requests and limits sized to each
component's role: controller is the most loaded, webhook and cainjector
handle short-lived admission/injection work, and startupapicheck is a
one-shot Job.

Signed-off-by: Matthieu <matthieu@hidora.com>
 2026-07-03 19:14:05 +03:00
Matthieu 50c4080ba3 [kubernetes] Add CPU/memory limits to csi-resizer sidecar 
The csi-resizer container was the only sidecar in the kcsi-controller
Deployment without resources.limits, triggering a 'no CPU limit'
conformance warning. Align it with the other six CSI sidecars in the
same Deployment (limits 512m/512Mi, requests 125m/128Mi) so capacity
planning and audits stay uniform.

Signed-off-by: Matthieu <matthieu@hidora.com>
 2026-07-03 19:04:30 +03:00
Aleksei Sviridkin 58d61d5652 fix(ci): build api-gate detector from base tip when merge base predates it (#3193) 
## What this PR does

The `API Review Gate` required check builds the `cmd/api-gate` detector
from the PR's merge-base checkout so a PR cannot neuter the detector by
editing its own code. When a PR's merge base predates the commit that
introduced the gate (`fa871249c`), `cmd/api-gate` is absent from both
the merge base and the equally old head, so the head fallback also fails
to build and the check errors out before evaluating any API content.
This red-blocks every open PR whose merge base is older than the gate,
and neither a re-run nor an approving review clears it.

This adds the base branch tip (`origin/<base>`) as an intermediate build
source: prefer the merge-base checkout, fall back to the base tip when
the merge base lacks the tool, and bootstrap from the head only while
the tool exists nowhere on the base branch yet. The base tip is a
protected, non-PR-authored branch, so it preserves the "analyze the head
as data, not code" property. The API surface is still diffed against the
merge base; only where the detector binary is compiled changes. This
unblocks the pre-gate backlog without forcing every affected PR to
rebase.

The selection is kept inline in the workflow rather than in a helper
script: a helper would be read from the untrusted PR head, which both
reintroduces the failure for heads that predate the helper and lets a PR
steer the selection away from the trusted build.

Closes #3192

### Screenshots

N/A — CI-only change, no user-facing UI.

### Release note

```release-note
fix(ci): API Review Gate now builds its detector from the base branch tip when the PR merge base predates the gate, so pre-gate PRs stop failing the check without needing a rebase.
```
 2026-07-03 18:50:02 +03:00
Aleksei Sviridkin 361748ea6a fix(ci): build api-gate detector from base tip when merge base predates it 
The API Review Gate builds cmd/api-gate from the PR's merge-base checkout
so a PR cannot neuter the detector by editing its own code. When the merge
base predates the commit that introduced the gate, cmd/api-gate is absent
from both the merge base and the equally old head, so the head fallback
also fails to build and the required check errors out before evaluating
any API content. This blocks every open PR whose merge base is older than
the gate, and neither a re-run nor an approving review clears it.

Add the base branch tip as an intermediate build source: prefer the
merge-base checkout, fall back to the base tip when the merge base lacks
the tool, and bootstrap from head only while the tool exists nowhere on
the base branch yet. The base tip is a protected branch, not PR-authored,
so it preserves the "analyze head as data, not code" property. The API
surface is still diffed against the merge base; only where the detector
binary is compiled changes. This unblocks the backlog without forcing
every affected PR to rebase.

The selection stays inline in the workflow rather than in a helper script:
a helper would be read from the untrusted PR head, which both reintroduces
the failure for heads that predate the helper and lets a PR steer the
selection away from the trusted build.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-03 18:39:21 +03:00
IvanHunters 1baf126b1a fix(tenant): inherit full ancestor label chain from parent namespace (#2912) 
## Summary

`packages/apps/tenant/templates/namespace.yaml` labels every tenant
namespace
with `tenant.cozystack.io/<ancestor>` for each of its ancestors. These
labels
drive the `<ancestor>-egress` `CiliumClusterwideNetworkPolicy`
(`networkpolicy.yaml`): an ancestor tenant's pods may reach a namespace
only if
that namespace carries `tenant.cozystack.io/<ancestor>`.

The chain was derived by splitting `.Release.Namespace`, which only ever
reached
one level up. A direct child of `tenant-root` got `tenant-root` (its
parent
namespace *is* `tenant-root`), but a grandchild — `tenant-ktj-htdev`
created
under `tenant-ktj` — got only `tenant-ktj` and dropped `tenant-root`
entirely.
Without `tenant.cozystack.io/tenant-root` on a nested namespace, the
root
ingress controller cannot reach the proxy service in a nested
`kubernetes`
tenant, surfacing as `upstream timed out (110) while connecting to
upstream` in
the nginx logs.

## Fix

Derive the ancestor chain deterministically from the namespace name
instead of
from any cluster lookup. Every tenant descends from `tenant-root`, so
that label
is emitted unconditionally; the remaining ancestors are encoded in the
namespace
name itself, because `tenant.name` builds each child as
`<parent-namespace>-<word>` — so every progressive dash-prefix of the
name is a
real ancestor namespace (`tenant-ktj-htdev` yields `tenant-ktj` and
`tenant-ktj-htdev`). The logic lives in a new
`tenant.ancestorTenantLabels`
helper:

```yaml
labels:
  {{- include "tenant.ancestorTenantLabels" $tenantName | nindent 4 }}
```

This needs no cluster state, renders identically offline (so it is
unit-testable), and converges on every reconcile regardless of the order
in
which parent and child `HelmRelease`s reconcile — unlike a
parent-namespace
`lookup`, which helm-controller does not re-render on and which is
ordering-dependent. The `ownerReference` `lookup` is kept but made
non-fatal:
during a real reconcile the parent namespace always exists (the tenant
`HelmRelease` and its `cozystack-values` Secret live in it), so the
`ownerReference` is always emitted; when the lookup is empty (offline /
unit
test) it is simply omitted so the labels still render.

Migration `49` backfills the labels on existing clusters so they heal on
upgrade
regardless of per-tenant `HelmRelease` state (e.g. suspended or failed
releases), and `migrations.targetVersion` is bumped to `50` so the
runner
iterates it. The migration derives the same chain from each namespace
name,
is idempotent (`kubectl label --overwrite`) and best-effort
(per-namespace
failures are tolerated so a transient apiserver error cannot abort the
upgrade).

## Test plan

- [x] `helm template packages/apps/tenant` at depth 1/2/3 — emits
exactly the
expected ancestor set at each depth: depth-1 `{tenant-root,
tenant-ktj}`,
depth-2 adds `tenant-ktj-htdev`, depth-3 adds `tenant-ktj-htdev-foo`. No
      missing, no stray labels.
- [x] `helm unittest packages/apps/tenant` — new
`tests/namespace_ancestor_labels_test.yaml` pins the chain at depths
1/2/3
      plus a negative assertion; whole suite green.
- [x] `bats hack/cozystack-version-stamp.bats` — migration `49` stamps
the
version through the shared `stamp_cozystack_version` helper (all 10
tests
pass, including the guard that migrations >= 42 must use the helper).
- [x] Migration `49` ancestor derivation verified under `/bin/sh` to
match the
helm output byte-for-byte, and dry-runs cleanly under `set -euo
pipefail`.
- [ ] Live cluster: create a 3-level tenant (`root -> ktj -> htdev`),
verify
      `tenant-ktj-htdev` has all three `tenant.cozystack.io/*` labels.
- [ ] Live cluster: create a `kubernetes` tenant under `htdev` with
`addons.ingressNginx.hosts`, verify an external HTTP request reaches the
tenant ingress controller via the root-ingress proxy (no `upstream timed
      out`).

## Closes

- Closes #2810
- Closes #2171

```release-note
fix(tenant): tenant namespaces now carry the full `tenant.cozystack.io/<ancestor>` label chain (including `tenant-root`) at any nesting depth, restoring root-ingress reachability to nested tenants; a migration backfills the labels on existing clusters
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved tenant namespace label inheritance across multi-level tenant
hierarchies, ensuring all ancestor tenant labels are consistently
applied.
* **New Features**
* Standardized ancestor label generation to reliably include the full
tenant chain for derived tenant namespaces.
* **Tests**
* Added regression coverage for ancestor label presence and absence at
multiple hierarchy depths.
* **Chores**
* Added a migration to backfill missing ancestor labels for existing
tenant namespaces during platform upgrades.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 16:12:24 +03:00
myasnikovdaniil c5ff2178e7 docs(release): add changelog for v1.4.5 (#3083) 
This PR adds the changelog for release `v1.4.5`.

✅ Changelog has been automatically generated in
`docs/changelogs/v1.4.5.md`.
 2026-07-03 17:50:16 +05:00
myasnikovdaniil 6f39beffb6 docs(release): add changelog for v1.5.2 (#3162) 
This PR adds the changelog for release `v1.5.2`.

✅ Changelog has been automatically generated in
`docs/changelogs/v1.5.2.md`.
 2026-07-03 17:14:55 +05:00
cozystack-ci[bot] ee29392584 docs: add changelog for v1.5.2 
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
 2026-07-03 10:22:23 +00:00
cozystack-ci[bot] a8220fa21f docs: add changelog for v1.4.5 
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
 2026-07-03 10:18:23 +00:00
myasnikovdaniil 1e5f53881f fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) (#3146) 
## What

Backports the upstream Kamaji fix
[clastix/kamaji#1122](https://github.com/clastix/kamaji/pull/1122)
(which fixes clastix/kamaji#1115) as a source patch on top of the pinned
`26.3.6-edge` build. No version bump, no CRD/RBAC changes.

This lands the fix on `main`; the `backport` / `backport-previous`
labels fan it out to `release-1.5` and `release-1.4` via the backport
automation. Supersedes the per-release PRs #3021 (release-1.5) and #3020
(release-1.4).

## Why

Deleting a Tenant whose etcd `DataStore` has **0 referencing
`TenantControlPlane`s** — e.g. a tenant deleted before any Kubernetes
app / TCP is ever created in it — leaves the `DataStore` stuck
`Terminating` with finalizer `kamaji.clastix.io/TenantControlPlane`.
That hangs `helm uninstall <etcd>` (which waits by default), freezes the
etcd HelmRelease at `status=uninstalling`, and wedges the tenant
namespace in `Terminating` indefinitely. Tracked in #3018.

Root cause: in `26.3.x-edge`, DataStore deletion drives an
`AllowedDeletion` condition that only transitions `False → True`. With 0
referencing TCPs at deletion time the condition is never initialized, so
the reconcile falls through to logging `DataStore can be safely deleted`
and the deferred finalizer-removal (gated on `IsStatusConditionTrue`)
never runs — with no requeue to retry. The pinned `26.3.6-edge` predates
the upstream fix (merged 2026-04-14, first released in `26.4.4-edge`).

## Change

Adds
`packages/system/kamaji/images/kamaji/patches/fix-datastore-unused-deletion.diff`,
applied by the existing `RUN git apply /patches/*.diff` step in the
kamaji image build. The one-line guard adds the missing 0-TCP path:

```go
- if meta.IsStatusConditionFalse(..., DataStoreConditionAllowedDeletionType) {
+ if meta.IsStatusConditionFalse(..., DataStoreConditionAllowedDeletionType) || len(tcpList.Items) == 0 {
```

## Test

- `git apply --check` confirms the patch applies cleanly to the
`26.3.6-edge` source pinned on `main`; after applying, the guard is
present at `controllers/datastore_controller.go`.
- Reproduced the deadlock on a v1.4.x cluster (tenant deleted before any
TCP existed) and confirmed the manual unstick; this patch removes the
need for it.

Workaround for an already-stuck DataStore (0 TCP refs): `kubectl patch
datastore <name> --type=merge -p '{"metadata":{"finalizers":null}}'`.

Relates to #2885 (full kamaji bump for CVEs — supersedes this patch once
it lands).

```release-note
Fix tenant namespace stuck `Terminating` when an etcd DataStore with no referencing TenantControlPlanes is deleted (Kamaji datastore unused-deletion deadlock).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Data stores are now recognized as unused even when no matching tenant
control plane resources are found, improving deletion handling in that
case.
* This prevents inactive data stores from being left behind when there
are no associated matches.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 14:52:11 +05:00
myasnikovdaniil 5b261b9daf fix(kubernetes): gate vmop cert-manager-crds dependency on certManager.enabled (#3130) 
## What this PR does

The `victoria-metrics-operator` HelmRelease in the `kubernetes` app
chart is rendered when `addons.monitoringAgents.enabled` is true, but it
listed `cert-manager-crds` in its `dependsOn` **unconditionally**. The
`cert-manager-crds` HelmRelease is only created when
`addons.certManager.enabled` is true. These are two independent toggles.

### The valid combination that triggers it

`addons.monitoringAgents.enabled: true` + `addons.certManager.enabled:
false` (both default-able, both schema-valid). In this configuration
vmop renders but the dependency it points at never does.

### Before / after

Before:
```yaml
  dependsOn:
  - name: {{ .Release.Name }}-cilium
    namespace: {{ .Release.Namespace }}
  - name: {{ .Release.Name }}-cert-manager-crds   # always present
    namespace: {{ .Release.Namespace }}
```

After:
```yaml
  dependsOn:
  - name: {{ .Release.Name }}-cilium
    namespace: {{ .Release.Namespace }}
  {{- if .Values.addons.certManager.enabled }}
  - name: {{ .Release.Name }}-cert-manager-crds
    namespace: {{ .Release.Namespace }}
  {{- end }}
```

### The cascade

With cert-manager disabled, Flux reports on the vmop HelmRelease:
`unable to get '...cert-manager-crds' dependency: helmreleases... not
found`. Because vmop never goes Ready, the dependency chain fails behind
it: `monitoring-agents` (dependsOn `cozy-victoria-metrics-operator`) →
not Ready → `vertical-pod-autoscaler` (dependsOn `monitoring-agents`) →
not Ready.

### Why gating (not unconditional CRDs) is correct

vmop does not require cert-manager. The vendored vmop chart
(`packages/system/victoria-metrics-operator`) defaults
`admissionWebhooks.certManager.enabled` to `false`, and the cozystack
wrapper does not override it. With that default the chart self-generates
the webhook CA/cert (`genCA`/`genSignedCert` in `_helpers.tpl`); the
cert-manager Issuer/Certificate path is fully gated behind
`certManager.enabled`. So the dependency only belongs there when
cert-manager is actually enabled.

Principle: a `dependsOn` must never reference a HelmRelease that isn't
created under the same value combination.

### How it was verified

`helm template` of the template with a CI values fixture:

`certManager.enabled=false` → cilium only:
```yaml
  dependsOn:
  - name: test-cilium
    namespace: tenant-root
```

`certManager.enabled=true` → cilium + cert-manager-crds:
```yaml
  dependsOn:
  - name: test-cilium
    namespace: tenant-root
  - name: test-cert-manager-crds
    namespace: tenant-root
```

A new `helm unittest` suite
(`tests/victoria-metrics-operator_test.yaml`) asserts the `dependsOn`
for both toggle states. Full package suite passes (13 suites / 125
tests).

### Release note
```release-note
fix(kubernetes): enabling monitoring agents without cert-manager no longer blocks the victoria-metrics-operator HelmRelease (and the monitoring-agents / vertical-pod-autoscaler chain behind it) on a non-existent cert-manager-crds dependency
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Summary by CodeRabbit

* **New Features**
* Added automated template test coverage for the Victoria Metrics
Operator HelmRelease.
* Validates rendered release name and dependency behavior across key
configuration scenarios.

* **Bug Fixes**
* Updated the HelmRelease dependency wiring so the cert-manager CRDs
dependency is included only when cert-manager is enabled.
* Preserved existing monitoring and outer conditional behavior while
refining dependency rendering.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 14:06:49 +05:00
Myasnikov Daniil 47638f8569 fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) 
Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes left the DataStore stuck Terminating (finalizer kamaji.clastix.io/TenantControlPlane), hanging the etcd helm uninstall and wedging the tenant namespace. Backports clastix/kamaji#1122 (fixes clastix/kamaji#1115) as a source patch on the pinned 26.3.6-edge build. Refs #3018.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-03 12:59:38 +05:00
myasnikovdaniil dad78bd57e fix(ci): nest backport conflict_resolution under experimental input (#3155) 
## What this PR does

The Automatic Backport workflow passed `conflict_resolution:
draft_commit_conflicts` as a **top-level** input to
`korthout/backport-action`, but the action only reads
`conflict_resolution` from inside its `experimental` JSON input. Passed
top-level it was silently ignored — the action run logs warn `Unexpected
input(s) 'conflict_resolution'` — so it fell back to the `experimental`
default of `fail`. On a cherry-pick conflict the action then created
**no** pull request and only left a "Backport failed" comment on the
source PR (this bit #3121 and #3119).

This moves the key into the `experimental` input, so a conflicting
backport instead opens a **draft PR with the first conflict committed**
for maintainers to resolve in place.

No version bump: the pinned `v3.2.1` already supports
`draft_commit_conflicts` — the bug was purely the input placement.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Backport pull requests now handle conflicts more reliably and open as
draft PRs when conflicts are detected.
* Improved configuration handling so the expected conflict behavior is
applied consistently during backport automation.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-03 12:53:45 +05:00
IvanHunters 59964043e3 fix(platform): migration 49 backfills tenant ancestor labels 
The tenant chart fix makes new and re-reconciled tenant namespaces
self-compute their full tenant.cozystack.io/<ancestor> chain, but existing
namespaces whose HelmRelease does not promptly re-reconcile (suspended or
failed releases) keep the incomplete labels — and with them the broken
<ancestor>-egress reachability that surfaces as upstream timeouts to nested
tenants. Backfill the labels directly so clusters heal on upgrade regardless
of per-tenant HelmRelease state.

Bump targetVersion to 50 so run-migrations.sh iterates the new script
(seq CURRENT..49); the missing-file guard otherwise never fires. Best-effort
and idempotent: kubectl label --overwrite adds only the listed labels, and
per-namespace failures are tolerated so a transient apiserver error cannot
abort the platform upgrade.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-07-03 09:54:12 +03:00
IvanHunters 1ac9398a4b fix(tenant): emit full ancestor label chain on tenant namespaces 
The tenant.cozystack.io/<ancestor> namespace labels drive the
<ancestor>-egress CiliumClusterwideNetworkPolicy, which lets an ancestor
tenant reach a descendant namespace only when the descendant carries
tenant.cozystack.io/<ancestor>. namespace.yaml derived that chain by
splitting .Release.Namespace, which only reached one level up: a direct
child of tenant-root got tenant-root, but a grandchild (tenant-ktj-htdev
under tenant-ktj) got only tenant-ktj and dropped tenant-root entirely.
The root ingress controller in tenant-root then could not reach nested
tenants, surfacing as upstream timeouts.

Derive the chain deterministically from the namespace name instead. Every
tenant descends from tenant-root, and tenant.name builds each child as
<parent-namespace>-<word>, so every progressive dash-prefix of the name is
a real ancestor namespace. This needs no cluster lookup, renders offline
(so it is unit-testable), and converges on every reconcile regardless of
parent/child HelmRelease ordering. The ownerReference lookup is made
non-fatal so the template still renders when the parent namespace lookup
is empty; in a real reconcile the parent always exists and the
ownerReference is always emitted.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-07-03 09:51:49 +03:00
Myasnikov DaniilandClaude Opus 4.8 2b1d3e904d fix(kubernetes): gate vmop cert-manager-crds dependency on certManager.enabled 
The victoria-metrics-operator HelmRelease is gated on
monitoringAgents.enabled but listed cert-manager-crds in dependsOn
unconditionally, while the cert-manager-crds HelmRelease is only
created when certManager.enabled is true. The valid combination
monitoringAgents.enabled=true + certManager.enabled=false therefore
left vmop blocked on a non-existent dependency
("...cert-manager-crds ... not found"), cascading to monitoring-agents
and vertical-pod-autoscaler. Gate the dependency on the same condition
that creates it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-03 11:18:46 +05:00
scooby87 71ca2b1bae fix(mariadb): re-guard replication so single-replica passes the operator webhook (#3180) 
## What this PR does

Fixes a regression where **single-replica MariaDB can no longer be
created**.

[PR #2279](https://github.com/cozystack/cozystack/pull/2279) made
`replication.enabled: true` unconditional in the MariaDB CR so the
operator would always create `<release>-primary` / `<release>-secondary`
Services (expected by the dashboard RBAC and the ApplicationDefinition).
But the vendored mariadb-operator's validating webhook
(`vmariadb-v1alpha1.kb.io`) **rejects `replication.enabled: true` when
`spec.replicas == 1`** — replication needs a primary + at least one
replica. As a result, single-replica installs never became Ready (the
HelmRelease failed at admission).

The webhook rule lives in the operator's Go code (not in the vendored
CRD — there is no CEL rule tying replication to replica count), so the
failing path is reproduced by an e2e test rather than a schema
assertion.

### Fix — re-guard replication, teach consumers about the bare service

The operator provisions a bare `<release>` Service in **both**
topologies; `-primary`/`-secondary` exist **only** when replication is
on (`replicas>1`). Instead of forcing replication for everyone, this PR
lets replication track the replica count and makes the service-name
consumers handle the single-replica bare service:

- **`mariadb.yaml`** — gate `replication` on `replicas>1`; restore
single-replica external access via `service.type: LoadBalancer`, and
keep `primaryService: LoadBalancer` for `replicas>1` only. (Effectively
reverts #2279's two hunks.)
- **`backup-cronjob.yaml`** — single-replica dumps target the bare
service (no `-secondary` exists without replication); `replicas>1` still
offloads to `-secondary`.
- **`dashboard-resourcemap.yaml`** — always grant RBAC on the bare
service, plus `-primary`/`-secondary` when `replicas>1`, matching what
the ApplicationDefinition lists.
- **`mariadb-rd`** — list the bare service in `services.include` so
single-replica installs are visible in the dashboard.
- **`hack/e2e-apps/mariadb.bats`** — new `Create single-replica MariaDB`
test that reproduces the webhook rejection and guards against
regression.

Verified: `helm template` at `replicas=1`/`2` renders the intended
CR/RBAC/backup host; `helm unittest` passes; `update-crd.sh` keeps the
RD stable; `select-e2e.sh` maps the change to the mariadb suite.

### Release note

```release-note
fix(mariadb): single-replica instances can be created again — replication is now enabled only for replicas>1 (matching the operator webhook), while the dashboard, RBAC, and backups correctly target the bare service for single-replica setups
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an end-to-end test covering MariaDB deployments with a single
replica, including validation of the main service, endpoints,
StatefulSet replica count, and metrics components.

* **Bug Fixes**
* Updated MariaDB chart behavior to correctly handle single-replica vs
multi-replica configurations for external access, replication settings,
and backup host selection.
* Refined dashboard RBAC and secret inclusion so access applies to the
main MariaDB service in addition to replica-specific services.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-02 22:54:14 +03:00
Andrei Kvapil 41fd08e2ba feat(keycloak): enable UPDATE_EMAIL required action in cozy realm (#3175) 
## What this PR does

The portal user-settings page lets a user change their email via a
Keycloak `kc_action=UPDATE_EMAIL` redirect. That flow only works when
the **UPDATE_EMAIL required action is enabled at the realm level** — and
on a fresh realm it is registered but **disabled by default**.

`update-email` is a supported, default-on feature since Keycloak 26.4
(keycloak/keycloak#39722), so the provider is already loaded on the 26.x
server this repo ships — **no `KC_FEATURES` change and no Keycloak
restart are needed**. The `edp-keycloak-operator` has no CRD field for
required actions, so this adds a small one-shot, idempotent
`post-install`/`post-upgrade` Job to `keycloak-configure` that enables
it via `kcadm.sh`, waiting for the operator to reconcile the realm
first.

Verified against a live Keycloak 26.5.2: the `UPDATE_EMAIL` action goes
from `enabled=false` to `enabled=true`, and re-running the Job is a
no-op.

### Release note

```release-note
feat(keycloak): enable the UPDATE_EMAIL required action in the cozy realm so users can change their email from the portal
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Keycloak now automatically enables the email update action during
install and upgrade, helping users change their email address in the
`cozy` realm.
* The configuration now includes the required Keycloak container image
for this setup task.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-02 21:02:18 +02:00
scooby87 e4c6802863 fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry (#3171) 
## What this PR does

Fixes #3088.

The post-delete cleanup Jobs in the **harbor** and **mariadb** charts
hardcoded the cleanup image as `docker.io/clastix/kubectl:v1.32` — a
moving Docker Hub tag. This:

- **broke reproducibility / was a supply-chain risk** — `v1.32` is
mutable and can be re-pushed to a different image;
- **could not be satisfied on air-gapped / mirrored clusters** — the
hardcoded `docker.io` bypassed the cluster images registry, and it
failed at the worst moment: during **uninstall**, when the `post-delete`
hook runs and a stuck Job stalls release teardown.

### Changes

- **New canonical helper `cozy-lib.image`**
(`packages/library/cozy-lib/templates/_cozyconfig.tpl`). Given a
registry-relative image path and the global context, it prefixes
`cozy-lib.images-registry` when set (`<registry>/<image>`) and returns
the image unchanged when empty — **with no leading `/`**, so standard
installs still resolve from the default registry. This establishes the
previously-undefined "how to reference a cluster image" pattern
(`images-registry` was defined but never consumed).
- **Digest-pinned image** in both cleanup hooks:
`clastix/kubectl:v1.32@sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c`
(the immutable multi-arch OCI index for the `v1.32` tag; the same digest
already pinned for `postgres-operator`'s webhook hook).
- **Renovate keeps the pin fresh.** The repo disables the helm-values
manager (`enabledManagers`), and the ref is assembled at render time by
the helper rather than being a Dockerfile / go.mod dependency, so the
built-in managers can't see it. A `custom.regex` manager over the hook
templates tracks `clastix/kubectl` by tag+digest — mirroring the
existing `extra/etcd` manager — so the supply-chain pin doesn't silently
age.
- **Hardened `cozy-lib.images-registry`** against a nil
`.Values._cluster` so the helper never crashes the render — relevant
precisely because this hook renders during teardown.
- **Helm unit tests** in both charts assert the rendered image for three
cases: empty registry (default), registry set, and the no-leading-slash
edge case. Assertions match by **pattern** (routed + digest-pinned + no
leading slash) rather than an exact digest, so a Renovate bump keeps
them green.

### Verification

- `helm unittest` passes for both charts (harbor: 16, mariadb: 10).
- Real `helm template` renders confirmed for both charts:
  - empty registry → `clastix/kubectl:v1.32@sha256:b9ef…`
- `images-registry=registry.internal:5000` →
`registry.internal:5000/clastix/kubectl:v1.32@sha256:b9ef…`
- The Renovate `matchStrings` regex verified to capture
`currentValue=v1.32` and the digest from both hook templates.

### Note on the wider footprint

The same `docker.io/clastix/kubectl:v1.32` hardcode is already merged on
`main` in several other cleanup hooks (bucket, qdrant, gateway,
seaweedfs, kubernetes, tenant, dashboard, keycloak-configure) and
appears in the open PRs #3170 (etcd), #3094 (monitoring), #3072
(clickhouse). This PR fixes harbor + mariadb per the issue and
establishes the `cozy-lib.image` + Renovate pattern; migrating the
remaining hooks is a follow-up.

### Release note

```release-note
fix(cozy-lib,harbor,mariadb): digest-pin the harbor and mariadb post-delete cleanup-hook kubectl image and route it through the cluster images registry (`cozy-lib.images-registry`) via the new `cozy-lib.image` helper, so mirrored / air-gapped installs can resolve it and uninstall no longer depends on a moving Docker Hub tag; a custom Renovate manager keeps the digest fresh
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Updated post-delete cleanup hook Jobs to use digest-pinned container
images for more secure, repeatable execution.
* Improved image reference rendering so cleanup hooks work correctly
with or without an image registry configured, avoiding invalid
leading/trailing slash issues.
* **Tests**
* Extended cleanup hook rendering tests to validate digest pinning and
correct registry prefix behavior, including edge cases.
* **Chores**
* Enhanced automated dependency detection to treat digest-pinned hook
image references as stable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-02 21:38:43 +03:00
scooby87 6bf836db30 fix(dashboard): show LoadBalancer external IP on the app Services tab (#3172) 
## What this PR does

For an application exposed with `external: true` (which creates a `type:
LoadBalancer` Service), the app detail → **Services** tab showed the
service Type and Cluster IP but not the external IP assigned by MetalLB
— so there was no way to learn the connection address from the per-app
view. The pre-1.4.0 dashboard showed this, so it read as a regression
(reported in #2789).

Root cause: `ServicesTab.tsx` queried Services with a spec-only local
type (`{ type, clusterIP, ports }`) and never read
`status.loadBalancer.ingress[]`. The data was already reachable —
`ExternalIpsPage.tsx` renders it — so this was a wiring gap in the
per-app tab, not missing data.

Changes:
- Add a `ServiceStatus` type and extend the `useK8sList` query to
`K8sResource<ServiceSpec, ServiceStatus>`.
- Add an **External IP** column: `ip ?? hostname ?? "Pending"` for
`LoadBalancer` services, `—` otherwise — mirroring `ExternalIpsPage`.
- Add a co-located regression test (`ServicesTab.test.tsx`) covering the
IP, hostname→Pending fallback, and the ClusterIP no-external case.

Verified: `pnpm typecheck` clean, `pnpm test` green (320 tests, incl. 3
new), `eslint` clean on the changed files.

Fixes #3111

### Screenshots
<img width="1220" height="572" alt="Снимок экрана 2026-07-02 в 14 12 11"
src="https://github.com/user-attachments/assets/82cd1da8-ff81-4b95-a6bf-18729b6c85c7"
/>


### Release note

```release-note
fix(dashboard): show the external LoadBalancer IP on the per-app Services tab
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Added an **External IP** column to the service details view.
* Load balancer services now show the external IP or hostname when
available.

* **Bug Fixes**
* Fixed the external IP display for load balancer services so it no
longer misses valid IPs.
* Improved fallback behavior by showing **Pending** when no external
endpoint is assigned.
* Non-load balancer services now show an em dash instead of misleading
external IP text.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-02 20:06:26 +03:00
scooby87 a16268152d fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant module delete (#3090) (#3170) 
## What this PR does

When the `etcd` tenant module is disabled (`etcd: false`), the etcd
HelmRelease is pruned but two classes of resources leak into the tenant
namespace (reported in #3090):

- **Data PVCs** — the etcd-operator StatefulSet `volumeClaimTemplate`
PVCs `data-etcd-<ordinal>` are never garbage-collected on `EtcdCluster`
deletion.
- **TLS secrets** — the five `etcd-*-tls` secrets survive the uninstall:
- `etcd-ca-tls` / `etcd-client-tls` are created by this chart as
`pre-install` hooks with `helm.sh/resource-policy: keep`, so Helm
deliberately keeps them;
- `etcd-peer-ca-tls` / `etcd-server-tls` / `etcd-peer-tls` are populated
by cert-manager from `Certificate` resources and carry no owner
reference back to the chart.

This adds a `post-delete` cleanup hook
(`packages/extra/etcd/templates/hooks/cleanup.yaml`), modeled on the
accepted seaweedfs (#3092) and monitoring (#3091) sibling fixes, that:

- deletes the data PVCs by name, one per replica (`data-etcd-0 …
data-etcd-<replicas-1>`);
- deletes the five `etcd-*-tls` secrets by name.

`check-release-name` pins the release name to `etcd`, so every target
name is deterministic and no sibling workload shares the `data-etcd-*`
prefix in a tenant namespace — deletion by name is safe. cert-manager
TLS secrets carry only the cluster-wide `controller.cert-manager.io/fao`
label, so the RBAC grants `delete` by explicit `resourceNames` rather
than an unsafe label sweep.

The cleanup Job is hardened: non-root, read-only root fs, dropped
capabilities, `seccompProfile: RuntimeDefault`, resource
requests/limits, `activeDeadlineSeconds`. Deletes use
`--ignore-not-found` and failures are logged but never block teardown.

A Helm unit-test suite (`tests/cleanup_test.yaml`) asserts the hook
metadata/RBAC/hardening and that the rendered commands target exactly
the reported PVCs and secrets, including scaling the PVC list to the
configured replica count.

> [!NOTE]
> The **stuck HelmRelease finalizer** noted in #3090 is the
platform-wide sharded-helm-controller side-bug (tracked separately); it
affects every `chartRef`-based tenant module identically, not just etcd.
This PR fixes the PVC and secret orphans the etcd chart itself owns,
matching the scope of the merged seaweedfs/monitoring fixes.

> [!WARNING]
> **Disabling the etcd module now permanently deletes its data PVCs and
the etcd data they hold.** This is intentional and fixes the storage
leak. Back up before disabling if the data matters.

Fixes #3090.

### Release note

```release-note
fix(etcd): garbage-collect the orphaned etcd data PVCs (data-etcd-*) and the five etcd-*-tls secrets left behind when a tenant's etcd module is disabled. WARNING: disabling etcd now permanently removes its data PVCs and the data they hold.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a Helm post-delete cleanup hook for etcd that removes
per-replica etcd PVCs and the related TLS secrets when a release is
deleted.
* Cleanup permissions and targets now scale with replica count,
including safe handling when replicas are set to zero.
* **Tests**
* Added Helm unit tests validating hook annotations/order, scoped RBAC,
secure job hardening, and the exact deletion behavior for PVCs/secrets
(including higher-replica scenarios).
* **Chores**
  * Marked the `test` make target as phony for the etcd package.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-02 18:21:36 +03:00
Алексей Артамонов f8e4222b7d test(mariadb): fix operator MariaDB CR name in single-replica failure dump 
The failure-path dump queried mariadbs.k8s.mariadb.com with $name, but the
operator CR is named after the Helm release ({{ .Release.Name }} = mariadb-$name),
so the dump would hit NotFound instead of showing the rejected CR.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
 2026-07-02 17:47:01 +03:00
Алексей Артамонов a76c5929fd fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry 
The post-delete cleanup Jobs in the harbor and mariadb charts hardcoded
`docker.io/clastix/kubectl:v1.32` — a moving tag pulled directly from
Docker Hub. This broke build reproducibility (the tag can be re-pushed),
was a supply-chain risk, and could not be satisfied on air-gapped /
mirrored clusters, where it failed at the worst moment: during uninstall,
when the post-delete hook runs and stalls release teardown.

Add a canonical `cozy-lib.image` helper that prefixes a registry-relative
image path with `cozy-lib.images-registry` when set and returns it
unchanged (no leading "/") when empty, so standard installs resolve from
the default registry and mirrored installs resolve internally. Consume it
from both cleanup hooks with the image digest-pinned to the immutable
multi-arch index sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c
(the same digest already pinned for postgres-operator's webhook hook).

Keep the pin fresh with a custom Renovate manager: the repo disables the
helm-values manager (enabledManagers), and the ref is built at render time
by the helper rather than being a Dockerfile/go.mod dependency, so the
built-in managers cannot see it. A regex manager over the hook templates
tracks clastix/kubectl by tag+digest — mirroring the existing extra/etcd
manager — so the supply-chain pin does not silently age.

Also harden `cozy-lib.images-registry` against a nil .Values._cluster so
the helper never crashes the render — relevant precisely because the hook
renders during teardown. Helm unit tests assert the rendered image (routed,
digest-pinned, no leading slash) by pattern rather than an exact digest, so
a Renovate bump keeps them green.

Closes #3088

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
 2026-07-02 17:44:59 +03:00
Andrei Kvapil caa6607f4d feat(keycloak): add optional KMS-encrypting database proxy (#3003) 
## What this PR does

Adds an optional, flag-gated **column-level PII encryption** feature to
the `keycloak` system package. When `encryption.enabled=true`, a
`keycloak-kms-proxy` Deployment + Service is placed on the wire between
the Keycloak StatefulSet and its CloudNativePG database, and Keycloak's
`KC_DB_URL_HOST`/`KC_DB_URL_PORT` are repointed at the proxy. The proxy
transparently encrypts Keycloak PII columns (username, email, first/last
name, `pii-*` attributes, credential secrets) before they reach
PostgreSQL and decrypts them on reads; the data-encryption key is
wrapped by a KMS and never lives inside the database.

The proxy lives in its own repository:
https://github.com/cozystack/keycloak-kms-proxy (image
`ghcr.io/cozystack/keycloak-kms-proxy:v0.1.0`).

**KMS backend** is selectable via `encryption.kms.backend`:
- `static` (default): a self-generated 32-byte KEK kept in a Secret —
works on any cluster, suitable for evaluation.
- `vault-transit`: the KEK lives in HashiCorp Vault (recommended for
production; supports KEK rotation).

The feature is **off by default**, so existing installs are unaffected —
`KC_DB_URL_HOST` keeps sourcing from the CNPG `keycloak-db-app` secret.

### Changes
- `values.yaml`: new `encryption` block.
- `templates/proxy.yaml` (new): gated Deployment + Service (+
self-generated KEK Secret for the static backend), hardened pod
(nonroot, read-only rootfs, dropped caps, seccomp).
- `templates/sts.yaml`: conditional DB-endpoint override when encryption
is enabled.
- `tests/encryption_test.yaml` (new): helm-unittest covering on/off and
both KMS backends.

`helm unittest .` → 11/11 pass.

### Follow-ups (not in this draft)
- Backend TLS re-origination to the CNPG endpoint + CA validation.
- Liquibase bootstrap-window handling (`KKP_LENIENT`).
- Offline backfill Job for migrating existing data.
- HA via a shared DEK set (currently single replica).

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(keycloak): add optional KMS-encrypting database proxy for column-level PII encryption (off by default; static KEK or Vault Transit)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added optional transparent encryption for Keycloak database sensitive
fields
* Users can configure encryption scope and select key management backend
options

* **Tests**
* Added comprehensive test coverage validating encryption feature across
multiple configuration scenarios

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-02 16:41:35 +02:00
Алексей АртамоновandClaude Opus 4.8 9472bb3412 fix(etcd): clamp cleanup PVC resourceNames so replicas:0 cannot widen RBAC 
Address CodeRabbit review on #3170: the PVC cleanup Role generates its
resourceNames from .Values.replicas. values.schema.json sets no minimum, so an
explicit replicas: 0 rendered an empty resourceNames list — and an empty
resourceNames in an RBAC rule matches every PVC in the namespace (a wildcard),
silently undoing the least-privilege scoping the previous commit added.

Clamp the count to max(1, replicas) via a shared $replicaCount used by both the
Role resourceNames range and the delete command, so at least data-etcd-0 is
always listed (it exists for any running cluster and the delete uses
--ignore-not-found). Add a regression test asserting replicas: 0 still yields a
name-scoped Role and a single-name delete rather than a namespace-wide grant.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
 2026-07-02 17:36:03 +03:00
Алексей АртамоновandClaude Opus 4.8 4a06df92e7 fix(etcd): scope cleanup Role to the data-etcd PVC names and assert hook metadata 
Address CodeRabbit review on #3170:
- Tighten the cleanup Role's PVC rule from a namespace-wide get/list/delete to
  get/delete by explicit resourceNames (data-etcd-<ordinal>, one per replica),
  mirroring the least-privilege pattern already used for the TLS secrets. The
  Job only ever deletes those deterministic names, so list is unnecessary and a
  regression can no longer reach a sibling workload's PVC.
- Extend the Role and RoleBinding unit tests to assert helm.sh/hook and
  helm.sh/hook-delete-policy (not just the weight), so the RBAC cannot regress
  into normal release resources, and assert the PVC resourceNames.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
 2026-07-02 15:58:09 +03:00
Алексей АртамоновandClaude Opus 4.8 a92f060f63 fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant module delete (#3090) 
When the etcd tenant module is disabled (etcd: false), the etcd-operator
StatefulSet data PVCs (data-etcd-<ordinal>) are never garbage-collected, and
the five etcd-*-tls secrets are left orphaned in the tenant namespace:
etcd-ca-tls / etcd-client-tls are pre-install hooks this chart marks with
helm.sh/resource-policy: keep, and etcd-peer-ca-tls / etcd-server-tls /
etcd-peer-tls are cert-manager-issued secrets with no owner reference back to
the chart. None are reclaimed by the Helm uninstall.

Add a post-delete cleanup Job (modeled on the seaweedfs/monitoring sibling
fixes) that deletes the data PVCs by name (one per replica) and the five TLS
secrets by name. check-release-name pins the release to "etcd", so every
target name is deterministic and no sibling workload shares the data-etcd-*
prefix. The Job runs non-root with a read-only root fs, dropped capabilities
and seccompProfile RuntimeDefault; its RBAC is scoped to PVC get/list/delete
and the five secrets by resourceNames. Deletes use --ignore-not-found and
failures are logged but never block teardown.

The stuck-HelmRelease finalizer noted in the issue is the platform-wide
sharded-helm-controller side-bug tracked separately; this change addresses the
PVC and secret orphans the etcd chart owns.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
 2026-07-02 15:58:08 +03:00
Алексей Артамонов c7e8f4d975 fix(mariadb): re-guard replication so single-replica passes the operator webhook 
PR #2279 made `replication.enabled: true` unconditional so the operator would
always create -primary/-secondary services. But the mariadb-operator validating
webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1,
so single-replica MariaDB could no longer be created — the HelmRelease never
became Ready.

Re-guard replication on replicas>1 and teach the service-name consumers about
the bare <release> service the operator provisions in both topologies:

- mariadb.yaml: gate `replication` on replicas>1; restore single-replica
  external access via `service.type: LoadBalancer` and keep `primaryService`
  LoadBalancer for replicas>1 only.
- backup-cronjob.yaml: route single-replica dumps to the bare service
  (no -secondary exists without replication).
- dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus
  -primary/-secondary when replicas>1, matching the ApplicationDefinition.
- mariadb-rd: list the bare service in services.include so single-replica
  installs are visible in the dashboard.

Adds a single-replica e2e test that reproduces the webhook rejection and
guards against regression.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
 2026-07-02 15:48:29 +03:00