mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 18:47:04 +09:00
ci/cozystack-api-pin-securitygroup
2887
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Andrei KvapilandClaude
|
21ee665585 |
fix(cozystack-api): pin the image to the build that ships the SecurityGroup API
#2922 added the sdn.cozystack.io SecurityGroup API to cozystack-api, but main's committed image is still the v1.5.0 release, which predates it — the new build was only pinned transiently inside #2922's own E2E and never committed. As a result every PR that does not itself rebuild cozystack-api installs the old apiserver, so `kubectl get securitygroup` returns "the server doesn't have a resource type" and the securitygroup / serviceexposure e2e tests fail on unrelated PRs. Pin cozystack-api to the current main-HEAD build that carries the API (image mirrored to ghcr from the #2922 build). Stopgap until a release re-pins it through the normal build. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
a31a73cb87 |
fix(tenant): gate VM-resource HelmReleases on victoria-metrics-operator readiness (#3141)
## What this PR does The tenant `etcd`, `ingress` and `monitoring` HelmReleases create VictoriaMetrics custom resources — `VMPodScrape` directly in `etcd` and `ingress`, and `VMCluster` / `VMAlert` / `VLCluster` / `VMAlertmanager` / `VMServiceScrape` via the `monitoring-system` child that `monitoring` creates. All of these are intercepted by the `victoria-metrics-operator` validating webhook, which is configured with `failurePolicy: Fail`. On a cold install the operator pod can be briefly unavailable on `:9443` (for example a transient API-server blip during startup leaves the webhook backend not yet serving). In that window the tenant releases reconcile, the webhook rejects their VM* resources with `connection refused`, and the releases churn through failed-install → rollback → retry. They carry `install.remediation.retries: -1` and recover eventually, but the churn can outlast a cold-install readiness deadline and surfaces as spurious install failures. This adds a Flux `spec.dependsOn` from each of those three tenant releases to the `victoria-metrics-operator` HelmRelease in `cozy-victoria-metrics-operator`, so they wait for the operator to be Ready before creating any VM* resource. The system monitoring components already gate on the operator the same way (a variant-level `dependsOn` in the platform package sources); this brings the tenant releases to parity. `failurePolicy: Fail` is preserved — validation is unchanged. The tenant releases are sharded (`sharding.fluxcd.io/key`) while the operator release is not. helm-controller resolves `dependsOn` through an uncached API read that bypasses the shard's watch-label-selector cache (since v1.1.0, PR fluxcd/helm-controller#1070), so the cross-shard reference is visible to the tenant shard's controller; this repo ships helm-controller v1.5.0. A helm-unittest suite pins the gate on all three releases and asserts that the tenant releases creating no VM* resources (`seaweedfs`, `gateway`) do not carry it, documenting the intentional narrowness of the gate. ### Release note ```release-note fix(tenant): gate tenant etcd, ingress and monitoring HelmReleases on victoria-metrics-operator readiness so they no longer fail to install when the operator's admission webhook is briefly unavailable on a cold install ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved tenant installation reliability by adding readiness gating: etcd, ingress, and monitoring Helm releases now wait for the Victoria Metrics operator to become Ready before applying dependent resources. * Prevents install/rollback failures caused by temporary admission webhook unavailability during cold starts. * **Tests** * Added Helm-unittest coverage to confirm the dependency gate is present only for templates that create VM* resources (and omitted for templates that don’t). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
03435dcc50 |
fix(flux): tune source-watcher concurrency + CPU to unblock platform install under load (#3163)
## What this PR does `source-watcher` is the Flux controller that turns each `PackageSource` into an `ExternalArtifact` (one `ArtifactGenerator` per source, ~100 builds on the management cluster during install). The `cozystack-platform` HelmRelease blocks until all of those artifacts are Ready, so source-watcher's throughput sits on the install's critical path. It was the only Flux controller left without concurrency/CPU tuning. In the bootstrap all-in-one pod it ran on the binary default (`--concurrent=10`) with the shared 100m CPU request; in the production `FluxInstance` it was absent from the reconciler patch that source-, kustomize- and helm-controller already receive. Under load its per-artifact builds finish at staggered times, leaving a random tail of not-yet-Ready artifacts that times out the platform install. A failed run's diagnostics show the shape of this: the `cozystack-platform` HelmRelease timed out with `timeout waiting for: PackageSource/… status: 'InProgress'` (a run-to-run varying tail — e.g. `bucket-application`, `linstor-gui`, `capi-provider-core`), while its ~200 ExternalArtifacts reached Ready in a staggered trickle spread over ~60 seconds rather than together. That staggered completion of independent per-artifact builds is the signature of a throughput-limited builder, not a single all-or-nothing pull. This raises source-watcher to `--concurrent=20` in both install paths: - **Bootstrap all-in-one pod** (`internal/fluxinstall/manifests/fluxcd.yaml`, regenerated from `flux-aio.cue` + the `flux-aio` Makefile): `--concurrent=20` plus a 500m CPU request. The AIO pod sets no CPU limits, so the request is what drives source-watcher's proportional CPU share when the node is saturated. - **Production `FluxInstance`** (`packages/system/fluxcd/values.yaml`): source-watcher is added to the shared reconciler patch, so it now gets the same `--concurrent=20`, `--requeue-dependency=5s` and 2000m/2048Mi limits as the other tuned controllers. The `flux-aio` module exposes a per-controller resources override for the watcher (set via CUE) but no args override, so the bootstrap `--concurrent` bump is a guarded post-render patch in the Makefile; the embedded manifest is the product of that pipeline. source-watcher's support for both `--concurrent` and `--requeue-dependency` was confirmed against its `cmd/main.go`, and the production `op: replace` on `resources/limits` is safe because the distribution ships source-watcher with a `resources.limits` block (same shape as source-controller). Optional follow-up, intentionally left out to keep this focused on the root cause: if the artifact-build tail still brushes the 10-minute platform HelmRelease timeout under load, the installer's HelmRelease install/upgrade timeout values give a safety margin. ### Release note ```release-note fix(flux): tune source-watcher concurrency and CPU so the platform install does not time out under load ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Increased the source watcher’s concurrency to improve throughput and better handle larger workloads. * Added higher CPU and memory settings for the watcher to support more stable operation under load. * **Bug Fixes** * Updated deployment patching so the watcher receives the same configuration adjustments as other controllers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6db5ae9c6f |
fix(tenant): gate VM-CR-creating HelmReleases on victoria-metrics-operator
The tenant etcd, ingress and monitoring HelmReleases create VM* custom resources (VMPodScrape directly in etcd/ingress; VMCluster, VMAlert, VLCluster, VMAlertmanager and VMServiceScrape via the monitoring-system child of monitoring) guarded by the victoria-metrics-operator validating webhook with failurePolicy: Fail. On a cold install the operator pod can be briefly unavailable on :9443, so these releases reconcile before the webhook backend is serving and are rejected with connection refused. They carry install.remediation.retries: -1 and would eventually recover, but the failed-install -> rollback -> retry churn can outlast a cold-install readiness deadline and surfaces as spurious install failures. Add a Flux spec.dependsOn from each of these releases to the victoria-metrics-operator HelmRelease in cozy-victoria-metrics-operator, so they wait for the operator to be Ready before creating VM* resources. The system monitoring components gate on the operator the same way, via a variant-level dependsOn. The cross-shard reference (tenant releases are sharded, the operator release is not) resolves via helm-controller's uncached dependency check (APIReader, v1.1.0+). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
41ff94eca9 |
fix(flux): tune source-watcher concurrency and CPU under load
source-watcher reconciles one ArtifactGenerator per PackageSource and builds the resulting ExternalArtifacts. On the management cluster that is ~100 artifact builds during install, and the platform HelmRelease blocks until all of them are Ready. It was the only Flux controller left untuned. In the bootstrap all-in-one pod it ran without an explicit --concurrent (binary default 10) and with the shared 100m CPU request; in the production FluxInstance it was missing from the --concurrent / CPU patch that source-, kustomize- and helm-controller already receive. Under load its per-artifact builds finish at staggered times, leaving a random tail of not-yet-Ready artifacts that times out the platform install. Raise source-watcher to --concurrent=20 in both deployments. In the bootstrap all-in-one pod also give it a 500m CPU request; the pod has no CPU limits, so the request is what drives its CPU share under contention. In the production FluxInstance add it to the shared reconciler patch, so it now receives the same --requeue-dependency=5s and 2000m/2048Mi limits the other tuned controllers already have. The flux-aio module exposes a resources override for the watcher (set via CUE) but no args override, so the bootstrap --concurrent bump is a post-render patch in the Makefile. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f7271b8011 |
fix(velero): add startupProbe so slow startup does not crashloop the install gate
Velero binds its metrics/health endpoint (:8085, http-monitoring) only after the server loads its plugins and connects to the API server. Under a heavy parallel platform install that can take minutes, but the upstream liveness probe begins checking /metrics at initialDelaySeconds=10 and kills the container after failureThreshold=5 (~130s total) before :8085 is bound. The kubelet SIGTERMs velero (exitCode 0/Completed), it BackOff-restarts, and never escapes the loop inside the install-gate window. Because backupstrategy-controller hard-depends on velero, this blocks the gate. Add a startupProbe to the velero server container. While it is failing the kubelet holds off liveness and readiness entirely, granting failureThreshold * periodSeconds = 30 * 10s = 300s of startup grace, then hands over to the unchanged, tight upstream liveness (10s / 30s / 5) for steady-state crash detection. This is strictly better than loosening liveness, which would also blunt steady-state detection. The upstream velero chart (12.0.3) does not render a startupProbe, so the package patches the deployment template via patches/add-startup-probe.patch (re-applied after re-vendor by the Makefile update target, matching the seaweedfs/ingress-nginx convention) and supplies the startupProbe through the package values override. Pin the startupProbe budget and the unchanged liveness with a helm-unittest regression guard. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9c18260f95 |
fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets) (#2957)
## What The e2e install pipeline still flakes on a loaded runner, failing at a different point each run. Root-caused via repeated runs and a live SSH breakpoint; the platform itself is fine (the components do converge), the failures are slowness + self-inflicted churn under resource pressure. ### Fixes 1. **VPA updater OOM** — the `vertical-pod-autoscaler-updater` memory limit was `110Mi`. The updater watches every pod in the cluster, so on a full install it OOM-kills → `ProgressDeadlineExceeded` → Flux fails the HelmRelease "early due to stalled resources", cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raised to `512Mi` (request `200Mi`). 2. **VPA-for-VPA eviction churn** — the two VPA-for-VPA `VerticalPodAutoscaler` objects ran `updateMode: Auto`, evicting the VPA recommender(s) during install and adding pod churn on an already busy cluster. Switched to `Initial`, matching the existing monitoring/etcd `VPA->Initial` fixes. 3. **Install wait budgets** — components reach readiness but overshoot tight waits on a busy runner: keycloak HR chain `10m→20m`, tenant aggregate wait `10m→20m`, per-app tenant readiness waits `5m→10m`. (The crust-gather failure-snapshot timeout that kept whole jobs hanging to the 3h cancel is already on `main`.) ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased several installation and readiness wait times to better handle slower environment startup and reduce premature failures. * Updated vertical autoscaling behavior to avoid unnecessary pod restarts during deployment. * Raised resource limits for the autoscaling updater to improve reliability under load. * **Tests** * Aligned automated checks with the new autoscaling behavior and longer readiness expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei KvapilandClaude
|
d6f66d2dcd |
fix(vertical-pod-autoscaler): stop the updater OOM + VPA-for-VPA eviction churn
Two root causes of e2e install flakiness, both in the VPA package: 1. The updater memory limit was 110Mi. The updater watches every pod in the cluster, so on a full cozystack install it OOM-kills, which trips the Deployment's ProgressDeadlineExceeded and makes Flux fail the HelmRelease 'early due to stalled resources' - cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raise the limit to 512Mi (request 200Mi); the recommender is already unlimited. 2. The two VPA-for-VPA VerticalPodAutoscaler objects ran updateMode: Auto, which evicts the VPA recommender(s) during install and adds pod churn on an already busy cluster. Switch to Initial, matching the existing monitoring/etcd VPA->Initial fixes. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
cbc314a8c5 |
fix(kubevirt-cdi): order behind cozystack-basics so cdi-clone-dv RoleBinding finds cozy-public
The kubevirt-cdi default variant only depended on cozystack.networking, which reconciles long before cozystack-basics provisions the cozy-public namespace. kubevirt-cdi ships the cdi-clone-dv RoleBinding into cozy-public, so on a fresh install the RoleBinding can race namespace creation and fail with `namespaces "cozy-public" not found`. Flux retries normally mask it, but under slow reconcile it surfaces as a hard UpgradeFailed. Add a cozystack.cozystack-basics edge so the namespace exists first, mirroring the backupstrategy-controller ordering. No cycle: cozystack-basics' own dependencies (tenant-application, cozystack-engine, gateway-api-crds) do not reach kubevirt-cdi. Pin the edge with a helm-unittest regression guard. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei Kvapil
|
c725a41f3a |
feat(tenant): recursive (hierarchical) resource quotas for sub-tenants (#3086)
## What this PR does Adds **hierarchical (recursive) resource quotas** for tenants so a tenant can no longer escalate its allocation by creating sub-tenants. Today `tenant.spec.resourceQuotas` becomes a plain per-namespace `ResourceQuota`, which ignores the tenant tree: a super-admin inside a quota'd tenant can create a sub-tenant with a larger quota — or no quota at all — and consume more than was allocated. A tenant's declared quota is now treated as the budget for its **whole sub-tree**: - A child that declares its own quota **carves a fixed slice** out of the parent's remaining budget (parent 10 CPU → child 4 CPU → parent has 6 left). - A child **without** a quota **shares** the parent's remaining pool rather than being unbounded. - A child quota that exceeds the parent's remaining budget is **rejected**. Two cooperating layers, mirroring OpenShift's ClusterResourceQuota split (admission + reconciliation): 1. **Declaration-time gate** in the aggregated apiserver (`apps.cozystack.io` Tenant create/update): deterministically rejects a sub-tenant whose declared quota exceeds the parent's remaining budget. 2. **Runtime enforcer** in `cozystack-controller`: maintains a per-namespace `tenant-quota-allocated` ResourceQuota that clamps each pool member to its share of the parent budget, aggregating sub-tree usage so members collectively stay within the budget. Kubernetes enforces the most restrictive ResourceQuota in a namespace, so this binds without fighting Flux over the chart-rendered `tenant-quota`. Overcommitted pools (a parent quota lowered below the sum of its children) emit a `QuotaOvercommitted` event. A `--tenant-quota-buffer-percent` flag temporarily inflates pool budgets during rollout so workloads already over a freshly-introduced quota keep running. The model and the selector-spans-namespaces / aggregated-usage approach are adapted from OpenShift's ClusterResourceQuota controllers (`openshift/api`, `library-go`, `cluster-policy-controller`, `apiserver-library-go`), Apache-2.0, Copyright Red Hat, Inc. and the OpenShift Authors; attribution is preserved in the source. The per-namespace quota arithmetic reuses the upstream Kubernetes quota helpers (`k8s.io/apiserver/pkg/quota/v1`). No API types or the tenant chart change; the feature works with the existing `tenant.spec.resourceQuotas` field. ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(tenant): tenant resource quotas are now hierarchical — a sub-tenant's quota is carved out of its parent's remaining budget and can no longer exceed it; sub-tenants without a quota share the parent's pool. Set `--tenant-quota-buffer-percent` on cozystack-controller for a temporary grace buffer during upgrades. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added hierarchical tenant quota enforcement across tenant namespaces, including shared pool handling and configurable quota buffer support. * Tenant quota changes are now validated on create and update to prevent over-allocation. * **Bug Fixes** * Improved quota calculations so usage, carve-outs, and nested tenant limits are applied more consistently. * Stale quota allocations are now cleaned up when namespaces no longer belong to an active tenant pool. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
5b26ec3462 |
fix(mariadb): clean up orphaned operator-generated secrets (#3056) (#3071)
## What this PR does After deleting a `MariaDB` app CR with metrics and/or replication enabled, two Secrets created by the mariadb-operator were left orphaned in the tenant namespace: - `<release>-metrics-password` - `<release>-repl-password` These are generated by the operator (`k8s.mariadb.com/v1alpha1`) as a side effect of `metrics.enabled` / `replication.enabled`, so they are **not** Helm-owned and survive `helm uninstall`. The existing post-delete hook (`templates/hooks/cleanup-pvc.yaml`) only swept PVCs, leaving the Secrets behind. This PR extends that hook to also clean them up: - grants the cleanup `Role` RBAC on `secrets`; - deletes both password Secrets by name with `--ignore-not-found`; - adds a `helm-unittest` suite locking in the behaviour, plus a `test` Makefile target so the package's tests actually run in CI (`hack/helm-unit-tests.sh` only runs packages that define a `test` target). Fixes #3056. ### Release note ```release-note fix(mariadb): garbage-collect operator-generated metrics-password / repl-password Secrets when a MariaDB app is deleted ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved MariaDB uninstall cleanup to remove leftover operator-generated password Secrets in addition to persistent volumes/claims. * Hardened the cleanup hook Job with stricter pod/container security settings, least-privilege Secret permissions, idempotent deletions for missing resources, and correct non-zero exit on real failures. * **Tests** * Added a cleanup hook test suite validating Secret deletion behavior, failure propagation, and that cleanup RBAC grants least-privilege access only to the required Secrets. * Added a Helm unit test `test` target for the MariaDB chart. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Алексей Артамонов
|
d988b3e8c8 |
fix(apiserver): read tenant pool quota usage via direct client and grant RBAC
parentPoolUsage listed ResourceQuotas through the aggregated apiserver's cached client, which lazily starts a cluster-wide ResourceQuota informer. cozystack-api has no RBAC to list resourcequotas, so the informer never synced and tenant create/update of a bounded child hung until the request timed out (observed on a cluster). Read quota usage through the direct (uncached) watch client instead, scoped to the pool's member namespaces, so no cluster-wide informer is started, and grant cozystack-api get/watch/list on resourcequotas for those reads. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Andrei KvapilandClaude
|
844deb170f |
feat(controller): enforce recursive tenant quotas at runtime
Add a tenant quota reconciler to cozystack-controller that realises the hierarchical (sub-tree) quota model at runtime. Each tenant's declared quota is the budget for its whole sub-tree: a child that declares its own quota carves a fixed slice out of the parent budget, while a child without a quota shares the parent's remaining pool. The controller maintains a per-namespace 'tenant-quota-allocated' ResourceQuota (Kubernetes enforces the most restrictive quota, so it binds without fighting Flux over the chart-rendered 'tenant-quota'), aggregating sub-tree usage so a pool's members collectively stay within the budget. A --tenant-quota-buffer-percent flag temporarily inflates pool budgets so workloads already over a freshly-introduced quota keep running during rollout; overcommitted pools emit a QuotaOvercommitted event. The model and the selector-spans-namespaces/aggregated-usage approach are adapted from OpenShift's ClusterResourceQuota controllers (openshift/api, library-go, cluster-policy-controller, apiserver-library-go), Apache-2.0, Copyright Red Hat, Inc. and the OpenShift Authors. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
eab9b034f6 |
fix(seaweedfs): enforce read-only bucket access by bumping cosi-driver to v0.3.1 (#3136)
## What this PR does The SeaweedFS COSI driver shipped at `v0.1.2` hardcodes read-write S3 actions (`Read,Write,List,Tagging`) for every BucketAccess and never reads the `accessPolicy` BucketAccessClass parameter. The chart already defines a `-readonly` BucketAccessClass carrying `parameters.accessPolicy: readonly`, and the `bucket` app routes `readonly: true` users to it — but `v0.1.2` ignores the parameter, so a credential meant to be read-only is silently issued with full read-write access and can upload and delete objects. This is an access-control bug. Upstream `seaweedfs/seaweedfs-cosi-driver` added `accessPolicy` handling in `v0.3.0`: `readonly` maps to `Read,List`, while `readwrite` and the empty default stay `Read,Write,List,Tagging`. This bumps the driver to `v0.3.1` via a `seaweedfs.cosi.image` override in the package `values.yaml`, so the override survives `make update` and the vendored chart is left untouched. The default read-write action set is unchanged, so existing read-write credentials are unaffected; only read-only grants are tightened to actually be read-only. The `extra/seaweedfs` chart already runs `v0.3.1` with the same objectstorage-sidecar, so the driver/sidecar combination is already proven in this repo. A chart unit test pins the rendered driver image so this fix cannot silently regress on a future re-vendor. This does not close #1921. That issue bundles two features — read-only bucket access (fixed here) and Object Lock / WORM (the `-lock` BucketClass), which is independent and out of scope here. Relates to #1921. ### Release note ```release-note Fix SeaweedFS read-only bucket credentials being issued with read-write access; the COSI driver now honours the read-only access policy. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Pin the SeaweedFS COSI driver image to `ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1` to keep access behavior consistent and avoid relying on a vendored default. * Ensure the rendered SeaweedFS COSI deployment uses the expected container name and the pinned driver image. * **Tests** * Add a YAML test suite to validate the generated COSI deployment template matches the expected container name and pinned image. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
bef42229d4 |
fix(harbor): clean up orphaned jobservice PVC and ACME solver resources on delete (#3061) (#3077)
## What this PR does After deleting a `Harbor` app CR, two things were left orphaned in the tenant namespace: - the **jobservice / trivy PVCs** (`<release>-jobservice`, and `<release>-trivy` when Trivy is enabled), because `packages/apps/harbor/templates/harbor.yaml` sets `persistence.resourcePolicy: "keep"`, which annotates the upstream Harbor subchart's PVCs with `helm.sh/resource-policy: keep` so Helm/Flux deliberately never deletes them; - cert-manager's **ACME http01-solver** resources (`cm-acme-http-solver-*`), via the same stuck-challenge mechanism as the Bucket issue (#3060) when Harbor issues a non-wildcard cert. This adds a single `post-delete` cleanup hook that: - **deletes the upstream Harbor subchart's kept PVCs** — selecting by `app.kubernetes.io/instance=<release>-system` (the subchart runs as the `<release>-system` HelmRelease, so its PVCs carry that label, **not** `<release>`). A hook is required precisely because `resourcePolicy: keep` blocks normal reclaim. This intentionally covers both the jobservice and trivy PVCs: both hold only regenerable data (jobservice file logs; trivy's vulnerability-DB cache), and registry images live in S3. - **deletes this release's `Certificate`** (deterministic name `<release>-ingress-tls`); cert-manager then cascade-deletes the `Order`, `Challenge` and solver `Pod`/`Service`/`Ingress` objects it owns via owner references. - **host-scoped belt-and-braces sweep** for any ACME http01 solver objects cert-manager may have orphaned (e.g. the Certificate was already gone, so deleting it did not cascade). The sweep is scoped **strictly to this release's Harbor host** — Challenges filtered by `.spec.dnsName`, solver Ingresses by `.spec.rules[0].host` — **not** a namespace-wide `acme.cert-manager.io/http01-solver=true` label sweep, because a Cozystack tenant namespace can host many apps and a blanket label sweep would delete sibling apps' in-flight challenge solvers. The Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`, with CPU/memory requests and limits set. RBAC is scoped to exactly the resources it deletes (`persistentvolumeclaims`, `certificates`, `orders`, `challenges`, solver `ingresses`) and verified against the vendored cert-manager CRDs. Failures are logged but never block teardown. ## Cleanup scope — what is and isn't removed **Removed:** the upstream Harbor subchart's kept PVCs (jobservice + trivy) and this release's cert-manager ACME artifacts (Certificate, and host-scoped Order/Challenge/solver-Ingress). **Not touched (by design):** - **CNPG database PVCs** (`<release>-db-N`) — owned by the CNPG `Cluster` and governed by CNPG's own reclaim policy; they carry CNPG labels, not `app.kubernetes.io/instance=<release>-system`. Re-installing Harbor with a different DB size into the same namespace can therefore leak the old DB PVCs; this is left to CNPG's lifecycle on purpose (the data is the user's). - **Redis PVCs** — managed by the redis-operator with their own labels, likewise untouched. - **Solver `Pod`/`Service`** — not deleted directly; they are removed by cert-manager's owner-ref cascade when the Certificate/Challenge is deleted, so the hook grants no RBAC for them. > [!NOTE] > The PVC sweep is eventually-consistent: while the inner `<release>-system` > HelmRelease is still being uninstalled by Flux, a PVC may briefly stay in > `Terminating` until its consumer Pods (jobservice/trivy) exit. The Job's > deletion request always succeeds; reclaim completes once the pods are gone. > [!WARNING] > **This overrides the intentional `persistence.resourcePolicy: "keep"` setting > and force-deletes the jobservice/trivy PVCs on app deletion.** Those PVCs hold > only jobservice file logs and the trivy vuln-DB cache (registry images live in > S3), so data-loss risk is low — but if retaining them across re-installs is > desired, this is a conscious product decision to revisit. The ACME cleanup > affects only cert-manager's transient artifacts. Fixes #3061. > Follow-up tracked in #3088: digest-pin the cleanup `kubectl` image and route it > through `cozy-lib.images-registry` (pre-existing debt shared with the MariaDB > cleanup hook), out of scope for this PR. ### Release note ```release-note fix(harbor): clean up the orphaned jobservice/trivy PVCs and cert-manager ACME http01-solver resources left behind after a Harbor app is deleted. NOTE: this force-deletes the jobservice/trivy PVCs, overriding persistence.resourcePolicy=keep (jobservice file logs and trivy vuln-DB cache only; registry images are in S3). CNPG database PVCs are not touched. ``` |
||
|
myasnikovdaniil
|
3e20a0cd41 |
fix(kubernetes): make node group resources optional when instanceType is set (#3121)
## What this PR does
A tenant Kubernetes node group can be sized two ways: by `instanceType`
or by an explicit `resources` block. The generated values schema,
however, marked `resources` as required on every node group, so a node
group sized purely by `instanceType` was rejected at install time:
```text
at '/nodeGroups/<name>': missing property 'resources'
```
The same failure occurs through the typed `Kubernetes` CR: an empty
`resources: {}` is dropped during the CR-to-HelmRelease translation, so
by validation time the key is absent and the required check fails — even
though sizing was already fully specified by `instanceType`.
This marks `resources` optional in the node group definition and
regenerates the dependent artifacts (`values.schema.json`, API types,
README, and the `ApplicationDefinition`). No change is needed to emit
the VM: the chart already renders `domain.cpu`/`domain.memory` only when
`resources` is present and `instancetype` only when set, so a node group
sized purely by `instanceType` now validates and installs. Node groups
that specify `resources` are unaffected.
`instanceType` and `resources` are alternative, mutually exclusive
sizing sources — KubeVirt rejects a VirtualMachine that references an
instancetype and also sets explicit `domain.cpu`/`domain.memory`, since
instancetype CPU/memory cannot be overridden. To avoid emitting a VM
that the instancetype admission webhook would later reject, the chart
now fails fast at render time with a clear message when a node group
sets both a non-empty `instanceType` and explicit `resources`.
Tests: helm-unittest coverage proves a node group with `instanceType`
and no `resources` passes schema validation and renders without explicit
`domain.cpu`/`memory`; a complementary case confirms explicit
`resources` are still honoured; and a `failedTemplate` case pins the
render-time rejection of the conflicting both-set combination. The
omit-resources case fails against the pre-fix schema with the exact
error above, so it is a real regression guard.
Addresses #3046.
### Screenshots
N/A — schema and chart change, no UI impact.
### Release note
```release-note
fix(kubernetes): allow a node group to omit `resources` when it is sized by `instanceType` (previously failed schema validation with "missing property 'resources'"); reject the unsupported combination of `instanceType` and explicit `resources` at render time
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Worker node groups can optionally size CPU/memory via `resources`, and
can omit `resources` when using `instanceType`.
* **Bug Fixes**
* Render now validates CPU/memory rules: `instanceType` with only one of
`resources.cpu`/`resources.memory` fails, and `instancetype` is omitted
when both `resources.cpu` and `resources.memory` are set (with
`instanceType` ignored).
* **Documentation**
* Updated API/Helm docs and JSON schema to clarify `resources` vs
`instanceType` precedence and the “set both CPU and memory (or neither)”
requirement.
* **Tests**
* Added helm-unittest scenarios covering omission, precedence, and
render-time failure cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
0892c5a38c |
fix(qdrant)!: clean up orphaned data PVC on delete (#3059) (#3075)
## What this PR does After deleting a `Qdrant` app CR, the StatefulSet data PVC (`qdrant-storage-<release>-0`) was left orphaned in the tenant namespace, consuming storage permanently. StatefulSet-templated PVCs are never garbage-collected on StatefulSet deletion by default. The vendored upstream Qdrant chart does **not** support `persistentVolumeClaimRetentionPolicy` (verified: its `statefulset.yaml` never renders that field), so the OpenBAO-style value passthrough is not possible here. Instead this adds a `post-delete` PVC cleanup hook (modeled on the MariaDB hook) that deletes the release's PVCs by label. The chart's volumeClaimTemplate only stamps `app: qdrant` (shared by every Qdrant in a namespace), so to select PVCs safely per release this also passes `persistence.additionalLabels.app.kubernetes.io/instance: <release>` through the HelmRelease — giving the data PVC a release-unique label the hook selects on. The cleanup Job is hardened: runs as non-root (uid 65534), read-only root fs, all capabilities dropped, `seccompProfile: RuntimeDefault`; RBAC scoped to `persistentvolumeclaims`. A real cleanup failure is logged (not silently swallowed) but never blocks namespace teardown. > [!WARNING] > **Deleting a Qdrant app now permanently removes its data PVC and the vector data it holds.** > This is intentional and fixes the storage leak; the hook only fires on full > release deletion (not on scale-down). Back up before deleting. Fixes #3059. ### Release note ```release-note fix(qdrant): garbage-collect the StatefulSet data PVC when a Qdrant app is deleted, instead of leaving it orphaned. WARNING: deleting a Qdrant app now permanently removes its data PVC and the data it holds. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a post-delete cleanup hook for Qdrant to remove leftover StatefulSet data PVCs after deletion. * Included hardened Kubernetes settings for the cleanup job to improve runtime safety. * **Tests** * Added Helm chart tests that validate the post-delete cleanup job and its RBAC permissions. * Enhanced end-to-end verification to confirm the data PVC is fully removed after deleting a Qdrant instance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
bf14a73b39 |
feat(api): add SecurityGroup network policy resource (sdn.cozystack.io) (#2922)
## What this PR does
Introduces a tenant-facing `SecurityGroup` resource in a new
`sdn.cozystack.io/v1alpha1` API group, served by the Cozystack
aggregated API server as a 1:1 projection of a `CiliumNetworkPolicy`. A
SecurityGroup attaches to one of the tenant's managed applications by
reference (`spec.targetRef: {apiGroup, kind, name}`) and declares
allow-list ingress/egress for that application. Tenants manage network
policy for their own apps entirely through
`securitygroups.sdn.cozystack.io`, without being granted any access to
the `cilium.io` API group.
The translation is synchronous and stateless — the API server converts
each SecurityGroup into a CiliumNetworkPolicy in the same namespace on
write and back on read; there is no controller. The backing policy's
`endpointSelector` is **derived** from `targetRef` (matched on the
application's lineage labels
`apps.cozystack.io/application.{group,kind,name}`), never copied from
tenant input, so the selector is machine-generated and can only ever
match the referenced application's own pods in the tenant's namespace.
Policies owned by this API carry an `sdn.cozystack.io/securitygroup`
marker label that the storage always re-asserts and hides from the view,
so it only ever surfaces or mutates its own policies and never touches
platform-managed network policies.
The value is self-service network policy on a Cozystack-native,
RBAC-scoped surface, without exposing Cilium's version-coupled CRDs to
tenants. Because the selector is derived from an authorized application
reference rather than free-form tenant input, the RBAC boundary is
structural, not selector-validated: a tenant cannot express network
policy against arbitrary or platform-owned pods. The full design —
selector derivation from lineage labels, the structural boundary, the
additive-policy reality (a SecurityGroup widens a managed application's
allowed traffic but cannot sever its platform-managed management plane,
and equally cannot tighten below the platform baseline), the in-tree
CiliumNetworkPolicy mirror that avoids a Cilium Go-module dependency,
RBAC, and deferred extensions (raw-pod targeting, cluster scope,
reusable groups) — is in `pkg/apis/sdn/DESIGN.md`.
Implements the attachment model from #1614.
### Release note
```release-note
feat(api): add the SecurityGroup resource (sdn.cozystack.io/v1alpha1) — a tenant-facing, namespace-scoped firewall that attaches to a managed application by reference and projects 1:1 onto a CiliumNetworkPolicy with a machine-derived endpoint selector, letting tenants manage their application's network policy without cilium.io access.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added the `SecurityGroup` API resource (`sdn.cozystack.io` v1alpha1)
with a namespaced projection backed by Cilium network policies.
* Enabled the SDN `APIService` and wired SDN routing, scheme, and REST
support.
* Expanded RBAC to include SecurityGroup access and required Cilium
policy permissions.
* **Documentation**
* Added a design document describing projection behavior and rule
semantics.
* **Bug Fixes**
* Improved API rule validation coverage for missing list-type fields.
* **Tests**
* Added SDN round-trip, fuzzing, RBAC, and comprehensive REST
create/get/list/update/delete/watch/resource-version tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Алексей АртамоновandClaude Opus 4.8
|
b25f294d46 |
fix(harbor): select kept PVCs by release label so the trivy PVC is also cleaned
The post-delete cleanup hook selected PVCs by app.kubernetes.io/instance=<release>-system, a label only the jobservice PVC carries (via the chart's "harbor.labels" helper). The trivy PVC is a StatefulSet volumeClaimTemplate labeled via "harbor.legacy.labels" with a selector using "harbor.matchLabels" — neither sets app.kubernetes.io/instance — so the sweep silently missed it. Trivy is enabled by default, so a default Harbor delete left the trivy PVC orphaned, contradicting the PR's own claim. Select on release=<release>-system instead: both kept PVCs carry it, while CNPG (db) and Redis PVCs use different label schemes and the registry is on S3, so only the two intended PVCs match. The unit test now asserts the corrected selector and guards against regressing to the instance label. Also drop the by-name delete of Orders/Challenges (their hash-suffixed names make a by-name delete a no-op; the Certificate owner-ref cascade plus the host-scoped sweep handle them), narrow the hook RBAC to challenges only, and note the Gateway-mode no-op. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
11c417e258 |
test(mariadb): assert both cleanup failure branches and exact RBAC rule set
Strengthen the cleanup hook suite per review: - assert the PVC and Secret delete branches each set cleanup_failed=1, not just the final exit, so a regression in either branch is caught; - pin the Role to exactly two rules so a broader unscoped secrets rule can't slip past the subset `contains` check. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
ae7c232409 |
fix(qdrant): anchor PVC ordinal so cleanup never deletes a sibling's data
The post-delete hook selected PVCs with an unanchored name prefix (qdrant-storage-<release>-). Because release names are qdrant-<app>, deleting release "qdrant-test" matched not only its own PVC (qdrant-storage-qdrant-test-0) but also a sibling "qdrant-test-2"'s PVC (qdrant-storage-qdrant-test-2-0) in the same namespace — irreversibly deleting the sibling's data (or wedging it in Terminating via pvc-protection). Anchor the ordinal with grep -E "...-[0-9]+$" so only the release's own data PVCs match. A release name that is a prefix of a sibling no longer collides. - cleanup-pvc.yaml: grep -E with anchored "-[0-9]+$". - tests: assert the anchored selector and add a notMatchRegex guard against regressing to the unanchored prefix. - e2e: gate the post-delete wait on a NotFound result specifically, so a transient API/auth error is not misread as "PVC reclaimed". Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
c2a8a3b41f |
fix(mariadb): fail cleanup hook on real delete errors
The post-delete cleanup hook used `kubectl delete ... --ignore-not-found || echo WARNING`, which swallowed genuine RBAC/API failures and let the Job succeed with orphaned password Secrets still present. Track delete failures and propagate them via a non-zero exit, while --ignore-not-found still tolerates already-absent resources. Add a helm-unittest assertion locking in the non-zero exit-on-failure behaviour. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Aleksei Sviridkin
|
a349c753ad |
fix(kubernetes): make node group resources optional and prefer them over instanceType
A node group is sized either by instanceType or by explicit resources, but the generated values schema required resources on every node group. A node group with instanceType and no resources — or an empty resources dropped during typed-CR to HelmRelease translation — failed schema validation with "missing property 'resources'" even though sizing was fully specified. Mark resources optional in the node group definition and regenerate the schema, API types, README, and ApplicationDefinition. The chart template emits domain.cpu/memory only when resources is present, so a node group sized purely by instanceType now passes validation and installs. When a node group sets both instanceType and explicit resources, explicit resources take precedence: the instancetype is omitted from the rendered VM and the per-group instancetype lookup is skipped, so KubeVirt accepts the explicit domain.cpu/memory with no instancetype to conflict with. instanceType stays required and defaulted (no schema change), so the console's required instanceType dropdown keeps working — sending the default instanceType plus explicit resources simply sizes the VM by resources. This replaces an earlier render-time hard failure on the both-set case, which was unreachable to clear from the dashboard (the required dropdown cannot be emptied). Explicit resources are all-or-nothing alongside an instanceType: a partial spec (only cpu, or only memory) on a group that carries a non-empty instanceType would emit one domain override next to the instancetype — which KubeVirt rejects — so it now fails at render time with a clear error. Partial resources on a group without an instanceType (used to tune kubelet reservations) are unaffected. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
0c6426feaa |
fix(seaweedfs): enforce read-only bucket access by bumping cosi-driver to v0.3.1
The seaweedfs COSI driver shipped at v0.1.2 hardcodes read-write S3 actions (Read, Write, List, Tagging) for every BucketAccess and never reads the accessPolicy parameter. The chart already defines a -readonly BucketAccessClass carrying parameters.accessPolicy: readonly, but v0.1.2 ignores it, so a credential meant to be read-only can still write (upload and delete) objects. Bump the driver to v0.3.1, which honours accessPolicy: readonly by issuing Read,List-only credentials (read-write and the empty default stay Read,Write,List,Tagging). The override lives in the package values.yaml rather than the vendored chart so it survives make update; the sister extra/seaweedfs chart already runs v0.3.1. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7c67aafed5 |
fix(kubernetes): run the Talos reconcile Job in the main install phase (#3145)
## What this PR does The tenant worker `MachineDeployment` `md0` intermittently never reaches `status.replicas=2`. CAPI blocks MachineSet creation until the `TalosConfigTemplate` that the deployment's `bootstrap.configRef` points to exists, and on failing installs that template is never created. ### Root cause — an ordering deadlock The `TalosConfigTemplate` is created only by the `talos-reconcile` **post-install Helm hook** (the chart can no longer render it at template time — three of its inputs are produced asynchronously after apply). A post-install hook runs only after Helm has applied the release's main resources and, when Helm waits, after they become Ready. But the same release also emits in-tenant addon HelmReleases — notably **cilium** — whose readiness needs tenant worker nodes, and those nodes can only be created once this hook has produced the `TalosConfigTemplate`. So when the main-resource wait is in effect, the install deadlocks: it waits for cilium to be Ready, cilium waits for worker nodes, the nodes wait for the `TalosConfigTemplate`, and the template waits for the post-install hook — which never runs, because the install never finishes the wait that precedes the hook phase. The install times out and `md0` never scales. The release sets `helm-install-disable-wait` precisely to skip that main-resource wait so the hooks run, but its effect is unreliable — which is what makes the failure bimodal: when the wait is skipped the hook runs and the template lands in ~46s; when it is not, the deadlock strands the install. A timestamped poller over a failing run (both `kubernetes-latest` and `kubernetes-previous`, 640 polls) captured: `TalosConfigTemplate` absent in 640/640 polls; both post-install hook Jobs (`talos-reconcile` and `bootstrap-token`) never created — the install never reached the post-install hook phase; all main resources applied (MachineDeployment, KubevirtMachineTemplate, MachineHealthCheck, WorkloadMonitor, child HelmReleases, Talos PKI/secrets) and the four hook inputs ready early and stable; the parent HelmRelease stuck at `Running 'install' action 20m` then looping into `upgrade`, with the cilium sub-release timing out on `cilium-operator` (which has no worker node to schedule on) — downstream of the missing template. ### The fix Render the `talos-reconcile` Job and its RBAC / ServiceAccount / egress `CiliumNetworkPolicy` as ordinary **main-phase** resources instead of post-install hooks. The Job is then created during the apply itself — not after the wait — and runs on the management cluster, which has nodes (unlike the tenant `cilium-operator`). It produces the `TalosConfigTemplate`, `md0` scales, workers join, cilium goes Ready, and the main-resource wait (if any) closes within the install budget: the Job completes in tens of seconds, and node boot plus cilium readiness is a few minutes — comfortably inside the 20m install timeout. This works whether or not `helm-install-disable-wait` takes effect, so it does not depend on that annotation's reliability. Because a main-phase Job is immutable, it carries a content-hash name suffix over its whole rendered spec (mirroring the `KubevirtMachineTemplate` idiom in `cluster.yaml`): an identical render keeps the same name and Helm no-ops it; any spec change yields a new name and a fresh Job, so no change ever lands on the same name and trips an immutable-field patch error. The applied `TalosConfigTemplate` keeps its stable `<release>-<group>` name (referenced by `bootstrap.configRef`), so an unchanged render re-applies a byte-identical template — a CAPI no-op that never rolls existing workers. The Job name is truncated to stay within the 63-char DNS-label limit while always preserving the hash. The runtime input-waits, the single `kubectl apply` of the template with a `KamajiControlPlane` ownerReference for GC, and the `extraHostEntries` are all unchanged. `helm-install-disable-wait`'s unreliable effectiveness is a real latent issue (it explains the created-vs-never bimodality), but this fix breaks the deadlock independent of it, so that is a separate follow-up rather than a blocker here. Supersedes #3139 (a 10-minute wait-budget bump that treated the symptom; the deadlock is structural and a larger budget would not have helped). ### Screenshots Not applicable — no UI changes. ### Testing helm-unittest extended to pin: no `helm.sh/hook` on any of the seven talos-reconcile documents, the content-hash name, `ttlSecondsAfterFinished`, the preserved template apply + ownerReference + extraHostEntries, the hash rotating on a spec change, and the long-name truncation staying within 63 chars. Full chart suite green (128 tests). The applied template is byte-identical to the previous hook output (verified by diff), so existing tenants are upgrade-safe. ### Release note ```release-note fix(kubernetes): create the tenant worker TalosConfigTemplate from a main-phase Job instead of a post-install hook, so worker MachineDeployments reliably scale during install ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes addon install behavior by running reconcile work during the main chart phase, reducing readiness deadlocks and upgrade issues. * Added more reliable reconciliation handling with stable, change-detected job recreation and updated network access for the reconcile workload. * Refined chart and test guidance to match the new install flow and Helm behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
f9d1138c6e |
feat(keycloak): configurable login theme via platform branding (#3124)
## What this PR does
Makes a realm's Keycloak login theme configurable through platform
`branding` values, closing a gap: `branding` could set the realm display
name but not its login theme, and the keycloak chart's `themes:` list
was not reachable from platform values.
Two new keys under `branding` (propagated via `_cluster.branding`, the
same path that already feeds `displayName`):
- `branding.loginTheme` — keycloak-configure renders it as
`ClusterKeycloakRealm.spec.themes.loginTheme`, so the realm selects that
login theme.
- `branding.loginThemeImage` — when set together with `loginTheme`, the
keycloak chart appends `{name: loginTheme, image: loginThemeImage}` to
its effective theme list, so the existing init-container mounts the
theme image into `/opt/keycloak/themes/`. No StatefulSet patching or
per-package values plumbing.
`loginTheme` alone (no image) still selects the theme — for themes
already baked into the Keycloak image. Explicit chart `themes:` entries
keep working; the branding-derived entry is appended after them.
Flow: platform `branding` → `_cluster.branding` (cozystack-values
Secret) → keycloak-configure (realm loginTheme) + keycloak (theme init
container). Covered by helm-unittest in both charts.
### Screenshots
No bundled UI assets — this PR adds the mechanism only; the login theme
is operator-supplied via `loginThemeImage`, so the visual result depends
on that image.
### Release note
```release-note
feat(keycloak): select the realm login theme and ship its theme image through platform `branding` values (`branding.loginTheme`, `branding.loginThemeImage`)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for configuring a custom login theme from platform
branding, including automatic selection in the realm setup.
* Theme entries can now be combined with existing theme settings and
rendered in the expected order.
* **Documentation**
* Expanded configuration guidance for branding and theme-related
settings with clearer examples.
* **Tests**
* Added coverage for login theme rendering and fallback behavior when
theme settings are missing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
31c15b282f |
feat(network): mechanism-agnostic LoadBalancer exposure (ExposureClass + ServiceExposure) (#3081)
## What this PR does Lays the foundation for migrating tenant LoadBalancer exposure off the deprecated `Service.spec.externalIPs` (KEP-5707) and, eventually, off MetalLB onto Cilium-native announcement — the mechanism-agnostic shape prescribed when the earlier chart-side approach (#2468) was reverted in #2603. It adds a new `network.cozystack.io/v1alpha1` API group and a controller that translates intent into whichever LoadBalancer backend the cluster admin configured: - `ExposureClass` (cluster-scoped, admin): binds a logical class name to a backend (`externalIPs` | `metallb` | `cilium` | `robotlb`) and its VIP address scope — the StorageClass/IngressClass pattern. - `ServiceExposure` (namespaced, tenant intent): "expose this Service via class X"; status reports the resolved backend and assigned IPs. - A `serviceexposure` controller renders the backend's pool + announcer once per class (MetalLB `IPAddressPool` + `L2Advertisement`; Cilium `CiliumLoadBalancerIPPool` + `CiliumL2AnnouncementPolicy`), scoped to the namespaces that opted in. Backends are pluggable behind a small Go interface, so adding BGP or another cloud LB is one implementation, not an API change. The controller never mutates the target Service — the chart still owns its shape. Platform wiring: `publishing.exposureClass.*` renders the admin `ExposureClass` and the host ingress publishes through it. The default (no class set, `externalIPs`) leaves every existing cluster's behaviour byte-for-byte unchanged; the MetalLB→Cilium announcer migration becomes a single `ExposureClass.spec.backend` change. This is the foundation only — Part of #2586. Converting per-app exposure, flipping Cilium `l2announcements` on, changing the default, and removing MetalLB are deferred to later phases. ### Release note ```release-note feat(network): add ExposureClass and ServiceExposure for mechanism-agnostic LoadBalancer exposure. Opt-in foundation; the default externalIPs publishing path is unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added new `ExposureClass` and `ServiceExposure` custom resources with support for multiple backends (externalIPs, MetalLB, Cilium, RobotLB), including optional L2 and address configuration. * Added Helm wiring and templates to enable service/ingress publishing via `ServiceExposure`, selecting a configured (or default) `ExposureClass`. * **Bug Fixes** * Improved reconciliation, finalizer-based cleanup, and status updates for readiness and assigned IP reporting, including safer handling when backends or managed resources change. * **Tests** * Added end-to-end and unit test coverage for backend rendering, readiness, and cleanup flows. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
50748fed98 |
fix(kubernetes): make the default md0 node group removable (#2936)
Same-repo mirror of #2753 by @mattia-eleuteri, opened so the `Build` CI job can run. Fork PRs skip the OCIR registry login (`if: !github.event.pull_request.head.repo.fork` in `pull-requests.yaml`) and so can't push the per-PR images the build needs — meaning a fork PR can never pass `Build`, regardless of the change. The commits here are unchanged and authored by @mattia-eleuteri; full description and discussion in #2753. This includes migration **44** (renumbered from 43 after the seaweedfs-db migration landed on `main`) and bumps `targetVersion` to **45**. Supersedes #2753. ```release-note fix(kubernetes): the default `md0` node group is now applied only when no node groups are configured, so it can be removed or replaced. Migration 44 pins `md0` explicitly on existing clusters to preserve their topology. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Empty `nodeGroups` settings now automatically provision a default `md0` worker group. * Existing environments are updated to keep this default group available during upgrades. * **Bug Fixes** * User-defined node groups now fully replace the default instead of being merged with it. * Improved handling of node-group resources and permissions in Kubernetes app rendering. * **Documentation** * Updated Kubernetes app docs and schema details to reflect the new default behavior. * **Tests** * Added coverage for default and custom node-group configurations, plus migration behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
81198b294f |
fix(bucket): clean up orphaned cert-manager ACME solver resources on delete (#3060) (#3076)
## What this PR does After deleting a `Bucket` app CR, cert-manager's temporary ACME HTTP-01 solver resources (`cm-acme-http-solver-*` Ingress/Pod/Service) were left orphaned in the tenant namespace. They are created by cert-manager to answer an ACME challenge and are owned by the `Challenge`/`Order` → `Certificate`, not by the app Ingress. If a challenge gets stuck and the app is deleted, the parent Ingress is removed but the orphaned challenge leaves the solver objects with no live owner to cascade-delete them, so Helm/Flux uninstall cannot reap them. This adds a `post-delete` cleanup hook that deletes the release's `Certificate`/`Order`/`Challenge` (cert-manager then cascade-removes the solver objects) and then sweeps any leftover solver objects by the reliable `acme.cert-manager.io/http01-solver=true` label. RBAC is verified against the vendored cert-manager CRDs (`cert-manager.io` → certificates; `acme.cert-manager.io` → orders, challenges) plus core `services`/`pods` and `networking.k8s.io` ingresses. The cleanup Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`. Failures are logged but never block teardown. The preventive wildcard-cert guard the issue mentions already exists in the vendored `bucket-system` Ingress template (it drops the ACME annotations in wildcard mode), so no preventive change is needed — only cleanup. No user data is affected: only cert-manager's transient challenge artifacts are removed. Fixes #3060. ### Release note ```release-note fix(bucket): clean up orphaned cert-manager ACME http01-solver resources (Ingress/Pod/Service) left behind after a Bucket app is deleted. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a bucket app `test` command that runs Helm unit tests. * Introduced a post-delete ACME cleanup hook to remove release-scoped cert-manager TLS and HTTP-01 solver resources, using dedicated least-privilege access controls. * **Bug Fixes** * Improved cleanup safety with bounded execution and hardened container security. * **Tests** * Added a Helm test suite covering rendered hook documents, including host-scoped targeting and negative assertions to prevent overly broad deletions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
544ba5bf16 |
fix(harbor): preserve jobservice and trivy storageClass on upgrade (#2930)
## What this PR does The `apps/harbor` chart never passed `storageClass` to the jobservice `jobLog` PVC, so the upstream chart rendered it without a `storageClassName`. On first install the default-StorageClass admission then stamps the cluster default onto the live PVC. Because a PVC's `storageClassName` is immutable, the rendered manifest (no class) and the live object (admission-defaulted class) drift apart, which surfaced as upgrade failures on the `harbor-*-system` HelmRelease. This propagates `storageClass` to the jobservice PVC, with one safeguard: a `lookup` reads the live jobservice PVC and reuses its current `storageClassName` when the PVC already exists, only falling back to the configured `storageClass` for a fresh install. Keeping the rendered value equal to the live one means an existing PVC whose admission-defaulted class differs from the configured `storageClass` is preserved rather than triggering an immutable-field rejection on upgrade. A live PVC pinned to an explicit `""` (no dynamic provisioning) round-trips through the upstream `"-"` sentinel instead of collapsing to an omitted field, and the spec access is guarded so a PVC with no `storageClassName` key stays omitted. The trivy `StatefulSet`'s `volumeClaimTemplate` carried the same latent drift — `.Values.storageClass` was forwarded blindly into a VCT, which cannot be patched at all — so it now gets the identical lookup-preserve treatment, keyed off `data-<release>-trivy-0`. This closes both halves of #2368. Also adds a helm-unittest regression suite and wires a `test:` target so `make unit-tests` stops skipping `packages/apps/harbor`. Validated with `helm template` across storageClass/trivy permutations and the new unit tests. The jobservice preserve path was additionally confirmed with a server-side dry-run against a running harbor release: `lookup` resolved the live PVC's class, and with `storageClass` overridden to a different class the jobservice render stayed pinned to the live PVC's class — confirming the immutable-field rejection is avoided. The trivy change applies the same proven pattern. Note: under `helm template`/helm-unittest `lookup` returns empty, so the unit suite exercises only the fresh-install fallback; the preserve branches are covered by the live `lookup`. Fixes #2368 ### Release note ```release-note fix(harbor): preserve the jobservice PVC and trivy volumeClaimTemplate storageClass on upgrade, fixing upgrade failures caused by storageClassName drift ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Harbor now preserves the existing jobservice storage class during upgrades and ensures it’s applied on initial installation when available, with consistent handling for Trivy persistence as well. * **Tests** * Added a Helm unittest regression suite covering storage class propagation and the correct conditional rendering of persistence blocks when storage class and Trivy settings vary. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
c818ece25b |
test(kubernetes): cover md0 removability with user-defined nodeGroups
The existing case pins the empty-nodeGroups -> default md0 path (the helper's else-branch). This adds the complementary if-branch: when a user supplies their own nodeGroups, they are authoritative and the built-in md0 is NOT merged in, so md0 is removable. The assertion selects every rendered MachineDeployment and requires it to be the user's own group, so a regressed md0 leak surfaces as a second MachineDeployment named <release>-md0 that fails the assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
ab55ff1924 |
test(kubernetes): cover the default md0 node group render
Assert that an empty nodeGroups renders the built-in md0 group via the kubernetes.nodeGroups helper else-branch (the behaviour that makes md0 removable): the default md0 MachineDeployment is emitted and the default 20Gi system disk flows through. A minimal VirtualMachineClusterInstancetype plus dummy CAPI objects are registered so the per-group instanceType lookup and the old-template preservation LISTs resolve under helm-unittest. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
b9749cec80 |
fix(platform): md0 migration -- fail closed and preserve the no-delete label
Migration 47 pins the implicit default md0 node group before the kubernetes chart upgrade so the live md0 MachineDeployment is not pruned. Two correctness fixes: - Stamp the cozystack-version ConfigMap via the shared stamp_cozystack_version helper instead of `kubectl create configmap | kubectl apply`. The label-less apply by the same field manager stripped the platform.cozystack.io/no-delete label migration 42 added, dropping cozystack-version out of the cozystack-no-delete-guardrail policy; the helper re-emits the label and is enforced for every migration >= 42 by hack/cozystack-version-stamp.bats. - Fail closed. A non-NotFound read error is now counted as a failure rather than silently skipped, and any failure exits 1 so the pre-upgrade hook aborts before Helm rolls out the new chart against clusters whose md0 is not yet pinned. The version stamp stays at 47 and the migration retries next upgrade. Also narrow the merge patch to only md0 so a concurrent edit to a sibling node group between the read and the patch cannot be reverted. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
mattia-eleuteri
|
602b8d06fd |
fix(platform): add migration 47 to pin implicit md0 node group
After the kubernetes chart stops force-merging a default md0 node group, clusters that relied on the implicit md0 (defined other nodeGroups without md0) would have their live md0 MachineDeployment pruned by Helm on the next reconcile. Migration 47 makes md0 explicit on those clusters to preserve the current node topology; it is idempotent, treats a missing CRD as a no-op, and skips clusters that already define md0. Signed-off-by: mattia-eleuteri <mattia@hidora.io> |
||
|
mattia-eleuteri
|
7e3c5f99db |
fix(kubernetes): apply default md0 node group only when nodeGroups is empty
The chart shipped a default nodeGroups.md0 in values.yaml. Helm deep-merges
values, so the default md0 was re-added on top of any user-supplied
nodeGroups; combined with Kubernetes stripping null values, md0 could never
be removed by configuration. Move the default into the template
("kubernetes.nodeGroups" helper) so it is applied only when nodeGroups is
empty. User-supplied node groups now fully replace the default, letting users
name and omit groups (including md0) freely.
Test fixtures that relied on the chart-default md0 to supply diskSize via
values merge now set it explicitly.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
|
||
|
Aleksei Sviridkin
|
32c4d60b43 |
fix(kubernetes): run the Talos reconcile Job in the main install phase
The talos-reconcile Job is the only thing that creates the TalosConfigTemplate the worker MachineDeployment's bootstrap.configRef points to; CAPI blocks MachineSet creation until it exists, so tenant workers cannot scale until the Job has run. The Job was a post-install/post-upgrade Helm hook, which runs only after Helm has applied the release's main resources and, when Helm waits, after they become Ready. But the same release emits in-tenant addon HelmReleases — notably cilium — whose readiness needs worker nodes, and those nodes need the TalosConfigTemplate this Job produces. So when the main-resource wait is in effect the install deadlocks: it waits for cilium to be Ready, cilium waits for workers, the workers wait for the template, and the template waits for this hook — which never runs because the install never finishes the wait that precedes the hook phase. The install times out and the MachineDeployment never scales. The release sets helm-install-disable-wait to skip that wait, but its effect is unreliable, which makes the failure bimodal: the template lands in ~46s when the wait is skipped, and the install is stranded when it is not. Render the Job and its RBAC/ServiceAccount/egress policy as ordinary main-phase resources. The Job is then created during the apply itself, not after the wait, and runs on the management cluster where nodes exist (unlike the tenant cilium-operator), so it produces the template, the MachineDeployment scales, workers join, cilium goes Ready, and the wait (if any) closes within the install budget — independent of whether helm-install-disable-wait takes effect. The Job is immutable, so it carries a content-hash name suffix over its rendered spec (mirrors the KubevirtMachineTemplate idiom in cluster.yaml): an identical render keeps the name and Helm no-ops it, while any spec change yields a new name and a fresh Job, so no spec change ever lands on the same name and triggers an immutable-field patch error. The applied TalosConfigTemplate keeps its stable <release>-<group> name, so an unchanged render re-applies a byte-identical template — a CAPI no-op that never rolls existing workers. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Алексей Артамонов
|
2587458bd4 |
fix(qdrant)!: reclaim data PVC by name prefix, not volumeClaimTemplate label
The previous approach routed a release-unique label through qdrant.persistence.additionalLabels, which the upstream chart renders into the StatefulSet's volumeClaimTemplates[].metadata.labels. That field is immutable on an existing StatefulSet, so for any already-deployed Qdrant the API server would reject the upgrade and the <release>-system HelmRelease (retries: -1) would retry-fail forever — permanently wedging reconciliation. Label changes on a volumeClaimTemplate also never apply to PVCs that already exist, so the cleanup hook could not reclaim those instances anyway. Switch the post-delete hook to select PVCs by the deterministic StatefulSet naming prefix (qdrant-storage-<release>-) instead. PVC names derive purely from the release name, so this needs no StatefulSet change (upgrade-safe) and also reclaims PVCs from instances created before the hook existed. - qdrant.yaml: drop the additionalLabels passthrough entirely. - hooks/cleanup-pvc.yaml: delete by name prefix via kubectl get -o name | grep. - tests: assert the name-prefix command instead of the label passthrough. - e2e: assert qdrant-storage-<name>-0 is gone after delete (async-waited). BREAKING CHANGE: deleting a Qdrant app now permanently removes its data PVC and the vector data it holds. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
6a43adbd09 |
fix(bucket): drop no-op label-based order/challenge delete
Per maintainer review: the label selector cert-manager.io/certificate-name=<release>-ui-tls matches nothing — Challenge objects carry no labels, and on Orders certificate-name is an annotation (not a label). Orders are already reaped by the Certificate owner cascade and Challenges by the host-scoped .spec.dnsName sweep, so the line deleted nothing and its comment misstated cert-manager labeling. Remove the dead delete, replace the inaccurate comment, and drop the test assertion that pinned it; add negative assertions so a label-based order/challenge delete cannot be reintroduced. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Aleksei Sviridkin
|
4404c5c5eb |
fix(etcd): pin tenant etcd to v3.5.31 to restore consistent watches (#3122)
## What this PR does Tenant Kubernetes control planes get their etcd from the `extra/etcd` chart, which creates an `EtcdCluster` (`etcd.aenix.io/v1alpha1`) reconciled by etcd-operator v0.4.5. That operator hardcodes the etcd image to `quay.io/coreos/etcd:v3.5.12`, and the v1alpha1 CRD exposes no version field, so every tenant etcd runs 3.5.12. kube-apiserver only enables the `RequestWatchProgress` storage feature on etcd >= 3.5.13 (or >= 3.4.31). Kubernetes >= 1.31 enables `ConsistentListFromCache` — locked-to-true in 1.35 — which depends on it. On etcd 3.5.12 every consistent watch therefore fails inside the stream with HTTP 500 (`the required storage feature RequestWatchProgress is disabled`), which breaks watch-based clients of aggregated APIs in modern (k8s 1.35) tenants. This pins the etcd container image to a >= 3.5.13 patch release (v3.5.31) by setting `image` on the `etcd` container in the EtcdCluster's `spec.podTemplate.spec`. etcd-operator v0.4.5 strategically merges `podTemplate.spec` over its generated pod spec (matched by container name), so the override takes effect with no operator change — the same mechanism the chart already uses for the metrics port, probes, and resources. It restores consistent watches without waiting on the etcd-operator v1alpha2 migration (#2859), which is the longer-term fix. A helm-unittest suite asserts the etcd container carries the pinned image and is no longer left on the operator default v3.5.12. Addresses #3080. ### Upgrade impact On merge, etcd-operator reconciles every existing tenant `EtcdCluster` and rolls its StatefulSet from 3.5.12 to 3.5.31. This is a safe in-minor rolling patch upgrade (no etcd storage-format change, quorum preserved by the operator's rolling strategy), so expect a brief, sequential etcd pod restart per tenant and no data migration. ### Release note ```release-note fix(etcd): pin tenant etcd to v3.5.31 so kube-apiserver can enable RequestWatchProgress (consistent watches) on Kubernetes 1.35 tenants ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Pinned the bundled etcd image to `quay.io/coreos/etcd:v3.5.31` to improve compatibility with Kubernetes watch-related storage behavior. * Prevented deployments from reverting to the older default etcd image. * **Tests** * Added a new test suite to verify the template uses the expected pinned etcd image and blocks regressions. * **Chores** * Updated dependency automation to recognize and manage the pinned etcd image version. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
4db565282b |
fix(postgres): garbage-collect init-job after completion (#3055) (#3069)
## Problem After deleting a `Postgres` app CR, the chart-managed init Job `<release>-init-job` and its completed Pod remain in the tenant namespace indefinitely (reported in #3055). The init Job is a Helm hook (`post-install,post-upgrade`) whose only delete policy is `before-hook-creation`. That policy deletes the previous hook only right before a new one is created (next upgrade) — it never fires on `helm uninstall`. With no `hook-succeeded` and no TTL, the completed Job and Pod leak. ## Root cause `packages/apps/postgres/templates/init-job.yaml` — the Job has no garbage-collection mechanism for its completed state. ## Fix Add `ttlSecondsAfterFinished: 120` to the Job spec. The native Kubernetes TTL controller then GCs the Job and its Pod ~2 minutes after completion, regardless of the Helm hook lifecycle. The init Job is a one-shot, idempotent reconciler of users/databases/roles (`init-script.yaml`) — once it has run it is not needed until the next upgrade (when Helm recreates it), so GC-ing the finished Job is safe. This approach deliberately **does not** touch the hook-delete-policy / hook-timeout semantics, which the `bootstrap.recovery` path depends on (see the header comment in `init-job.yaml`). Under `bootstrap.enabled` the Job is skipped entirely, so the change is inert there. ## Tests Added a `helm-unittest` suite `packages/apps/postgres/tests/init_job_cleanup_test.yaml`: - asserts the init Job renders `spec.ttlSecondsAfterFinished: 120` - asserts the init Job is skipped under `bootstrap.enabled: true` Verified locally with `helm template` (TTL present on default render; Job absent under bootstrap recovery). ## Test plan - [x] `helm template` renders `ttlSecondsAfterFinished: 120` on the init Job - [ ] `helm unittest packages/apps/postgres` green in CI - [ ] e2e `test-apps-postgres` green Fixes #3055 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Postgres bootstrap init Jobs now automatically clean up after completion using a post-finish TTL. * **Tests** * Added a new integration test suite to verify the init Job renders with the expected cleanup TTL, and that the init Job is skipped when bootstrap is enabled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
49555e562b |
fix(openbao): garbage-collect the data PVC on delete (#3058) (#3073)
## What this PR does
After deleting an `OpenBAO` app CR, the StatefulSet data PVC
(`data-<release>-0`)
was left orphaned in the tenant namespace, consuming storage
permanently. The
OpenBAO app renders a Flux `HelmRelease` over the vendored upstream
OpenBAO chart,
and StatefulSet-templated PVCs are never garbage-collected on
StatefulSet deletion
by default — the chart set no `persistentVolumeClaimRetentionPolicy`.
This passes a retention policy through the HelmRelease values. The
upstream chart
renders it onto the StatefulSet from
`server.persistentVolumeClaimRetentionPolicy`
(verified in the vendored `server-statefulset.yaml`) — **not** from
`server.dataStorage.*`:
```yaml
server:
persistentVolumeClaimRetentionPolicy:
whenDeleted: Delete
whenScaled: Retain
```
A `test` target is added to the package Makefile so the new
helm-unittest suite
(asserting the passthrough) actually runs in CI —
`hack/helm-unit-tests.sh` only
runs packages that define a `test` target.
> [!WARNING]
> **Deleting an OpenBAO app now permanently removes its data PVC and the
data it holds.**
> This is intentional and fixes the storage leak. `whenScaled: Retain`
is kept on
> purpose so scaling an HA/raft cluster down does **not** destroy a
member's data —
> only full deletion of the release reclaims storage. Back up before
deleting.
Fixes #3058.
### Release note
```release-note
fix(openbao): set persistentVolumeClaimRetentionPolicy so the data PVC is removed when an OpenBAO app is deleted, instead of being orphaned. WARNING: deleting an OpenBAO app now permanently removes its data PVC and the data it holds (scale-down still retains data).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable PVC retention behavior for OpenBao: storage is
deleted when the release is removed and retained when scaling down.
* **Tests**
* Added a Helm unit testing workflow to run chart/unit tests.
* Introduced assertions to confirm PVC retention settings are correctly
passed through and rendered as expected.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
scooby87
|
959a9b1fdb |
fix(kubernetes): clear kamaji datastore-secret finalizer to unblock namespace deletion (#3062) (#3078)
## What this PR does After deleting a `Kubernetes` app CR / tenant, the secret `<release>-datastore-config` kept the finalizer `finalizer.kamaji.clastix.io/datastore-secret`, which was never removed — **blocking deletion of the entire namespace** (observed stuck in `Terminating` for 115 minutes). Kamaji adds this finalizer to the datastore connection Secret and normally strips it as part of TenantControlPlane (TCP) deletion; if the TCP is force-deleted or Kamaji does not reconcile in time, the finalizer hangs and the Secret can never be garbage-collected, wedging the namespace. This extends the existing `pre-delete` cleanup hook (`packages/apps/kubernetes/templates/delete.yaml`): - **Step 4** now deletes the TenantControlPlane with `--wait=true --timeout=120s` so Kamaji's own finalizer-cleanup chain has a chance to run before we fall through. - **Step 4b (backstop)** patches `metadata.finalizers: null` on `<release>-datastore-config` only if it still exists. By this point the TCP is gone, the Secret is owner-ref'd to the deleted TCP and only awaiting GC, so clearing the finalizer merely lets GC reap it — it cannot strip a finalizer while a live TCP still needs the datastore. The Secret holds only connection config, not tenant data (which lives in the external etcd datastore), so no data is lost. - **RBAC** adds a `secrets` rule scoped via `resourceNames` to exactly `<release>-datastore-config`, verbs `get`/`patch` only. The Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`, writable `/tmp` for kubectl cache. Failures at both steps are logged (the Step 4b warning explicitly mentions the namespace may stay Terminating) but the hook still completes so it cannot itself block teardown. This is the most impactful of the create→delete leak sweep findings: without it, tenant teardown hangs and the create→delete cycle requires manual intervention. Fixes #3062. ### Release note ```release-note fix(kubernetes): clear the leftover Kamaji `finalizer.kamaji.clastix.io/datastore-secret` finalizer on the datastore-config Secret during tenant teardown, so deleting a Kubernetes app / tenant no longer leaves the namespace stuck in Terminating. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Extended the pre-delete cleanup workflow to more reliably remove `TenantControlPlane` and handle slow deletions with a bounded wait. * **Security** * Hardened the cleanup job by running as non-root with restricted security settings, dropped capabilities, and a safe writable temp location. * **Bug Fixes** * Added a conditional backstop to remove leftover `finalizers` from the relevant datastore Secret so namespace termination isn’t blocked. * **Tests** * Added a YAML test suite to validate the wait-and-patch behavior and ensure the patch runs only after the `TenantControlPlane` is gone. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
080ceac9c0 |
fix(gateway): clean up orphaned ACME account secret on tenant module delete (#3089) (#3093)
## What this PR does When the `gateway` tenant module is disabled (its HelmRelease uninstalled), cert-manager's ACME account private-key secret `cozystack-acme-account` was left orphaned in the tenant namespace (label `app.kubernetes.io/managed-by: cert-manager`, no `ownerReferences`). This adds a `post-delete` cleanup hook (`packages/extra/gateway/templates/hooks/cleanup-acme.yaml`) that deletes the secret by name. Verified on a live cluster that `cozystack-acme-account` is referenced by exactly one Issuer — the gateway module's own `cozystack-gateway` Issuer — so it is gateway-specific and safe to remove on gateway teardown (it is not shared with other modules, which use the cluster-wide `letsencrypt-prod` issuer). The cleanup Job is hardened: non-root (uid 65534), read-only root fs, all capabilities dropped, `seccompProfile: RuntimeDefault`, resource requests/limits; RBAC scoped via `resourceNames` to exactly `cozystack-acme-account` (get/delete). A failure is logged but never blocks teardown. No user data is affected — only cert-manager's ACME account key (re-registered on next enable). Fixes #3089. ### Release note ```release-note fix(gateway): clean up the orphaned cert-manager ACME account secret (cozystack-acme-account) left behind when a tenant's gateway module is disabled. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added automatic cleanup after gateway removal to delete the stored ACME account secret in the release namespace. * Included a dedicated, restricted cleanup job with the required permissions to run safely during uninstall. * **Bug Fixes** * Made the gateway test target always runnable. * Cleanup now ignores missing secrets and warns on failure instead of stopping uninstall flows. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6896a2a4db |
feat(kubernetes): expose tenant apiserver authentication in app values (#3123)
## What this PR does The managed `Kubernetes` app (Kamaji-backed tenant control planes) exposed no way to configure the tenant `kube-apiserver` authentication through the application values — there was no knob for OIDC, the structured `AuthenticationConfiguration`, extra apiserver flags, or extra volumes. The only workaround was patching the underlying `KamajiControlPlane` out-of-band, which is invisible to GitOps and lost when the cluster is recreated from values. This adds three opt-in fields under `controlPlane.apiServer`, all empty by default so existing clusters render unchanged: - `extraArgs` → `KamajiControlPlane.spec.apiServer.extraArgs` - `extraVolumeMounts` → `spec.apiServer.extraVolumeMounts` - `extraVolumes` → appended to `spec.deployment.extraVolumes` This unblocks OIDC SSO via the `--oidc-*` flags and, more importantly, structured authentication: `--authentication-config` mounted from a ConfigMap/Secret plus `--requestheader-uid-headers`, which is the only way to map a user UID for aggregated/extension apiservers. Because the control-plane Deployment runs on the management cluster, the passthrough is validated to keep tenant and management isolated: `extraVolumes` accept only `configMap`/`secret` sources (host-reaching sources such as `hostPath`/`csi` and token projection via `projected` are rejected); each volume needs a unique, non-empty name and exactly one source; the reserved `talos-ca`/`talos-tls-cert` names are rejected; and each `extraVolumeMount` must reference a user-declared `extraVolume`, so the chart-managed talos secret volumes cannot be mounted. Secret/ConfigMap references resolve in the tenant's own management-side namespace and are intentionally allowed — this stays within the tenant's existing security domain. Generated artifacts (`values.schema.json`, README, Go types + deepcopy, ApplicationDefinition) are regenerated from the values annotations. Helm unit tests cover the default no-op render, each field's passthrough, and every rejection path. Addresses #3074. ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(kubernetes): expose tenant apiserver authentication via controlPlane.apiServer.extraArgs / extraVolumes / extraVolumeMounts, enabling OIDC and structured AuthenticationConfiguration on tenant clusters ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for customizing the Kubernetes API server with extra command-line arguments, additional volume mounts, and extra volumes. * Updated chart values and schema so these options can be configured directly. * **Bug Fixes** * Added validation to prevent invalid or conflicting volume names and disallowed volume sources. * Ensured mounts only reference declared extra volumes and reserved control-plane names can’t be overridden. * **Tests** * Added coverage for default behavior, valid customization paths, and invalid configuration scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
80e3fa9cbe |
fix(dashboard): render error and unknown-type states instead of an infinite spinner (#3098)
## What this PR does Two console pages ordered their loading guard ahead of the terminal state, making the latter unreachable. Both are pre-existing upstream behaviors carried in by the cozystack-ui vendoring (#2963) and flagged as non-blocking findings in that PR's review. - **`ApplicationDetailPage`** checked `isLoading || !instance || !ad` before `if (error)`. On a failed GET (403/404/5xx) `isLoading` is false and `instance` stays undefined, so the page rendered the loading spinner forever and never reached the "not found" branch. The error branch now runs first. - **`ApplicationListPage`**'s `defsLoading || (!ad && plural)` guard stayed true for any unrecognized `plural` once definitions had loaded, so the "Unknown application type." branch was unreachable. The guard now only spins while definitions are loading. ## Verification - Added co-located regression tests for the failed-GET (detail) and unknown-plural (list) paths. - `pnpm typecheck` clean; full console suite green (`pnpm test`: 317 passed, +4 new). ```release-note fix(dashboard): the console now shows a not-found message on a failed application GET and an unknown-type message for an unrecognized resource, instead of spinning forever ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved loading and error handling on application pages so users see the correct message at the right time. * “Unknown application type” now appears only after application definitions finish loading, instead of being masked by a loading state. * Application detail pages now show a “not found” message when the requested app can’t be loaded, rather than continuing to display a spinner. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
61a574393a |
build(dashboard): pin console image builder to pnpm 11.1.2 and repoint vendor doc link (#3099)
## What this PR does Two post-vendor cleanups from the #2963 review: - **Pin the console image builder to pnpm 11.1.2.** The `Containerfile` ran `corepack prepare pnpm@latest`, so the container build could drift to a newer pnpm major than the `pnpm@11.1.2` pinned in `package.json` and used by CI. `--frozen-lockfile` guards the lockfile, but pinning the builder makes the image build reproducible and consistent with CI. - **Repoint a stale doc link.** `docs/storage-immutability.md` referenced the now-vendored `cozystack/cozystack-ui#6`; it now points at the in-tree console path. ## Verification - No source change; `pnpm typecheck` + `pnpm test` unaffected. - Builder pnpm version now matches `package.json`'s `packageManager` field. ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Clarified the storage immutability documentation by updating the UI reference to an in-repo path. * **Chores** * Standardized the build environment to use a specific pnpm version, helping keep installs and builds consistent. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
bdf520bf74 |
fix(harbor): preserve jobservice and trivy storageClass on upgrade
The cozy-harbor wrapper never passed storageClass to the jobservice jobLog PVC, so it was created without a storageClassName and the default-StorageClass admission stamped the cluster default. Because PVC storageClassName is immutable, this caused rendered-vs-live drift on upgrade. Preserve an existing PVC's class via lookup and fall back to the configured storageClass for fresh installs. The trivy StatefulSet's volumeClaimTemplate had the same latent drift -- .Values.storageClass was forwarded blindly into a VCT that cannot be patched at all -- so it gets the same lookup-preserve treatment, keyed off data-<release>-trivy-0. A live object pinned to an explicit "" (no dynamic provisioning) now round-trips through upstream's "-" sentinel instead of collapsing to an omitted storageClassName, and the spec lookup is guarded with `| default dict` so a PVC carrying no storageClassName key stays omitted. Adds a helm-unittest regression suite and wires a test target so make unit-tests stops skipping packages/apps/harbor. Fixes #2368 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
60c9b39578 |
docs(platform): document branding keys including login theme
The branding map shipped as an undocumented empty map. Describe the keys it actually supports (brandName, brandHtmlName) and the new login-theme keys (loginTheme, loginThemeImage) consumed by the Keycloak charts, with an example. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
e78a5842f5 |
feat(keycloak): ship a login theme image via branding
Build the StatefulSet theme list from .Values.themes plus, when branding.loginTheme and branding.loginThemeImage are both set, an auto-derived entry. This lets an operator deliver a Keycloak login theme through platform branding values alone (the init container mounts the theme image) without patching the StatefulSet or wiring per-package values. branding arrives via _cluster.branding, the same path that already feeds the realm displayName. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
fd9100c632 |
feat(keycloak-configure): select realm login theme from branding
Render ClusterKeycloakRealm.spec.themes.loginTheme from _cluster.branding.loginTheme, alongside the existing brandName -> displayName path. Without it a realm keeps the stock Keycloak login theme and there is no declarative way to switch it through platform values. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |