mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 20:53:52 +09:00
d13e2c586fcdc2a34384e9f27f97c12e6a92db0a
1097
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d13e2c586f |
fix(apps/kubernetes): align OIDC mount + path with the design spec
Self-review fix: the design proposal specifies `name: authentication-config` and `mountPath: /etc/kubernetes/authentication-config`, but the previous revision had renamed both to `cozy-oidc-authn-config` and `/etc/kubernetes/cozy-oidc` to dodge a collision with the volume name used in `apiserver_authentication_test.yaml`. That test was demonstrating the GENERIC user-supplied `extra*` passthrough — not the OIDC use case — so rename the test fixture instead and restore the design-spec names on the chart side. The reservation guard against user-supplied volumes reusing `authentication-config` only fires when `spec.oidc.mode` is not None, so clusters with OIDC off retain the previous freedom to name their own volumes anything. Renames in `apiserver_authentication_test.yaml`: the user-supplied volume name `authentication-config` becomes `tenant-byo-mount`; the Secret name `tenant-authentication-config` and the apiserver flag `--authentication-config=` are untouched. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
48fb4fe520 |
fix(apps/kubernetes): release-own the OIDC SA/Role/RoleBinding + pre-render the kubeconfig Secret
Self-review fixes for four linked lifecycle issues:
- The post-install Job created `<release>-oidc-kubeconfig` via
`kubectl apply` from inside the script; the resulting Secret had
no Helm release labels and was orphaned on `helm uninstall`. Now
a release-owned placeholder Secret is rendered up front in
`oidc-kubeconfig-secret.yaml` and the Job patches its `data`
field at runtime (the tenant CA still has to come from the
Kamaji-issued admin kubeconfig, which appears asynchronously).
- The ServiceAccount/Role/RoleBinding for the bootstrap Job were
hook resources with `helm.sh/hook-delete-policy: before-hook-creation`,
so they leaked on `helm uninstall`. They are now regular release
resources (no hook annotations) and are reaped by `helm uninstall`
alongside everything else. Both the post-install bootstrap Job
and the pre-delete cleanup Job find a working SA.
- The previous design tried to add `hook-succeeded` to the SA, but
that would delete the SA right after the bootstrap hook
succeeded — leaving the pre-delete cleanup Job with no SA to
schedule. Release-owned resources sidestep this race entirely.
- The Role had `verbs: ["create"]` on `secrets` without
`resourceNames` (Kubernetes RBAC cannot scope `create` by name).
With the placeholder Secret pre-rendered the Job only needs
`get`/`patch` on the specific name — the broad `create` rule is
removed.
Also tightens the script: `set -euo pipefail` (was `set -eu`, which
silently swallowed `kubectl get` failures inside command
substitutions) and uses `kubectl patch --type=merge` against the
existing Secret instead of `kubectl create --dry-run | kubectl apply`.
Adds three helm-unittest cases for the placeholder Secret across all
three modes.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
|
b7c553e653 |
docs(oidc): Phase-1 operator guide + chart NOTES
Add the operator-facing guide for the new selector — what the three modes mean, what the chart provisions, how a user logs in, what the failure modes are, and what is intentionally out of scope (per-tenant realms, federation, cross-cluster SSO). The architectural rationale points back to cozystack/community#24. Update the chart NOTES.txt to print the OIDC kubeconfig retrieval snippet, the per-cluster client/issuer values, and the listed `users[]` roster — only when OIDC is on, otherwise the previous admin-kubeconfig hint is unchanged. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
5bf7219f73 |
test(apps/kubernetes): helm-unittest cases for OIDC modes
Cover the three branches of the new selector API:
- `mode: None` (default) — every OIDC template is empty, the
KamajiControlPlane has no extra args and the standard two talos
volumes, the bootstrap Job is not rendered.
- `mode: System` — per-cluster `KeycloakClient` + audience scope
in the `cozy` realm, namespace-prefixed clientId, PKCE attribute
on the client, soft-render when the EDP Keycloak operator CRDs
are absent, the AuthenticationConfiguration Secret pinned to the
cozy issuer, the chart-owned `--authentication-config` flag and
`cozy-oidc-authn-config` mount injected into the KCP, the
bootstrap Job carrying `policy.cozystack.io/allow-to-apiserver`,
and one ClusterRoleBinding per `users[]` entry with the right
ClusterRole (`cluster-admin` vs `view`).
- `mode: CustomConfig` — inline config written verbatim; secretRef
path mounted on the KCP without a chart-owned Secret; no
Keycloak objects emitted; mutual-exclusion and missing-source
guards both hard-fail.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
|
c5295d8bad |
feat(apps/kubernetes): bootstrap Job applies per-user CRBs and writes OIDC kubeconfig Secret
Add a post-install,post-upgrade Helm-hook Job that reconciles the
operator-supplied `users[]` list into the tenant cluster:
- For every entry, apply a ClusterRoleBinding labelled by the
release (managed-by=cozystack-oidc, instance=<release>).
`role: admin` maps to `cluster-admin`; `role: view` maps to
`view`. The subject is the literal `email` value, matched
against the issuer's `preferred_username` claim.
- Prune any orphan CRBs with the same labels — removing a user
from `users[]` revokes their kubectl access on the next loop.
For `mode: System` the same Job also writes
`<release>-oidc-kubeconfig` on the management cluster: a ready
kubeconfig with the per-cluster apiserver endpoint, the tenant CA
extracted from the Kamaji-issued admin kubeconfig, and a
`kubectl oidc-login` exec block keyed to the per-cluster client.
A pre-delete cleanup Job removes the labelled CRBs before the
control plane is torn down (best-effort: the CRBs disappear with
the tenant cluster anyway).
Pod is labelled `policy.cozystack.io/allow-to-apiserver: "true"`
so Cilium lets it reach the konnectivity tunnel.
ServiceAccount/Role/RoleBinding are minimum-privilege — only the
specific admin-kubeconfig and OIDC kubeconfig Secrets are listed.
Renders nothing for `mode: None`.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
|
cef556ba4c |
feat(apps/kubernetes): wire structured AuthenticationConfiguration
Render a per-release Secret carrying the structured kube-apiserver `AuthenticationConfiguration` (apiserver.config.k8s.io/v1beta1) and mount it into the KamajiControlPlane. For `mode: System` the Secret is chart-owned: one JWT entry pointing at the `cozy` realm issuer with the per-cluster audience, plus the standard `preferred_username`/`groups` claim mappings. For `mode: CustomConfig` the chart either writes the operator-supplied inline config into a Secret, or — when `secretRef.name` is set — mounts an existing operator-owned Secret directly (mutually exclusive; the chart fails the render if neither or both are set). The `--authentication-config=` flag, mount, and volume on the KamajiControlPlane are kept orthogonal to the user-supplied `controlPlane.apiServer.extra*` knobs: the OIDC additions are appended, and the chart-managed volume name `cozy-oidc-authn-config` is reserved (only when `mode` is not None) so user-supplied volumes cannot collide. Switches from the legacy `--oidc-*` flag form to the file form so multiple issuers, inline private-CA PEM, and future Phase-2 issuers extend the same Secret without rewiring the chart. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
83980c8c47 |
feat(apps/kubernetes): render per-cluster Keycloak client + audience scope
For `oidc.mode: System`, emit a per-cluster public `KeycloakClient` plus a per-cluster `KeycloakClientScope` inside the platform `cozy` realm. The scope carries an `oidc-audience-mapper` that pins `id_token.aud` to the per-cluster clientId, so a token minted for cluster A is rejected by cluster B's apiserver. The clientId is namespace-prefixed (`<ns>-<release>`) to stay unique across the realm — two clusters with the same name in different tenants would otherwise collide. The client is `public: true`, redirects locked to localhost:8000 / localhost:18000 (the kubelogin/oidc-login defaults), with PKCE required via the standard Keycloak attribute. `defaultClientScopes` includes both `groups` and the per-cluster audience scope. A render-time guard fails fast when `oidc.mode: System` is set but `_cluster.oidc-enabled` is not `"true"` — the Keycloak operator CRDs are otherwise absent and the chart would silently no-op. Nothing is rendered for `mode: CustomConfig` or `mode: None`. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
5950450758 |
feat(apps/kubernetes): add Kubernetes.spec.oidc selector API
Introduce a flat selector — `mode: System | CustomConfig | None` — plus a `users[]` list and a `customConfig` block for tenant-supplied AuthenticationConfiguration. Default `None` preserves today's behavior. The `System` mode trusts the platform `cozy` realm via a per-cluster public client; `CustomConfig` accepts an inline `AuthenticationConfiguration` or a Secret reference (mutually exclusive). The `users[]` list drives per-user ClusterRoleBindings inside the tenant cluster, with two roles: `admin` and `view`. Regenerates deepcopy, JSON schema, README, and the kubernetes-rd cozyrds embedded schema. Implements Phase 1 API of the per-tenant OIDC design proposal (cozystack/community#24). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
50c4080ba3 |
[kubernetes] Add CPU/memory limits to csi-resizer sidecar
The csi-resizer container was the only sidecar in the kcsi-controller Deployment without resources.limits, triggering a 'no CPU limit' conformance warning. Align it with the other six CSI sidecars in the same Deployment (limits 512m/512Mi, requests 125m/128Mi) so capacity planning and audits stay uniform. Signed-off-by: Matthieu <matthieu@hidora.com> |
||
|
|
1baf126b1a |
fix(tenant): inherit full ancestor label chain from parent namespace (#2912)
## Summary
`packages/apps/tenant/templates/namespace.yaml` labels every tenant
namespace
with `tenant.cozystack.io/<ancestor>` for each of its ancestors. These
labels
drive the `<ancestor>-egress` `CiliumClusterwideNetworkPolicy`
(`networkpolicy.yaml`): an ancestor tenant's pods may reach a namespace
only if
that namespace carries `tenant.cozystack.io/<ancestor>`.
The chain was derived by splitting `.Release.Namespace`, which only ever
reached
one level up. A direct child of `tenant-root` got `tenant-root` (its
parent
namespace *is* `tenant-root`), but a grandchild — `tenant-ktj-htdev`
created
under `tenant-ktj` — got only `tenant-ktj` and dropped `tenant-root`
entirely.
Without `tenant.cozystack.io/tenant-root` on a nested namespace, the
root
ingress controller cannot reach the proxy service in a nested
`kubernetes`
tenant, surfacing as `upstream timed out (110) while connecting to
upstream` in
the nginx logs.
## Fix
Derive the ancestor chain deterministically from the namespace name
instead of
from any cluster lookup. Every tenant descends from `tenant-root`, so
that label
is emitted unconditionally; the remaining ancestors are encoded in the
namespace
name itself, because `tenant.name` builds each child as
`<parent-namespace>-<word>` — so every progressive dash-prefix of the
name is a
real ancestor namespace (`tenant-ktj-htdev` yields `tenant-ktj` and
`tenant-ktj-htdev`). The logic lives in a new
`tenant.ancestorTenantLabels`
helper:
```yaml
labels:
{{- include "tenant.ancestorTenantLabels" $tenantName | nindent 4 }}
```
This needs no cluster state, renders identically offline (so it is
unit-testable), and converges on every reconcile regardless of the order
in
which parent and child `HelmRelease`s reconcile — unlike a
parent-namespace
`lookup`, which helm-controller does not re-render on and which is
ordering-dependent. The `ownerReference` `lookup` is kept but made
non-fatal:
during a real reconcile the parent namespace always exists (the tenant
`HelmRelease` and its `cozystack-values` Secret live in it), so the
`ownerReference` is always emitted; when the lookup is empty (offline /
unit
test) it is simply omitted so the labels still render.
Migration `49` backfills the labels on existing clusters so they heal on
upgrade
regardless of per-tenant `HelmRelease` state (e.g. suspended or failed
releases), and `migrations.targetVersion` is bumped to `50` so the
runner
iterates it. The migration derives the same chain from each namespace
name,
is idempotent (`kubectl label --overwrite`) and best-effort
(per-namespace
failures are tolerated so a transient apiserver error cannot abort the
upgrade).
## Test plan
- [x] `helm template packages/apps/tenant` at depth 1/2/3 — emits
exactly the
expected ancestor set at each depth: depth-1 `{tenant-root,
tenant-ktj}`,
depth-2 adds `tenant-ktj-htdev`, depth-3 adds `tenant-ktj-htdev-foo`. No
missing, no stray labels.
- [x] `helm unittest packages/apps/tenant` — new
`tests/namespace_ancestor_labels_test.yaml` pins the chain at depths
1/2/3
plus a negative assertion; whole suite green.
- [x] `bats hack/cozystack-version-stamp.bats` — migration `49` stamps
the
version through the shared `stamp_cozystack_version` helper (all 10
tests
pass, including the guard that migrations >= 42 must use the helper).
- [x] Migration `49` ancestor derivation verified under `/bin/sh` to
match the
helm output byte-for-byte, and dry-runs cleanly under `set -euo
pipefail`.
- [ ] Live cluster: create a 3-level tenant (`root -> ktj -> htdev`),
verify
`tenant-ktj-htdev` has all three `tenant.cozystack.io/*` labels.
- [ ] Live cluster: create a `kubernetes` tenant under `htdev` with
`addons.ingressNginx.hosts`, verify an external HTTP request reaches the
tenant ingress controller via the root-ingress proxy (no `upstream timed
out`).
## Closes
- Closes #2810
- Closes #2171
```release-note
fix(tenant): tenant namespaces now carry the full `tenant.cozystack.io/<ancestor>` label chain (including `tenant-root`) at any nesting depth, restoring root-ingress reachability to nested tenants; a migration backfills the labels on existing clusters
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved tenant namespace label inheritance across multi-level tenant
hierarchies, ensuring all ancestor tenant labels are consistently
applied.
* **New Features**
* Standardized ancestor label generation to reliably include the full
tenant chain for derived tenant namespaces.
* **Tests**
* Added regression coverage for ancestor label presence and absence at
multiple hierarchy depths.
* **Chores**
* Added a migration to backfill missing ancestor labels for existing
tenant namespaces during platform upgrades.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
1ac9398a4b |
fix(tenant): emit full ancestor label chain on tenant namespaces
The tenant.cozystack.io/<ancestor> namespace labels drive the <ancestor>-egress CiliumClusterwideNetworkPolicy, which lets an ancestor tenant reach a descendant namespace only when the descendant carries tenant.cozystack.io/<ancestor>. namespace.yaml derived that chain by splitting .Release.Namespace, which only reached one level up: a direct child of tenant-root got tenant-root, but a grandchild (tenant-ktj-htdev under tenant-ktj) got only tenant-ktj and dropped tenant-root entirely. The root ingress controller in tenant-root then could not reach nested tenants, surfacing as upstream timeouts. Derive the chain deterministically from the namespace name instead. Every tenant descends from tenant-root, and tenant.name builds each child as <parent-namespace>-<word>, so every progressive dash-prefix of the name is a real ancestor namespace. This needs no cluster lookup, renders offline (so it is unit-testable), and converges on every reconcile regardless of parent/child HelmRelease ordering. The ownerReference lookup is made non-fatal so the template still renders when the parent namespace lookup is empty; in a real reconcile the parent always exists and the ownerReference is always emitted. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
2b1d3e904d |
fix(kubernetes): gate vmop cert-manager-crds dependency on certManager.enabled
The victoria-metrics-operator HelmRelease is gated on
monitoringAgents.enabled but listed cert-manager-crds in dependsOn
unconditionally, while the cert-manager-crds HelmRelease is only
created when certManager.enabled is true. The valid combination
monitoringAgents.enabled=true + certManager.enabled=false therefore
left vmop blocked on a non-existent dependency
("...cert-manager-crds ... not found"), cascading to monitoring-agents
and vertical-pod-autoscaler. Gate the dependency on the same condition
that creates it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
71ca2b1bae |
fix(mariadb): re-guard replication so single-replica passes the operator webhook (#3180)
## What this PR does Fixes a regression where **single-replica MariaDB can no longer be created**. [PR #2279](https://github.com/cozystack/cozystack/pull/2279) made `replication.enabled: true` unconditional in the MariaDB CR so the operator would always create `<release>-primary` / `<release>-secondary` Services (expected by the dashboard RBAC and the ApplicationDefinition). But the vendored mariadb-operator's validating webhook (`vmariadb-v1alpha1.kb.io`) **rejects `replication.enabled: true` when `spec.replicas == 1`** — replication needs a primary + at least one replica. As a result, single-replica installs never became Ready (the HelmRelease failed at admission). The webhook rule lives in the operator's Go code (not in the vendored CRD — there is no CEL rule tying replication to replica count), so the failing path is reproduced by an e2e test rather than a schema assertion. ### Fix — re-guard replication, teach consumers about the bare service The operator provisions a bare `<release>` Service in **both** topologies; `-primary`/`-secondary` exist **only** when replication is on (`replicas>1`). Instead of forcing replication for everyone, this PR lets replication track the replica count and makes the service-name consumers handle the single-replica bare service: - **`mariadb.yaml`** — gate `replication` on `replicas>1`; restore single-replica external access via `service.type: LoadBalancer`, and keep `primaryService: LoadBalancer` for `replicas>1` only. (Effectively reverts #2279's two hunks.) - **`backup-cronjob.yaml`** — single-replica dumps target the bare service (no `-secondary` exists without replication); `replicas>1` still offloads to `-secondary`. - **`dashboard-resourcemap.yaml`** — always grant RBAC on the bare service, plus `-primary`/`-secondary` when `replicas>1`, matching what the ApplicationDefinition lists. - **`mariadb-rd`** — list the bare service in `services.include` so single-replica installs are visible in the dashboard. - **`hack/e2e-apps/mariadb.bats`** — new `Create single-replica MariaDB` test that reproduces the webhook rejection and guards against regression. Verified: `helm template` at `replicas=1`/`2` renders the intended CR/RBAC/backup host; `helm unittest` passes; `update-crd.sh` keeps the RD stable; `select-e2e.sh` maps the change to the mariadb suite. ### Release note ```release-note fix(mariadb): single-replica instances can be created again — replication is now enabled only for replicas>1 (matching the operator webhook), while the dashboard, RBAC, and backups correctly target the bare service for single-replica setups ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an end-to-end test covering MariaDB deployments with a single replica, including validation of the main service, endpoints, StatefulSet replica count, and metrics components. * **Bug Fixes** * Updated MariaDB chart behavior to correctly handle single-replica vs multi-replica configurations for external access, replication settings, and backup host selection. * Refined dashboard RBAC and secret inclusion so access applies to the main MariaDB service in addition to replica-specific services. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a76c5929fd |
fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry
The post-delete cleanup Jobs in the harbor and mariadb charts hardcoded `docker.io/clastix/kubectl:v1.32` — a moving tag pulled directly from Docker Hub. This broke build reproducibility (the tag can be re-pushed), was a supply-chain risk, and could not be satisfied on air-gapped / mirrored clusters, where it failed at the worst moment: during uninstall, when the post-delete hook runs and stalls release teardown. Add a canonical `cozy-lib.image` helper that prefixes a registry-relative image path with `cozy-lib.images-registry` when set and returns it unchanged (no leading "/") when empty, so standard installs resolve from the default registry and mirrored installs resolve internally. Consume it from both cleanup hooks with the image digest-pinned to the immutable multi-arch index sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c (the same digest already pinned for postgres-operator's webhook hook). Keep the pin fresh with a custom Renovate manager: the repo disables the helm-values manager (enabledManagers), and the ref is built at render time by the helper rather than being a Dockerfile/go.mod dependency, so the built-in managers cannot see it. A regex manager over the hook templates tracks clastix/kubectl by tag+digest — mirroring the existing extra/etcd manager — so the supply-chain pin does not silently age. Also harden `cozy-lib.images-registry` against a nil .Values._cluster so the helper never crashes the render — relevant precisely because the hook renders during teardown. Helm unit tests assert the rendered image (routed, digest-pinned, no leading slash) by pattern rather than an exact digest, so a Renovate bump keeps them green. Closes #3088 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
c7e8f4d975 |
fix(mariadb): re-guard replication so single-replica passes the operator webhook
PR #2279 made `replication.enabled: true` unconditional so the operator would always create -primary/-secondary services. But the mariadb-operator validating webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1, so single-replica MariaDB could no longer be created — the HelmRelease never became Ready. Re-guard replication on replicas>1 and teach the service-name consumers about the bare <release> service the operator provisions in both topologies: - mariadb.yaml: gate `replication` on replicas>1; restore single-replica external access via `service.type: LoadBalancer` and keep `primaryService` LoadBalancer for replicas>1 only. - backup-cronjob.yaml: route single-replica dumps to the bare service (no -secondary exists without replication). - dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus -primary/-secondary when replicas>1, matching the ApplicationDefinition. - mariadb-rd: list the bare service in services.include so single-replica installs are visible in the dashboard. Adds a single-replica e2e test that reproduces the webhook rejection and guards against regression. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
fdbe6b299c |
fix(clickhouse)!: clean up orphaned keeper and data PVCs on delete (#3057) (#3072)
## What this PR does After deleting a `ClickHouse` app CR, data PVCs were left orphaned in the tenant namespace and consumed storage permanently: - the keeper PVC `default-chk-<release>-keeper-cluster1-0-0-0`, backing a `ClickHouseKeeperInstallation` reconciled by the clickhouse-keeper-operator; - the main ClickHouse data / log PVCs. The keeper operator does not delete data PVCs when the CR is removed (data-safety default) and the CRD exposes no `persistentVolumeClaimRetentionPolicy` equivalent, so the volumes leaked. This adds a `post-delete` cleanup hook (modeled on the existing MariaDB hook at `packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml`) that deletes every PVC carrying the release label `app.kubernetes.io/instance=<release>`. It also adds that label to the keeper `volumeClaimTemplates` in `chkeeper.yaml`: the keeper PVC previously had **no** release label (the chart set labels only on the keeper pod template, not the PVC), so the operator never stamped one and a single selector would have missed it — leaving the reported leak unfixed. ### Existing releases (label backfill migration) The altinity clickhouse-keeper-operator only stamps `volumeClaimTemplates.metadata.labels` onto PVCs it creates **after** the chart change; it never re-labels PVCs that already exist. So for releases installed before this fix, the keeper PVC stays unlabeled and the cleanup selector would still miss it. To close #3057 for existing customers too, this PR adds migration `48` (`48 -> 49`, `packages/core/platform/images/migrations/migrations/48`) that back-fills `app.kubernetes.io/instance=<release>` onto pre-existing keeper PVCs, and bumps `migrations.targetVersion` to `49`. The migration: - routes its version stamp through the shared `lib/cozystack-version.sh` helper (`stamp_cozystack_version 49`), per the `cozystack-version-stamp.bats` convention; - no-ops when the keeper CRD is absent, and tolerates a transient apiserver error on the keeper list (`|| true`) so it cannot abort a platform-wide upgrade — the backfill is best-effort (worst case is the pre-existing leak, no worse than today); - is idempotent (skips PVCs already labeled). The main ClickHouse data/log PVCs have carried `app.kubernetes.io/instance` since 2025-06-03 (well before the migration era), so they need no backfill. ### Hook failure handling The cleanup hook does **not** mask failures: `kubectl delete` failing makes the Job exit non-zero. The **Job** uses `helm.sh/hook-delete-policy: before-hook-creation` only (no `hook-succeeded`) so a failed cleanup Job and its logs survive for debugging; the **ServiceAccount/Role/RoleBinding** keep `before-hook-creation,hook-succeeded` so the RBAC artifacts are reclaimed after a successful uninstall instead of lingering in the namespace. ### Scope The hook intentionally reclaims **only PVCs** of the release. Other release state (Secrets, ConfigMaps) is owned/garbage-collected through the normal Helm uninstall path and is out of scope here. > [!WARNING] > **This is destructive — deleting a ClickHouse app now permanently deletes its data.** > The hook removes **all** PVCs of the release: the keeper PVC **and** the main > ClickHouse data and log PVCs. After deletion nothing is left behind to recover > data from. This is intentional and matches the existing MariaDB cleanup-hook > behaviour and the general cozystack model (deleting a managed app reclaims its > storage). Back up before deleting. Fixes #3057. ### Release note ```release-note fix(clickhouse): garbage-collect the keeper and data/log PVCs when a ClickHouse app is deleted, so no orphaned volumes are left behind. A migration back-fills the release label onto keeper PVCs from releases installed before this fix so they are cleaned up too. WARNING: deleting a ClickHouse app now permanently removes all of its PVCs (keeper + data + logs) and the data they hold. ``` |
||
|
|
9e89ae84ac |
fix(kubernetes): default worker disk storageClass to the application storageClass (#3129)
## What this PR does
Worker node VMs in a tenant Kubernetes cluster set `evictionStrategy:
External`, so they live-migrate. Live migration requires the worker
system DataVolume to be ReadWriteMany, and linstor-csi only grants RWX
on a DRBD-backed StorageClass. When a node group leaves `storageClass`
empty, the worker DataVolume template previously omitted
`storageClassName` and fell through to the management cluster's default
StorageClass — which is operator-configured and not guaranteed to be
DRBD-backed. On any cluster whose default StorageClass is not a
replicated/DRBD class, the worker disk lands on a non-RWX class and live
migration breaks.
This changes the empty-`storageClass` fallback from the management
cluster default to the application-level `storageClass` (default
`replicated`, a DRBD/LINSTOR class): `{{- with .group.storageClass |
default $.Values.storageClass }}`. A per-node-group `storageClass` still
wins when set; when both the node group and the application
`storageClass` are empty, `storageClassName` is omitted so the cluster
default still applies (explicit opt-out preserved).
Doc and codegen surfaces (`types.go`, `values.yaml`,
`values.schema.json`, `README.md`, and the generated `kubernetes-rd`
ResourceDefinition) are regenerated in lockstep with the new field
description. Chart unit tests cover the full contract: fallback to the
application class, a custom application class, per-node-group override
winning, and the both-empty omit path.
**Upgrade impact — one-time worker-pool roll.** The
`KubevirtMachineTemplate` name embeds a hash of the rendered worker
template body, and the MachineDeployment references that name. On the
shipped default (`md0` leaves `storageClass` empty) the body changes
from "no `storageClassName`" to `storageClassName: replicated`, so on
upgrade the template is renamed and Cluster API performs a one-time
rolling replacement of the worker pool — for every existing tenant
cluster, including those whose default StorageClass already was
`replicated` (the roll is driven by the body hash, not by whether the
effective class differs). This is the same class of roll any
worker-template change triggers and is unavoidable with this approach.
Two test pins document and guard the hash so future body changes are
deliberate.
### Release note
```release-note
fix(kubernetes): worker node disks now fall back to the application-level storageClass (a replicated/DRBD class) instead of the management-cluster default when a node group leaves storageClass empty, so live migration of worker VMs gets the ReadWriteMany volume it requires. Upgrading rolls each tenant cluster's worker pool once.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved worker disk `storageClassName` fallback for empty
per-node-group values: it now reliably uses the app-level
replicated/DRBD StorageClass (rather than relying on the
management-cluster default).
* Ensured KubeVirt data volume templates and rendering behavior are
consistent across single- and multi-node-group setups.
* **Documentation**
* Refined `storageClass`/RWX/DRBD guidance across configuration docs and
schema descriptions, including the intended non-immutable behavior
(optional/undefaulted).
* **Tests**
* Updated template test expectations to cover the new fallback vs
omission cases, including naming assertions for rendered templates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
7faa32508d |
fix(clickhouse): reclaim cleanup hook RBAC on successful uninstall
Address review [MINOR] on #3072: keep the cleanup Job on before-hook-creation (so a failed Job and its logs survive for post-mortem), but restore before-hook-creation,hook-succeeded on the ServiceAccount/Role/RoleBinding so those artifacts do not linger in the namespace after a successful uninstall. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
2e65ce108b |
fix(clickhouse): guard migration when keeper CRD absent and quote hook scalars
Address CodeRabbit review on #3072: - Migration 45 now skips the backfill when the clickhousekeeperinstallations CRD is not installed. The migration runs platform-wide via targetVersion, so on clusters that never deployed ClickHouse the kubectl get would otherwise fail under set -euo pipefail and abort the whole upgrade. - Quote the templated name/label/serviceAccountName scalars in the cleanup hook so the source parses under YAMLlint; rendered output is unchanged. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
a2ae1621af |
fix(clickhouse): surface cleanup hook failures and backfill legacy keeper PVC labels
Address review on #3072: - Cleanup hook no longer masks kubectl delete failures: the command now fails (exit 1) when PVC deletion fails instead of swallowing the error with || echo. The hook-delete-policy drops hook-succeeded (keeps only before-hook-creation) so a failed cleanup Job survives for inspection. - Add migration 45 (45 -> 46) that backfills app.kubernetes.io/instance onto keeper PVCs created before this change; the altinity keeper operator never re-stamps volumeClaimTemplate labels onto existing PVCs, so without this the cleanup selector misses pre-upgrade keeper PVCs and #3057 persists for existing releases. Bump migrations.targetVersion to 46. - Unify helm.sh/hook-weight key quoting across the four hook manifests. - Extend the unittest to assert the new hook-delete-policy and exit-on-failure. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
2c8f6b8ca1 |
fix(clickhouse)!: clean up orphaned keeper and data PVCs on delete (#3057)
After deleting a ClickHouse app CR, the keeper PVC and the main data/log PVCs were left orphaned in the tenant namespace, consuming storage permanently. The clickhouse-keeper-operator does not delete data PVCs on CR removal (data-safety default) and the CRD exposes no persistentVolumeClaimRetentionPolicy equivalent. Add a post-delete cleanup hook (modeled on the MariaDB hook) that deletes every PVC carrying the release label app.kubernetes.io/instance=<release>. Also stamp that label on the keeper volumeClaimTemplates in chkeeper.yaml: the keeper PVC previously had no release label (the chart labelled only the keeper pod template, not the PVC), so a single selector would have missed the very PVC the issue reports. BREAKING CHANGE: deleting a ClickHouse app now permanently removes all of its PVCs (keeper + data + logs) and the data they hold. Nothing is left behind to recover from. This matches the existing MariaDB cleanup-hook behaviour and the general cozystack model where deleting a managed app reclaims its storage. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
|
6db5ae9c6f |
fix(tenant): gate VM-CR-creating HelmReleases on victoria-metrics-operator
The tenant etcd, ingress and monitoring HelmReleases create VM* custom resources (VMPodScrape directly in etcd/ingress; VMCluster, VMAlert, VLCluster, VMAlertmanager and VMServiceScrape via the monitoring-system child of monitoring) guarded by the victoria-metrics-operator validating webhook with failurePolicy: Fail. On a cold install the operator pod can be briefly unavailable on :9443, so these releases reconcile before the webhook backend is serving and are rejected with connection refused. They carry install.remediation.retries: -1 and would eventually recover, but the failed-install -> rollback -> retry churn can outlast a cold-install readiness deadline and surfaces as spurious install failures. Add a Flux spec.dependsOn from each of these releases to the victoria-metrics-operator HelmRelease in cozy-victoria-metrics-operator, so they wait for the operator to be Ready before creating VM* resources. The system monitoring components gate on the operator the same way, via a variant-level dependsOn. The cross-shard reference (tenant releases are sharded, the operator release is not) resolves via helm-controller's uncached dependency check (APIReader, v1.1.0+). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
22c96d8dac |
fix(kubernetes): default worker disk storageClass to the application storageClass
Worker node VMs set evictionStrategy: External, so they live-migrate. Live migration requires the system DataVolume on a ReadWriteMany volume, and linstor-csi only grants RWX on a DRBD-backed StorageClass. When a node group leaves storageClass empty the template previously omitted the field and fell through to the management cluster default StorageClass, which may be a non-DRBD class the CSI driver rejects for RWX — breaking migration. Fall back to the application-level storageClass (the replicated/DRBD class) when the node group storageClass is empty, instead of the cluster default. When both are empty the field is still omitted, preserving the explicit cluster-default opt-out. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
5b26ec3462 |
fix(mariadb): clean up orphaned operator-generated secrets (#3056) (#3071)
## What this PR does After deleting a `MariaDB` app CR with metrics and/or replication enabled, two Secrets created by the mariadb-operator were left orphaned in the tenant namespace: - `<release>-metrics-password` - `<release>-repl-password` These are generated by the operator (`k8s.mariadb.com/v1alpha1`) as a side effect of `metrics.enabled` / `replication.enabled`, so they are **not** Helm-owned and survive `helm uninstall`. The existing post-delete hook (`templates/hooks/cleanup-pvc.yaml`) only swept PVCs, leaving the Secrets behind. This PR extends that hook to also clean them up: - grants the cleanup `Role` RBAC on `secrets`; - deletes both password Secrets by name with `--ignore-not-found`; - adds a `helm-unittest` suite locking in the behaviour, plus a `test` Makefile target so the package's tests actually run in CI (`hack/helm-unit-tests.sh` only runs packages that define a `test` target). Fixes #3056. ### Release note ```release-note fix(mariadb): garbage-collect operator-generated metrics-password / repl-password Secrets when a MariaDB app is deleted ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved MariaDB uninstall cleanup to remove leftover operator-generated password Secrets in addition to persistent volumes/claims. * Hardened the cleanup hook Job with stricter pod/container security settings, least-privilege Secret permissions, idempotent deletions for missing resources, and correct non-zero exit on real failures. * **Tests** * Added a cleanup hook test suite validating Secret deletion behavior, failure propagation, and that cleanup RBAC grants least-privilege access only to the required Secrets. * Added a Helm unit test `test` target for the MariaDB chart. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
bef42229d4 |
fix(harbor): clean up orphaned jobservice PVC and ACME solver resources on delete (#3061) (#3077)
## What this PR does After deleting a `Harbor` app CR, two things were left orphaned in the tenant namespace: - the **jobservice / trivy PVCs** (`<release>-jobservice`, and `<release>-trivy` when Trivy is enabled), because `packages/apps/harbor/templates/harbor.yaml` sets `persistence.resourcePolicy: "keep"`, which annotates the upstream Harbor subchart's PVCs with `helm.sh/resource-policy: keep` so Helm/Flux deliberately never deletes them; - cert-manager's **ACME http01-solver** resources (`cm-acme-http-solver-*`), via the same stuck-challenge mechanism as the Bucket issue (#3060) when Harbor issues a non-wildcard cert. This adds a single `post-delete` cleanup hook that: - **deletes the upstream Harbor subchart's kept PVCs** — selecting by `app.kubernetes.io/instance=<release>-system` (the subchart runs as the `<release>-system` HelmRelease, so its PVCs carry that label, **not** `<release>`). A hook is required precisely because `resourcePolicy: keep` blocks normal reclaim. This intentionally covers both the jobservice and trivy PVCs: both hold only regenerable data (jobservice file logs; trivy's vulnerability-DB cache), and registry images live in S3. - **deletes this release's `Certificate`** (deterministic name `<release>-ingress-tls`); cert-manager then cascade-deletes the `Order`, `Challenge` and solver `Pod`/`Service`/`Ingress` objects it owns via owner references. - **host-scoped belt-and-braces sweep** for any ACME http01 solver objects cert-manager may have orphaned (e.g. the Certificate was already gone, so deleting it did not cascade). The sweep is scoped **strictly to this release's Harbor host** — Challenges filtered by `.spec.dnsName`, solver Ingresses by `.spec.rules[0].host` — **not** a namespace-wide `acme.cert-manager.io/http01-solver=true` label sweep, because a Cozystack tenant namespace can host many apps and a blanket label sweep would delete sibling apps' in-flight challenge solvers. The Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`, with CPU/memory requests and limits set. RBAC is scoped to exactly the resources it deletes (`persistentvolumeclaims`, `certificates`, `orders`, `challenges`, solver `ingresses`) and verified against the vendored cert-manager CRDs. Failures are logged but never block teardown. ## Cleanup scope — what is and isn't removed **Removed:** the upstream Harbor subchart's kept PVCs (jobservice + trivy) and this release's cert-manager ACME artifacts (Certificate, and host-scoped Order/Challenge/solver-Ingress). **Not touched (by design):** - **CNPG database PVCs** (`<release>-db-N`) — owned by the CNPG `Cluster` and governed by CNPG's own reclaim policy; they carry CNPG labels, not `app.kubernetes.io/instance=<release>-system`. Re-installing Harbor with a different DB size into the same namespace can therefore leak the old DB PVCs; this is left to CNPG's lifecycle on purpose (the data is the user's). - **Redis PVCs** — managed by the redis-operator with their own labels, likewise untouched. - **Solver `Pod`/`Service`** — not deleted directly; they are removed by cert-manager's owner-ref cascade when the Certificate/Challenge is deleted, so the hook grants no RBAC for them. > [!NOTE] > The PVC sweep is eventually-consistent: while the inner `<release>-system` > HelmRelease is still being uninstalled by Flux, a PVC may briefly stay in > `Terminating` until its consumer Pods (jobservice/trivy) exit. The Job's > deletion request always succeeds; reclaim completes once the pods are gone. > [!WARNING] > **This overrides the intentional `persistence.resourcePolicy: "keep"` setting > and force-deletes the jobservice/trivy PVCs on app deletion.** Those PVCs hold > only jobservice file logs and the trivy vuln-DB cache (registry images live in > S3), so data-loss risk is low — but if retaining them across re-installs is > desired, this is a conscious product decision to revisit. The ACME cleanup > affects only cert-manager's transient artifacts. Fixes #3061. > Follow-up tracked in #3088: digest-pin the cleanup `kubectl` image and route it > through `cozy-lib.images-registry` (pre-existing debt shared with the MariaDB > cleanup hook), out of scope for this PR. ### Release note ```release-note fix(harbor): clean up the orphaned jobservice/trivy PVCs and cert-manager ACME http01-solver resources left behind after a Harbor app is deleted. NOTE: this force-deletes the jobservice/trivy PVCs, overriding persistence.resourcePolicy=keep (jobservice file logs and trivy vuln-DB cache only; registry images are in S3). CNPG database PVCs are not touched. ``` |
||
|
|
3e20a0cd41 |
fix(kubernetes): make node group resources optional when instanceType is set (#3121)
## What this PR does
A tenant Kubernetes node group can be sized two ways: by `instanceType`
or by an explicit `resources` block. The generated values schema,
however, marked `resources` as required on every node group, so a node
group sized purely by `instanceType` was rejected at install time:
```text
at '/nodeGroups/<name>': missing property 'resources'
```
The same failure occurs through the typed `Kubernetes` CR: an empty
`resources: {}` is dropped during the CR-to-HelmRelease translation, so
by validation time the key is absent and the required check fails — even
though sizing was already fully specified by `instanceType`.
This marks `resources` optional in the node group definition and
regenerates the dependent artifacts (`values.schema.json`, API types,
README, and the `ApplicationDefinition`). No change is needed to emit
the VM: the chart already renders `domain.cpu`/`domain.memory` only when
`resources` is present and `instancetype` only when set, so a node group
sized purely by `instanceType` now validates and installs. Node groups
that specify `resources` are unaffected.
`instanceType` and `resources` are alternative, mutually exclusive
sizing sources — KubeVirt rejects a VirtualMachine that references an
instancetype and also sets explicit `domain.cpu`/`domain.memory`, since
instancetype CPU/memory cannot be overridden. To avoid emitting a VM
that the instancetype admission webhook would later reject, the chart
now fails fast at render time with a clear message when a node group
sets both a non-empty `instanceType` and explicit `resources`.
Tests: helm-unittest coverage proves a node group with `instanceType`
and no `resources` passes schema validation and renders without explicit
`domain.cpu`/`memory`; a complementary case confirms explicit
`resources` are still honoured; and a `failedTemplate` case pins the
render-time rejection of the conflicting both-set combination. The
omit-resources case fails against the pre-fix schema with the exact
error above, so it is a real regression guard.
Addresses #3046.
### Screenshots
N/A — schema and chart change, no UI impact.
### Release note
```release-note
fix(kubernetes): allow a node group to omit `resources` when it is sized by `instanceType` (previously failed schema validation with "missing property 'resources'"); reject the unsupported combination of `instanceType` and explicit `resources` at render time
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Worker node groups can optionally size CPU/memory via `resources`, and
can omit `resources` when using `instanceType`.
* **Bug Fixes**
* Render now validates CPU/memory rules: `instanceType` with only one of
`resources.cpu`/`resources.memory` fails, and `instancetype` is omitted
when both `resources.cpu` and `resources.memory` are set (with
`instanceType` ignored).
* **Documentation**
* Updated API/Helm docs and JSON schema to clarify `resources` vs
`instanceType` precedence and the “set both CPU and memory (or neither)”
requirement.
* **Tests**
* Added helm-unittest scenarios covering omission, precedence, and
render-time failure cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
0892c5a38c |
fix(qdrant)!: clean up orphaned data PVC on delete (#3059) (#3075)
## What this PR does After deleting a `Qdrant` app CR, the StatefulSet data PVC (`qdrant-storage-<release>-0`) was left orphaned in the tenant namespace, consuming storage permanently. StatefulSet-templated PVCs are never garbage-collected on StatefulSet deletion by default. The vendored upstream Qdrant chart does **not** support `persistentVolumeClaimRetentionPolicy` (verified: its `statefulset.yaml` never renders that field), so the OpenBAO-style value passthrough is not possible here. Instead this adds a `post-delete` PVC cleanup hook (modeled on the MariaDB hook) that deletes the release's PVCs by label. The chart's volumeClaimTemplate only stamps `app: qdrant` (shared by every Qdrant in a namespace), so to select PVCs safely per release this also passes `persistence.additionalLabels.app.kubernetes.io/instance: <release>` through the HelmRelease — giving the data PVC a release-unique label the hook selects on. The cleanup Job is hardened: runs as non-root (uid 65534), read-only root fs, all capabilities dropped, `seccompProfile: RuntimeDefault`; RBAC scoped to `persistentvolumeclaims`. A real cleanup failure is logged (not silently swallowed) but never blocks namespace teardown. > [!WARNING] > **Deleting a Qdrant app now permanently removes its data PVC and the vector data it holds.** > This is intentional and fixes the storage leak; the hook only fires on full > release deletion (not on scale-down). Back up before deleting. Fixes #3059. ### Release note ```release-note fix(qdrant): garbage-collect the StatefulSet data PVC when a Qdrant app is deleted, instead of leaving it orphaned. WARNING: deleting a Qdrant app now permanently removes its data PVC and the data it holds. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a post-delete cleanup hook for Qdrant to remove leftover StatefulSet data PVCs after deletion. * Included hardened Kubernetes settings for the cleanup job to improve runtime safety. * **Tests** * Added Helm chart tests that validate the post-delete cleanup job and its RBAC permissions. * Enhanced end-to-end verification to confirm the data PVC is fully removed after deleting a Qdrant instance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b25f294d46 |
fix(harbor): select kept PVCs by release label so the trivy PVC is also cleaned
The post-delete cleanup hook selected PVCs by app.kubernetes.io/instance=<release>-system, a label only the jobservice PVC carries (via the chart's "harbor.labels" helper). The trivy PVC is a StatefulSet volumeClaimTemplate labeled via "harbor.legacy.labels" with a selector using "harbor.matchLabels" — neither sets app.kubernetes.io/instance — so the sweep silently missed it. Trivy is enabled by default, so a default Harbor delete left the trivy PVC orphaned, contradicting the PR's own claim. Select on release=<release>-system instead: both kept PVCs carry it, while CNPG (db) and Redis PVCs use different label schemes and the registry is on S3, so only the two intended PVCs match. The unit test now asserts the corrected selector and guards against regressing to the instance label. Also drop the by-name delete of Orders/Challenges (their hash-suffixed names make a by-name delete a no-op; the Certificate owner-ref cascade plus the host-scoped sweep handle them), narrow the hook RBAC to challenges only, and note the Gateway-mode no-op. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
11c417e258 |
test(mariadb): assert both cleanup failure branches and exact RBAC rule set
Strengthen the cleanup hook suite per review: - assert the PVC and Secret delete branches each set cleanup_failed=1, not just the final exit, so a regression in either branch is caught; - pin the Role to exactly two rules so a broader unscoped secrets rule can't slip past the subset `contains` check. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
ae7c232409 |
fix(qdrant): anchor PVC ordinal so cleanup never deletes a sibling's data
The post-delete hook selected PVCs with an unanchored name prefix (qdrant-storage-<release>-). Because release names are qdrant-<app>, deleting release "qdrant-test" matched not only its own PVC (qdrant-storage-qdrant-test-0) but also a sibling "qdrant-test-2"'s PVC (qdrant-storage-qdrant-test-2-0) in the same namespace — irreversibly deleting the sibling's data (or wedging it in Terminating via pvc-protection). Anchor the ordinal with grep -E "...-[0-9]+$" so only the release's own data PVCs match. A release name that is a prefix of a sibling no longer collides. - cleanup-pvc.yaml: grep -E with anchored "-[0-9]+$". - tests: assert the anchored selector and add a notMatchRegex guard against regressing to the unanchored prefix. - e2e: gate the post-delete wait on a NotFound result specifically, so a transient API/auth error is not misread as "PVC reclaimed". Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
c2a8a3b41f |
fix(mariadb): fail cleanup hook on real delete errors
The post-delete cleanup hook used `kubectl delete ... --ignore-not-found || echo WARNING`, which swallowed genuine RBAC/API failures and let the Job succeed with orphaned password Secrets still present. Track delete failures and propagate them via a non-zero exit, while --ignore-not-found still tolerates already-absent resources. Add a helm-unittest assertion locking in the non-zero exit-on-failure behaviour. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
a349c753ad |
fix(kubernetes): make node group resources optional and prefer them over instanceType
A node group is sized either by instanceType or by explicit resources, but the generated values schema required resources on every node group. A node group with instanceType and no resources — or an empty resources dropped during typed-CR to HelmRelease translation — failed schema validation with "missing property 'resources'" even though sizing was fully specified. Mark resources optional in the node group definition and regenerate the schema, API types, README, and ApplicationDefinition. The chart template emits domain.cpu/memory only when resources is present, so a node group sized purely by instanceType now passes validation and installs. When a node group sets both instanceType and explicit resources, explicit resources take precedence: the instancetype is omitted from the rendered VM and the per-group instancetype lookup is skipped, so KubeVirt accepts the explicit domain.cpu/memory with no instancetype to conflict with. instanceType stays required and defaulted (no schema change), so the console's required instanceType dropdown keeps working — sending the default instanceType plus explicit resources simply sizes the VM by resources. This replaces an earlier render-time hard failure on the both-set case, which was unreachable to clear from the dashboard (the required dropdown cannot be emptied). Explicit resources are all-or-nothing alongside an instanceType: a partial spec (only cpu, or only memory) on a group that carries a non-empty instanceType would emit one domain override next to the instancetype — which KubeVirt rejects — so it now fails at render time with a clear error. Partial resources on a group without an instanceType (used to tune kubelet reservations) are unaffected. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
7c67aafed5 |
fix(kubernetes): run the Talos reconcile Job in the main install phase (#3145)
## What this PR does The tenant worker `MachineDeployment` `md0` intermittently never reaches `status.replicas=2`. CAPI blocks MachineSet creation until the `TalosConfigTemplate` that the deployment's `bootstrap.configRef` points to exists, and on failing installs that template is never created. ### Root cause — an ordering deadlock The `TalosConfigTemplate` is created only by the `talos-reconcile` **post-install Helm hook** (the chart can no longer render it at template time — three of its inputs are produced asynchronously after apply). A post-install hook runs only after Helm has applied the release's main resources and, when Helm waits, after they become Ready. But the same release also emits in-tenant addon HelmReleases — notably **cilium** — whose readiness needs tenant worker nodes, and those nodes can only be created once this hook has produced the `TalosConfigTemplate`. So when the main-resource wait is in effect, the install deadlocks: it waits for cilium to be Ready, cilium waits for worker nodes, the nodes wait for the `TalosConfigTemplate`, and the template waits for the post-install hook — which never runs, because the install never finishes the wait that precedes the hook phase. The install times out and `md0` never scales. The release sets `helm-install-disable-wait` precisely to skip that main-resource wait so the hooks run, but its effect is unreliable — which is what makes the failure bimodal: when the wait is skipped the hook runs and the template lands in ~46s; when it is not, the deadlock strands the install. A timestamped poller over a failing run (both `kubernetes-latest` and `kubernetes-previous`, 640 polls) captured: `TalosConfigTemplate` absent in 640/640 polls; both post-install hook Jobs (`talos-reconcile` and `bootstrap-token`) never created — the install never reached the post-install hook phase; all main resources applied (MachineDeployment, KubevirtMachineTemplate, MachineHealthCheck, WorkloadMonitor, child HelmReleases, Talos PKI/secrets) and the four hook inputs ready early and stable; the parent HelmRelease stuck at `Running 'install' action 20m` then looping into `upgrade`, with the cilium sub-release timing out on `cilium-operator` (which has no worker node to schedule on) — downstream of the missing template. ### The fix Render the `talos-reconcile` Job and its RBAC / ServiceAccount / egress `CiliumNetworkPolicy` as ordinary **main-phase** resources instead of post-install hooks. The Job is then created during the apply itself — not after the wait — and runs on the management cluster, which has nodes (unlike the tenant `cilium-operator`). It produces the `TalosConfigTemplate`, `md0` scales, workers join, cilium goes Ready, and the main-resource wait (if any) closes within the install budget: the Job completes in tens of seconds, and node boot plus cilium readiness is a few minutes — comfortably inside the 20m install timeout. This works whether or not `helm-install-disable-wait` takes effect, so it does not depend on that annotation's reliability. Because a main-phase Job is immutable, it carries a content-hash name suffix over its whole rendered spec (mirroring the `KubevirtMachineTemplate` idiom in `cluster.yaml`): an identical render keeps the same name and Helm no-ops it; any spec change yields a new name and a fresh Job, so no change ever lands on the same name and trips an immutable-field patch error. The applied `TalosConfigTemplate` keeps its stable `<release>-<group>` name (referenced by `bootstrap.configRef`), so an unchanged render re-applies a byte-identical template — a CAPI no-op that never rolls existing workers. The Job name is truncated to stay within the 63-char DNS-label limit while always preserving the hash. The runtime input-waits, the single `kubectl apply` of the template with a `KamajiControlPlane` ownerReference for GC, and the `extraHostEntries` are all unchanged. `helm-install-disable-wait`'s unreliable effectiveness is a real latent issue (it explains the created-vs-never bimodality), but this fix breaks the deadlock independent of it, so that is a separate follow-up rather than a blocker here. Supersedes #3139 (a 10-minute wait-budget bump that treated the symptom; the deadlock is structural and a larger budget would not have helped). ### Screenshots Not applicable — no UI changes. ### Testing helm-unittest extended to pin: no `helm.sh/hook` on any of the seven talos-reconcile documents, the content-hash name, `ttlSecondsAfterFinished`, the preserved template apply + ownerReference + extraHostEntries, the hash rotating on a spec change, and the long-name truncation staying within 63 chars. Full chart suite green (128 tests). The applied template is byte-identical to the previous hook output (verified by diff), so existing tenants are upgrade-safe. ### Release note ```release-note fix(kubernetes): create the tenant worker TalosConfigTemplate from a main-phase Job instead of a post-install hook, so worker MachineDeployments reliably scale during install ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes addon install behavior by running reconcile work during the main chart phase, reducing readiness deadlocks and upgrade issues. * Added more reliable reconciliation handling with stable, change-detected job recreation and updated network access for the reconcile workload. * Refined chart and test guidance to match the new install flow and Helm behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
50748fed98 |
fix(kubernetes): make the default md0 node group removable (#2936)
Same-repo mirror of #2753 by @mattia-eleuteri, opened so the `Build` CI job can run. Fork PRs skip the OCIR registry login (`if: !github.event.pull_request.head.repo.fork` in `pull-requests.yaml`) and so can't push the per-PR images the build needs — meaning a fork PR can never pass `Build`, regardless of the change. The commits here are unchanged and authored by @mattia-eleuteri; full description and discussion in #2753. This includes migration **44** (renumbered from 43 after the seaweedfs-db migration landed on `main`) and bumps `targetVersion` to **45**. Supersedes #2753. ```release-note fix(kubernetes): the default `md0` node group is now applied only when no node groups are configured, so it can be removed or replaced. Migration 44 pins `md0` explicitly on existing clusters to preserve their topology. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Empty `nodeGroups` settings now automatically provision a default `md0` worker group. * Existing environments are updated to keep this default group available during upgrades. * **Bug Fixes** * User-defined node groups now fully replace the default instead of being merged with it. * Improved handling of node-group resources and permissions in Kubernetes app rendering. * **Documentation** * Updated Kubernetes app docs and schema details to reflect the new default behavior. * **Tests** * Added coverage for default and custom node-group configurations, plus migration behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
81198b294f |
fix(bucket): clean up orphaned cert-manager ACME solver resources on delete (#3060) (#3076)
## What this PR does After deleting a `Bucket` app CR, cert-manager's temporary ACME HTTP-01 solver resources (`cm-acme-http-solver-*` Ingress/Pod/Service) were left orphaned in the tenant namespace. They are created by cert-manager to answer an ACME challenge and are owned by the `Challenge`/`Order` → `Certificate`, not by the app Ingress. If a challenge gets stuck and the app is deleted, the parent Ingress is removed but the orphaned challenge leaves the solver objects with no live owner to cascade-delete them, so Helm/Flux uninstall cannot reap them. This adds a `post-delete` cleanup hook that deletes the release's `Certificate`/`Order`/`Challenge` (cert-manager then cascade-removes the solver objects) and then sweeps any leftover solver objects by the reliable `acme.cert-manager.io/http01-solver=true` label. RBAC is verified against the vendored cert-manager CRDs (`cert-manager.io` → certificates; `acme.cert-manager.io` → orders, challenges) plus core `services`/`pods` and `networking.k8s.io` ingresses. The cleanup Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`. Failures are logged but never block teardown. The preventive wildcard-cert guard the issue mentions already exists in the vendored `bucket-system` Ingress template (it drops the ACME annotations in wildcard mode), so no preventive change is needed — only cleanup. No user data is affected: only cert-manager's transient challenge artifacts are removed. Fixes #3060. ### Release note ```release-note fix(bucket): clean up orphaned cert-manager ACME http01-solver resources (Ingress/Pod/Service) left behind after a Bucket app is deleted. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a bucket app `test` command that runs Helm unit tests. * Introduced a post-delete ACME cleanup hook to remove release-scoped cert-manager TLS and HTTP-01 solver resources, using dedicated least-privilege access controls. * **Bug Fixes** * Improved cleanup safety with bounded execution and hardened container security. * **Tests** * Added a Helm test suite covering rendered hook documents, including host-scoped targeting and negative assertions to prevent overly broad deletions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
544ba5bf16 |
fix(harbor): preserve jobservice and trivy storageClass on upgrade (#2930)
## What this PR does The `apps/harbor` chart never passed `storageClass` to the jobservice `jobLog` PVC, so the upstream chart rendered it without a `storageClassName`. On first install the default-StorageClass admission then stamps the cluster default onto the live PVC. Because a PVC's `storageClassName` is immutable, the rendered manifest (no class) and the live object (admission-defaulted class) drift apart, which surfaced as upgrade failures on the `harbor-*-system` HelmRelease. This propagates `storageClass` to the jobservice PVC, with one safeguard: a `lookup` reads the live jobservice PVC and reuses its current `storageClassName` when the PVC already exists, only falling back to the configured `storageClass` for a fresh install. Keeping the rendered value equal to the live one means an existing PVC whose admission-defaulted class differs from the configured `storageClass` is preserved rather than triggering an immutable-field rejection on upgrade. A live PVC pinned to an explicit `""` (no dynamic provisioning) round-trips through the upstream `"-"` sentinel instead of collapsing to an omitted field, and the spec access is guarded so a PVC with no `storageClassName` key stays omitted. The trivy `StatefulSet`'s `volumeClaimTemplate` carried the same latent drift — `.Values.storageClass` was forwarded blindly into a VCT, which cannot be patched at all — so it now gets the identical lookup-preserve treatment, keyed off `data-<release>-trivy-0`. This closes both halves of #2368. Also adds a helm-unittest regression suite and wires a `test:` target so `make unit-tests` stops skipping `packages/apps/harbor`. Validated with `helm template` across storageClass/trivy permutations and the new unit tests. The jobservice preserve path was additionally confirmed with a server-side dry-run against a running harbor release: `lookup` resolved the live PVC's class, and with `storageClass` overridden to a different class the jobservice render stayed pinned to the live PVC's class — confirming the immutable-field rejection is avoided. The trivy change applies the same proven pattern. Note: under `helm template`/helm-unittest `lookup` returns empty, so the unit suite exercises only the fresh-install fallback; the preserve branches are covered by the live `lookup`. Fixes #2368 ### Release note ```release-note fix(harbor): preserve the jobservice PVC and trivy volumeClaimTemplate storageClass on upgrade, fixing upgrade failures caused by storageClassName drift ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Harbor now preserves the existing jobservice storage class during upgrades and ensures it’s applied on initial installation when available, with consistent handling for Trivy persistence as well. * **Tests** * Added a Helm unittest regression suite covering storage class propagation and the correct conditional rendering of persistence blocks when storage class and Trivy settings vary. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
c818ece25b |
test(kubernetes): cover md0 removability with user-defined nodeGroups
The existing case pins the empty-nodeGroups -> default md0 path (the helper's else-branch). This adds the complementary if-branch: when a user supplies their own nodeGroups, they are authoritative and the built-in md0 is NOT merged in, so md0 is removable. The assertion selects every rendered MachineDeployment and requires it to be the user's own group, so a regressed md0 leak surfaces as a second MachineDeployment named <release>-md0 that fails the assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
ab55ff1924 |
test(kubernetes): cover the default md0 node group render
Assert that an empty nodeGroups renders the built-in md0 group via the kubernetes.nodeGroups helper else-branch (the behaviour that makes md0 removable): the default md0 MachineDeployment is emitted and the default 20Gi system disk flows through. A minimal VirtualMachineClusterInstancetype plus dummy CAPI objects are registered so the per-group instanceType lookup and the old-template preservation LISTs resolve under helm-unittest. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
7e3c5f99db |
fix(kubernetes): apply default md0 node group only when nodeGroups is empty
The chart shipped a default nodeGroups.md0 in values.yaml. Helm deep-merges
values, so the default md0 was re-added on top of any user-supplied
nodeGroups; combined with Kubernetes stripping null values, md0 could never
be removed by configuration. Move the default into the template
("kubernetes.nodeGroups" helper) so it is applied only when nodeGroups is
empty. User-supplied node groups now fully replace the default, letting users
name and omit groups (including md0) freely.
Test fixtures that relied on the chart-default md0 to supply diskSize via
values merge now set it explicitly.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
|
||
|
|
32c4d60b43 |
fix(kubernetes): run the Talos reconcile Job in the main install phase
The talos-reconcile Job is the only thing that creates the TalosConfigTemplate the worker MachineDeployment's bootstrap.configRef points to; CAPI blocks MachineSet creation until it exists, so tenant workers cannot scale until the Job has run. The Job was a post-install/post-upgrade Helm hook, which runs only after Helm has applied the release's main resources and, when Helm waits, after they become Ready. But the same release emits in-tenant addon HelmReleases — notably cilium — whose readiness needs worker nodes, and those nodes need the TalosConfigTemplate this Job produces. So when the main-resource wait is in effect the install deadlocks: it waits for cilium to be Ready, cilium waits for workers, the workers wait for the template, and the template waits for this hook — which never runs because the install never finishes the wait that precedes the hook phase. The install times out and the MachineDeployment never scales. The release sets helm-install-disable-wait to skip that wait, but its effect is unreliable, which makes the failure bimodal: the template lands in ~46s when the wait is skipped, and the install is stranded when it is not. Render the Job and its RBAC/ServiceAccount/egress policy as ordinary main-phase resources. The Job is then created during the apply itself, not after the wait, and runs on the management cluster where nodes exist (unlike the tenant cilium-operator), so it produces the template, the MachineDeployment scales, workers join, cilium goes Ready, and the wait (if any) closes within the install budget — independent of whether helm-install-disable-wait takes effect. The Job is immutable, so it carries a content-hash name suffix over its rendered spec (mirrors the KubevirtMachineTemplate idiom in cluster.yaml): an identical render keeps the name and Helm no-ops it, while any spec change yields a new name and a fresh Job, so no spec change ever lands on the same name and triggers an immutable-field patch error. The applied TalosConfigTemplate keeps its stable <release>-<group> name, so an unchanged render re-applies a byte-identical template — a CAPI no-op that never rolls existing workers. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
2587458bd4 |
fix(qdrant)!: reclaim data PVC by name prefix, not volumeClaimTemplate label
The previous approach routed a release-unique label through qdrant.persistence.additionalLabels, which the upstream chart renders into the StatefulSet's volumeClaimTemplates[].metadata.labels. That field is immutable on an existing StatefulSet, so for any already-deployed Qdrant the API server would reject the upgrade and the <release>-system HelmRelease (retries: -1) would retry-fail forever — permanently wedging reconciliation. Label changes on a volumeClaimTemplate also never apply to PVCs that already exist, so the cleanup hook could not reclaim those instances anyway. Switch the post-delete hook to select PVCs by the deterministic StatefulSet naming prefix (qdrant-storage-<release>-) instead. PVC names derive purely from the release name, so this needs no StatefulSet change (upgrade-safe) and also reclaims PVCs from instances created before the hook existed. - qdrant.yaml: drop the additionalLabels passthrough entirely. - hooks/cleanup-pvc.yaml: delete by name prefix via kubectl get -o name | grep. - tests: assert the name-prefix command instead of the label passthrough. - e2e: assert qdrant-storage-<name>-0 is gone after delete (async-waited). BREAKING CHANGE: deleting a Qdrant app now permanently removes its data PVC and the vector data it holds. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
6a43adbd09 |
fix(bucket): drop no-op label-based order/challenge delete
Per maintainer review: the label selector cert-manager.io/certificate-name=<release>-ui-tls matches nothing — Challenge objects carry no labels, and on Orders certificate-name is an annotation (not a label). Orders are already reaped by the Certificate owner cascade and Challenges by the host-scoped .spec.dnsName sweep, so the line deleted nothing and its comment misstated cert-manager labeling. Remove the dead delete, replace the inaccurate comment, and drop the test assertion that pinned it; add negative assertions so a label-based order/challenge delete cannot be reintroduced. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
|
4db565282b |
fix(postgres): garbage-collect init-job after completion (#3055) (#3069)
## Problem After deleting a `Postgres` app CR, the chart-managed init Job `<release>-init-job` and its completed Pod remain in the tenant namespace indefinitely (reported in #3055). The init Job is a Helm hook (`post-install,post-upgrade`) whose only delete policy is `before-hook-creation`. That policy deletes the previous hook only right before a new one is created (next upgrade) — it never fires on `helm uninstall`. With no `hook-succeeded` and no TTL, the completed Job and Pod leak. ## Root cause `packages/apps/postgres/templates/init-job.yaml` — the Job has no garbage-collection mechanism for its completed state. ## Fix Add `ttlSecondsAfterFinished: 120` to the Job spec. The native Kubernetes TTL controller then GCs the Job and its Pod ~2 minutes after completion, regardless of the Helm hook lifecycle. The init Job is a one-shot, idempotent reconciler of users/databases/roles (`init-script.yaml`) — once it has run it is not needed until the next upgrade (when Helm recreates it), so GC-ing the finished Job is safe. This approach deliberately **does not** touch the hook-delete-policy / hook-timeout semantics, which the `bootstrap.recovery` path depends on (see the header comment in `init-job.yaml`). Under `bootstrap.enabled` the Job is skipped entirely, so the change is inert there. ## Tests Added a `helm-unittest` suite `packages/apps/postgres/tests/init_job_cleanup_test.yaml`: - asserts the init Job renders `spec.ttlSecondsAfterFinished: 120` - asserts the init Job is skipped under `bootstrap.enabled: true` Verified locally with `helm template` (TTL present on default render; Job absent under bootstrap recovery). ## Test plan - [x] `helm template` renders `ttlSecondsAfterFinished: 120` on the init Job - [ ] `helm unittest packages/apps/postgres` green in CI - [ ] e2e `test-apps-postgres` green Fixes #3055 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Postgres bootstrap init Jobs now automatically clean up after completion using a post-finish TTL. * **Tests** * Added a new integration test suite to verify the init Job renders with the expected cleanup TTL, and that the init Job is skipped when bootstrap is enabled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
49555e562b |
fix(openbao): garbage-collect the data PVC on delete (#3058) (#3073)
## What this PR does
After deleting an `OpenBAO` app CR, the StatefulSet data PVC
(`data-<release>-0`)
was left orphaned in the tenant namespace, consuming storage
permanently. The
OpenBAO app renders a Flux `HelmRelease` over the vendored upstream
OpenBAO chart,
and StatefulSet-templated PVCs are never garbage-collected on
StatefulSet deletion
by default — the chart set no `persistentVolumeClaimRetentionPolicy`.
This passes a retention policy through the HelmRelease values. The
upstream chart
renders it onto the StatefulSet from
`server.persistentVolumeClaimRetentionPolicy`
(verified in the vendored `server-statefulset.yaml`) — **not** from
`server.dataStorage.*`:
```yaml
server:
persistentVolumeClaimRetentionPolicy:
whenDeleted: Delete
whenScaled: Retain
```
A `test` target is added to the package Makefile so the new
helm-unittest suite
(asserting the passthrough) actually runs in CI —
`hack/helm-unit-tests.sh` only
runs packages that define a `test` target.
> [!WARNING]
> **Deleting an OpenBAO app now permanently removes its data PVC and the
data it holds.**
> This is intentional and fixes the storage leak. `whenScaled: Retain`
is kept on
> purpose so scaling an HA/raft cluster down does **not** destroy a
member's data —
> only full deletion of the release reclaims storage. Back up before
deleting.
Fixes #3058.
### Release note
```release-note
fix(openbao): set persistentVolumeClaimRetentionPolicy so the data PVC is removed when an OpenBAO app is deleted, instead of being orphaned. WARNING: deleting an OpenBAO app now permanently removes its data PVC and the data it holds (scale-down still retains data).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable PVC retention behavior for OpenBao: storage is
deleted when the release is removed and retained when scaling down.
* **Tests**
* Added a Helm unit testing workflow to run chart/unit tests.
* Introduced assertions to confirm PVC retention settings are correctly
passed through and rendered as expected.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
959a9b1fdb |
fix(kubernetes): clear kamaji datastore-secret finalizer to unblock namespace deletion (#3062) (#3078)
## What this PR does After deleting a `Kubernetes` app CR / tenant, the secret `<release>-datastore-config` kept the finalizer `finalizer.kamaji.clastix.io/datastore-secret`, which was never removed — **blocking deletion of the entire namespace** (observed stuck in `Terminating` for 115 minutes). Kamaji adds this finalizer to the datastore connection Secret and normally strips it as part of TenantControlPlane (TCP) deletion; if the TCP is force-deleted or Kamaji does not reconcile in time, the finalizer hangs and the Secret can never be garbage-collected, wedging the namespace. This extends the existing `pre-delete` cleanup hook (`packages/apps/kubernetes/templates/delete.yaml`): - **Step 4** now deletes the TenantControlPlane with `--wait=true --timeout=120s` so Kamaji's own finalizer-cleanup chain has a chance to run before we fall through. - **Step 4b (backstop)** patches `metadata.finalizers: null` on `<release>-datastore-config` only if it still exists. By this point the TCP is gone, the Secret is owner-ref'd to the deleted TCP and only awaiting GC, so clearing the finalizer merely lets GC reap it — it cannot strip a finalizer while a live TCP still needs the datastore. The Secret holds only connection config, not tenant data (which lives in the external etcd datastore), so no data is lost. - **RBAC** adds a `secrets` rule scoped via `resourceNames` to exactly `<release>-datastore-config`, verbs `get`/`patch` only. The Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`, writable `/tmp` for kubectl cache. Failures at both steps are logged (the Step 4b warning explicitly mentions the namespace may stay Terminating) but the hook still completes so it cannot itself block teardown. This is the most impactful of the create→delete leak sweep findings: without it, tenant teardown hangs and the create→delete cycle requires manual intervention. Fixes #3062. ### Release note ```release-note fix(kubernetes): clear the leftover Kamaji `finalizer.kamaji.clastix.io/datastore-secret` finalizer on the datastore-config Secret during tenant teardown, so deleting a Kubernetes app / tenant no longer leaves the namespace stuck in Terminating. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Extended the pre-delete cleanup workflow to more reliably remove `TenantControlPlane` and handle slow deletions with a bounded wait. * **Security** * Hardened the cleanup job by running as non-root with restricted security settings, dropped capabilities, and a safe writable temp location. * **Bug Fixes** * Added a conditional backstop to remove leftover `finalizers` from the relevant datastore Secret so namespace termination isn’t blocked. * **Tests** * Added a YAML test suite to validate the wait-and-patch behavior and ensure the patch runs only after the `TenantControlPlane` is gone. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
bdf520bf74 |
fix(harbor): preserve jobservice and trivy storageClass on upgrade
The cozy-harbor wrapper never passed storageClass to the jobservice jobLog PVC, so it was created without a storageClassName and the default-StorageClass admission stamped the cluster default. Because PVC storageClassName is immutable, this caused rendered-vs-live drift on upgrade. Preserve an existing PVC's class via lookup and fall back to the configured storageClass for fresh installs. The trivy StatefulSet's volumeClaimTemplate had the same latent drift -- .Values.storageClass was forwarded blindly into a VCT that cannot be patched at all -- so it gets the same lookup-preserve treatment, keyed off data-<release>-trivy-0. A live object pinned to an explicit "" (no dynamic provisioning) now round-trips through upstream's "-" sentinel instead of collapsing to an omitted storageClassName, and the spec lookup is guarded with `| default dict` so a PVC carrying no storageClassName key stays omitted. Adds a helm-unittest regression suite and wires a test target so make unit-tests stops skipping packages/apps/harbor. Fixes #2368 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
243ce65071 |
feat(kubernetes): expose tenant apiserver authentication in app values
Tenant Kubernetes clusters had no way to configure the kube-apiserver authentication through the app values: no knob for OIDC, the structured AuthenticationConfiguration, extra apiserver flags, or extra volumes. The only workaround was patching the underlying KamajiControlPlane out-of-band, which is invisible to GitOps and lost on recreation. Add three opt-in fields under controlPlane.apiServer, all empty by default so existing clusters render unchanged: - extraArgs -> KamajiControlPlane spec.apiServer.extraArgs - extraVolumeMounts -> spec.apiServer.extraVolumeMounts - extraVolumes -> appended to spec.deployment.extraVolumes This unblocks OIDC SSO via the --oidc-* flags and, more importantly, structured authentication: --authentication-config mounted from a ConfigMap/Secret plus --requestheader-uid-headers, which is the only way to map a user UID for aggregated/extension apiservers. Because the control-plane Deployment runs on the management cluster, the passthrough is validated to preserve tenant<->management isolation: - extraVolumes allow only configMap and secret sources; host-reaching sources (hostPath, csi, nfs, persistentVolumeClaim) and token projection (projected) are rejected, so a tenant with self-service access to the Kubernetes CR cannot mount management-node paths or project arbitrary tokens into the apiserver pod. - each extraVolume must have a unique, non-empty name and exactly one source; the reserved names talos-ca and talos-tls-cert are rejected to avoid a duplicate volume name in the rendered podspec. - each extraVolumeMount must reference a user-declared extraVolume; the chart-managed talos secret volumes cannot be mounted. Regenerate values.schema.json, README, the Go types and deepcopy, and the ApplicationDefinition. Add helm unittest coverage for the default (no-op) render, each field's passthrough, the name/source guards, and the mount-reference guard (valid and rejected inputs). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
6423cb53e0 |
fix(kubernetes): never auto-select the kubevirt alias as tenant default SC
The legacy "kubevirt" StorageClass alias was inserted into the candidate map before the default was chosen, so the sortAlpha fallback could pick it when propagated class names sort after "kubevirt" (e.g. nvme, ssd) and no configured/replicated class is present. That made the tenant default point at an infraStorageClass that may not exist on the infra cluster, leaving default PVCs Pending. Choose the default from the propagated set before adding the alias. Also quote the templated StorageClass name and infraStorageClassName in kubevirt-csi-node, and document the propagation upgrade caveat, render-time evaluation limitation, and allowRemoteVolumeAccess assumption in the kubernetes README. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
6a1b56f2b5 |
test(kubernetes): cover LINSTOR StorageClass propagation
- csi_test.yaml: assert the fallback storageClasses map yields a default "replicated" plus the non-default "kubevirt" alias, and that the alias tracks the storageClass selector. - kubevirt-csi-node: add a Makefile test target (otherwise the package is silently skipped by the unit-test runner) and storageclasses_test covering per-entry rendering, the single-default invariant, and the absence of any enforcement allowList in driver-config. - run-kubernetes.sh: verify in e2e that "replicated" propagates as the default kubevirt-CSI class, the "kubevirt" alias is retained, and the node-local "local" infra class is filtered out. Refs: #2094 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
021f6141ca |
feat(kubernetes): propagate remote-accessible LINSTOR StorageClasses to tenant clusters
Tenant Kubernetes clusters previously exposed a single hardcoded "kubevirt" StorageClass mapped to one infra class. Discover all remote-accessible LINSTOR infra StorageClasses (provisioner linstor.csi.linbit.com, allowRemoteVolumeAccess != false) via Helm lookup and propagate each to the tenant under the same name. - csi.yaml builds a storageClasses map filtered to remote LINSTOR classes; node-local classes (e.g. "local") are excluded because the backing volume cannot follow a live-migrated worker VM. - The class named by .Values.storageClass (default "replicated") is marked the tenant default; the legacy "kubevirt" StorageClass is kept as a non-default alias so existing tenant PVCs keep binding. - A non-destructive fallback preserves the legacy mapping when the lookup returns empty (dry-run), so tenant StorageClasses are never wiped on a transient reconcile. - infraStorageClassEnforcement is intentionally left disabled (AllowAll) to avoid breaking tenants that already self-served custom StorageClasses under the current default. Refs: #2094, #1424 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |