Commit Graph
3871 Commits
Author SHA1 Message Date
Myasnikov Daniil 9aa8710ef2 feat(fluxcd): bump flux-operator/instance to v0.48.0, web UI opt-in
Bump flux-operator and flux-instance vendored charts from v0.33.0 to
v0.48.0, moving the Flux distribution from 2.7.x to 2.8.x. This brings
helm-controller v1.5.x, which exposes spec.install.healthCheckExprs on
HelmRelease — a prerequisite for adding real readiness gating to
dependency-target packages (postgres, mongodb, kafka, etc.).

The new flux-operator chart defaults web.enabled=true; override it to
false in packages/system/fluxcd-operator/values.yaml so the Flux Status
Web UI stays opt-in. Operators wanting it can set web.enabled=true and
configure web.config / web.configSecretName / web.ingress per the
upstream chart values.

Patches reapplied:
- kubernetesEnvs.diff: matched cleanly (offset shift only)
- networkPolicy.diff: extended to also restrict port 9080 (web UI port);
  hunk count updated 20 -> 22

Pre-flight: no references to the v0.39-removed --disable-wait-interruption
flag anywhere in packages/ or internal/.

Verified locally:
- helm template renders both packages without errors
- web.enabled=false (default): no web Deployment/Service/NetworkPolicy
- web.enabled=true: renders web NetworkPolicy on port 9080
- CiliumClusterwideNetworkPolicy lists 8080/8081/9080
- FluxInstance distribution.version: "2.8.x", all 7 components present
- Cozystack kustomize patches (concurrent, requeue, storage-adv-addr,
  events-addr) intact

Notable upstream changes between v0.33 and v0.48:
- Flux 2.8.x with HelmRelease inventory format change (auto-migrated)
- Reliability fixes for stuck FluxInstance/ResourceSet on health-check
  cancellation (v0.39, v0.46)
- New CLI: flux-operator migrate, diff, patch instance, distro mirror
- ResourceSetInputProvider: GitLab Environments, GitHub App, Gitea/Forgejo
- CVE-2026-23990 (Web UI OIDC impersonation bypass) — N/A while web is off

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-28 12:43:49 +05:00
myasnikovdaniil 8afb99b282 docs(release): add changelog for v1.4.2 (#2759)
This PR adds the changelog for release `v1.4.2`.

✅ Changelog has been automatically generated in
`docs/changelogs/v1.4.2.md`.
2026-05-28 11:35:15 +05:00
cozystack-ci[bot] 6270c728eb docs: add changelog for v1.4.2
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-05-28 01:52:14 +00:00
myasnikovdaniil 74de393e1a test(e2e): ouroboros fold + CSI/NFS ordering + OIDC keycloakInternalUrl (#2728)
## What this PR does

Bundles four independent e2e improvements lifted from #2619:

1. **`test(e2e): fold ouroboros assertions into kubernetes-latest`** —
moves the standalone `hack/e2e-apps/ouroboros.bats` (which provisioned
its own full Kamaji control plane, ~25m) into the existing
`kubernetes-latest` run as an optional addon. The assertions cover
ouroboros HR Ready, controller pod Running, `Ingress → coredns-custom`
rewrite-line injection, and end-to-end DNS resolution from inside the
tenant. Reuses the existing tenant cluster instead of building a second
one.

2. **`test(e2e): wait for tenant CSI HR Ready before NFS test`** —
explicit `kubectl wait hr/kubernetes-${test_name}-csi
--for=condition=ready` ahead of the NFS pod. Otherwise the NFS pod
schedules while the `kubevirt-csi-node` DaemonSet is still rolling out,
eats ~1m on `FailedAttachVolume` retries, and trips the pod-Succeeded
wait.

3. **`test(e2e): bump NFS pod wait to 10m to fit slow-path chain`** —
was 5m. With dependencies properly serialized (per #2), 5m was tight on
slow runners; 10m gives the chain genuine room without masking real
bugs.

4. **`test(e2e): set keycloakInternalUrl when enabling OIDC`** — the
dashboard gatekeeper (oauth2-proxy) does OIDC discovery against the
keycloak Service URL; without an explicit `keycloakInternalUrl`, it
falls back to the public ingress URL which isn't routable from inside
the cluster yet during install.

## Origin

All four commits lifted unchanged from #2619.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Integrated Ouroboros validations into the main Kubernetes end-to-end
flow (removed separate standalone Ouroboros test)
* Added hairpin-NAT and in-tenant DNS resolution checks for Ouroboros
scenarios
* Improved Keycloak OIDC test by configuring an in-cluster internal URL
* Increased various timeouts and readiness waits and added tenant CSI
readiness gating to improve test reliability

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2728?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-27 23:58:12 +05:00
myasnikovdaniil 6b76db6929 fix(platform): add cert-manager dependency to webhook-cert consumers (#2726)
## What this PR does

Adds `cert-manager` as a `dependsOn` entry to the five PackageSources
whose charts apply `cert-manager.io/v1` `Certificate` CRs and need the
webhook ready before their first reconcile. Without this, the helm
install renders Certificate manifests, the cert-manager webhook isn't
admitting requests yet (cold cluster), the apply fails, and
helm-controller retries until the webhook eventually comes up — burning
~2-3 minutes per affected package on cold installs.

The five PackageSources: `capi-operator`, `cozystack-engine`,
`ingress-nginx`, `kamaji`, and `rabbitmq-operator` (the platform
`cozy-system` set with webhook-cert consumers; not the apps layer).

In `cozystack-engine`, the dependency is added to both the `default` and
`oidc` variants — the `oidc` variant installs the same `cozystack-api`
and `lineage-controller-webhook` components that render Certificate
resources, so it had the same race.

## Origin

First commit lifted unchanged from #2619. Second commit (`oidc` variant)
added in response to review feedback.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **Chores**
* Updated package source dependencies across multiple components to
ensure proper installation sequencing.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2726?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-27 22:44:21 +05:00
Aleksei Sviridkin 78b9740094 feat(dashboard): add cluster-usage RBAC for the new admin page (#2743)
## What this PR does

Adds a new `ClusterRole` `cozystack-dashboard-cluster-usage` and its
`ClusterRoleBinding` under `packages/system/dashboard/templates/`, plus
the matching helm-unittest fixture. The role grants the
`cozystack-cluster-admin` group cluster-wide read on `nodes`, `pods`
(get/list/watch) and on `metrics.k8s.io` `nodes` (get/list).

Companion to the upcoming UI work in cozystack/cozystack-ui#15 — a new
admin page `Console → Administration → Cluster Usage` that renders
cluster-scoped resource utilisation and a per-node table including
arbitrary extended resources (GPUs and other accelerators). The
page-side sidebar item is gated by a `SelfSubjectAccessReview` on `nodes
list`; without this binding the gate fails closed and the entry stays
invisible, which is the desired fallback for clusters that have not
adopted the new RBAC yet.

The built-in `edit` `ClusterRole` already bound to
`cozystack-cluster-admin` via `cozystack-dashboard-cluster-admin` does
not grant cluster-wide pod read or any access to `nodes`, so an explicit
grant is required for the page to render anything beyond an error.

This chart intentionally does not add permissions for deeper sources
(Prometheus, DCGM, HAMi). Those belong in their respective packages or
in operator-provided RBAC; the page detects them via discovery and gates
each section with its own per-source review.

### Screenshots

No UI in this PR — RBAC only.

### Release note

```release-note
feat(dashboard): grant cluster-admin read access to nodes, pods and metrics.k8s.io for the new Cluster Usage admin page
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Dashboard cluster usage monitoring now includes configured role-based
access control with appropriate read permissions for nodes, pods, and
metrics data.

* **Tests**
* Added comprehensive test suite to validate the dashboard's role-based
access control configuration, including verification of proper
permission bindings and authorization rules.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2743?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-27 18:55:49 +03:00
Myasnikov Daniil ea7a8c8675 test(e2e): quote HelmRelease names in kubectl wait
Both \${test_name} and \${component} are literal-safe today (no
whitespace, no glob characters), so this is a style-only change — but
quoting variable expansions in kubectl arguments matches the rest of
the file and removes a future-foot-gun if a test name ever grows a
hyphen-y suffix that interacts oddly with completion or globbing.

Scoped to the two HelmRelease wait lines flagged by CodeRabbit and
echoed by the human reviewer; the machinedeployment wait on the
preceding line keeps its existing style.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-27 20:36:10 +05:00
Myasnikov Daniil 64c443caf1 test(e2e): split local k8s_version declaration from yq invocation
\`local k8s_version=\$(yq ...)\` masks the exit status of the command
substitution — the assignment to a local variable always succeeds, so
a failing yq is swallowed. Declare the local first, then assign, so
non-zero yq exits propagate via shell errexit semantics if the script
ever runs under \`set -e\`.

Pure shell hygiene; no behavioral change on the success path. Surfaced
by CodeRabbit and confirmed by the human reviewer as non-blocking but
worth applying.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-27 20:36:10 +05:00
Myasnikov Daniil 8c6e149177 test(e2e): delete standalone ouroboros.bats now folded into kubernetes-latest
The previous commit (3c81261b) folded the ouroboros hairpin-NAT
reconciliation assertions into the kubernetes-latest tenant under
run-kubernetes.sh, but did not remove the standalone bats file.

The e2e workflow auto-discovers every hack/e2e-apps/*.bats and dispatches
test-apps-<name> via the wildcard rule in packages/core/testing/Makefile.
Leaving ouroboros.bats in the tree meant the assertions ran twice: once
on the standalone ~25m Kamaji-backed tenant and again as the folded
addon on the kubernetes-latest tenant — the opposite of the consolidation
the fold was intended to deliver.

The folded assertions in hack/e2e-apps/run-kubernetes.sh cover the same
surface (HR Ready, controller pod Running, coredns-custom rewrite-line
injection, and end-to-end DNS resolution from inside the tenant), so
the standalone file is dead weight after the fold.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-27 20:36:09 +05:00
Myasnikov Daniil 3ecc030a64 test(e2e): set keycloakInternalUrl when enabling OIDC
The dashboard gatekeeper (oauth2-proxy) does OIDC discovery against the
external host at container startup. The e2e sandbox uses
host=example.org which does not resolve, so the auth-proxy crashloops:

  Get "https://keycloak.example.org/realms/cozy/.well-known/openid-configuration":
    dial tcp: lookup keycloak.example.org: no such host

Under Flux v2.8 kstatus the gatekeeper Deployment then flips to
'Failed' (rather than the pre-v2.8 'InProgress') and stalls the
cozy-dashboard/dashboard HelmRelease, which in turn blocks
cozy-fluxcd/flux-plunger via dependsOn. Previous runs masked this with
an unrelated ImagePullBackOff on cozy-dashboard-console; with the
console image fixed the OIDC discovery failure now surfaces.

packages/core/platform/values.yaml:194-198 already declares
authentication.oidc.keycloakInternalUrl. When set, the dashboard
chart (packages/system/dashboard/templates/gatekeeper.yaml:59-66)
renders --skip-oidc-discovery plus explicit redeem/jwks/validate/
backend-logout URLs pointing at the in-cluster keycloak Service.
Browser-facing redirect-url and login-url stay on the external host
(they are not exercised by the e2e flow).

The patch sets the value alongside oidc.enabled in the same
kubectl-patch so the configmap and gatekeeper rollout happen in one
reconcile pass.

Observed on CI run 26142986969 (PR #2619).

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5915a579ee)
2026-05-27 20:36:09 +05:00
Myasnikov Daniil 68a9a6e5ad test(e2e): bump NFS pod wait to 10m to fit slow-path chain
Previous run failed kubernetes-latest (and then kubernetes-previous via
cascade) on the 5m budget for nfs-test-pod to reach Succeeded. The
chain is host linstor prime PVC + CDI import + tenant CSI attach +
busybox image pull + containerd CreateContainer (which occasionally
stalls past kubelet's 2m CRI deadline and retries with name-reservation
conflict). ~40s on the happy path, 6m+ in observed worst case.

When kubernetes-latest times out, its tenant cluster (2x 20Gi worker
PVCs) leaks because set -e exits before any cleanup. kubernetes-
previous then runs, adds another 40Gi, and the host tenant-quota
(100Gi) is exhausted -- every downstream app test (mariadb, mongo,
postgres, qdrant, redis, openbao, vminstance) fails with
ErrCreatingPVCPrime / exceeded quota.

Bumping the wait to 10m gives the slow path room. Headroom costs
nothing on green runs since the pod is busybox echo.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit c5cc7b60b8)
2026-05-27 20:36:09 +05:00
Myasnikov Daniil 04a54b01cf test(e2e): wait for tenant CSI HR Ready before NFS test
kubernetes-previous fails on the NFS pod step. From cozyreport + job log:

  - Pod scheduled to tenant worker
  - 7x FailedAttachVolume over 35s: CSINode does not contain driver
    csi.kubevirt.io (kubevirt-csi-node DaemonSet still rolling out)
  - Attach succeeds at T+1m08s after the driver registers
  - busybox pull queued ~1m25s, completes at T+2m48s
  - CreateContainer in containerd then stalls past kubelet's CRI
    deadline; retry trips "failed to reserve container name ... is
    reserved for <prev>" -- the 5m pod-Succeeded wait then expires.

Root cause: run-kubernetes.sh creates the NFS PVC+Pod before asserting
kubernetes-${test_name}-csi is Ready. That HR lives in the parent
cluster but installs cozy-kubevirt-csi-node into the tenant via
kubeConfig.secretRef; helm-controller walks kstatus on the tenant
DaemonSet using the tenant kubeconfig. HR Ready in parent therefore
implies kubevirt-csi-node Pods Ready on every tenant worker, which
implies csi.kubevirt.io is published in CSINode. The existing wait at
line 354 runs AFTER the NFS test, so it does not gate the race.

Move the wait earlier with a 10m budget. Post-NFS sibling waits stay
as is, so component coverage is unchanged.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 394b1b2b9a)
2026-05-27 20:36:09 +05:00
Myasnikov Daniil 5035ac1414 test(e2e): fold ouroboros assertions into kubernetes-latest
The standalone hack/e2e-apps/ouroboros.bats provisioned a full Kamaji-
backed tenant Kubernetes cluster (control plane + 2 CAPI workers + base
addons) just to flip `addons.ouroboros.enabled=true` on top, then ran
~15m of hairpin-NAT reconciliation assertions. Cluster shape was
identical to what hack/e2e-apps/run-kubernetes.sh already provisions for
kubernetes-latest, so the second Kamaji bringup was pure duplication —
roughly 25-45m of CI wall-time per run.

Add an `enable_ouroboros` 4th parameter to `run_kubernetes_test()`. When
"true", inject the ouroboros addon block into the Kubernetes CR's
addons map and run the post-cluster assertions inline:

- Wait for the ouroboros HR Ready in the host namespace.
- Verify coredns-custom ConfigMap exists in tenant kube-system.
- Wait for the ouroboros controller pod Ready.
- Apply an Ingress with a TLS host in tenant default namespace.
- Poll coredns coredns-custom for the rewrite line (5m budget).
- Resolve the hairpin host from inside the tenant via a netshoot pod
  and assert it points at the ouroboros-proxy ClusterIP.
- Clean up dnscheck pod + Ingress.

kubernetes-latest passes "true" (gets ouroboros coverage); kubernetes-
previous leaves it empty (control-plane-only smoke remains).

(The standalone ouroboros.bats was removed in the preceding commit
alongside an unrelated foundationdb chart fix — the git rm staging had
already happened when the foundationdb commit landed.)

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit b5c9c2ac90)
2026-05-27 20:36:09 +05:00
Myasnikov Daniil 535f881160 fix(platform): include cert-manager in cozystack-engine oidc variant
The oidc variant installs system/cozystack-api and
system/lineage-controller-webhook, both of which render
cert-manager.io/v1 Certificate resources. Add cozystack.cert-manager to
the oidc variant's dependsOn so it matches the default variant and
webhook-cert consumers do not race the cert-manager controller on cold
installs.

Address review feedback from gemini-code-assist and coderabbitai on
packages/core/platform/sources/cozystack-engine.yaml.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-27 20:35:49 +05:00
Myasnikov Daniil 69cb7b5f18 fix(platform): add cert-manager dependency to webhook-cert consumers
Five PackageSources whose charts apply cert-manager Certificate CRs and
mount the resulting Secret on a webhook Deployment lacked an explicit
dependsOn: cozystack.cert-manager. Under Flux v2.8 helm-controller v1.5.0
kstatus, the dependent Deployment trips progressDeadlineSeconds (10m
default) when cert-manager controller isn't running yet, and the HR is
failed early. Repeatedly observed on PR #2619 with capi-operator,
kamaji, rabbitmq-operator, cozystack-api (cozystack-engine), and
ingress-nginx all cascading to InstallFailed at ~12-15m into a fresh
install.

The mariadb-operator, kubeovn-webhook, snapshot-controller, etcd-operator,
postgres-operator, mongodb-operator, victoria-metrics-operator, linstor,
and linstor-scheduler PackageSources already had the dependency, which
is why they weren't in the failure set.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 0fc0796b19)
2026-05-27 20:35:49 +05:00
myasnikovdaniil 628ebce653 feat(platform): add deletion-protection guardrail via ValidatingAdmissionPolicy (#2650)
## What this PR does

Adds a `ValidatingAdmissionPolicy` that blocks `DELETE` on critical
platform
objects labeled `platform.cozystack.io/no-delete=true`. Native,
in-process CEL
evaluation — no webhook DaemonSet, no Service, no TLS, no image to
maintain.
Requires Kubernetes 1.30+.

This is an alternative to the closed #2402 (admission webhook approach).

### Protected objects (labeled in this PR)

| Resource | Kind | Source |
|---|---|---|
| `cozy-system` | Namespace |
`packages/core/installer/templates/cozy-system-{namespace,labels}.yaml`
|
| `tenant-root` | Namespace |
`packages/system/cozystack-basics/templates/tenant-root.yaml` |
| `tenant-root` | HelmRelease (ns: tenant-root) | same |
| `cozystack-version` | ConfigMap (ns: cozy-system) |
`packages/core/platform/templates/cozystack-version.yaml` |
| `cozystack-packages` | OCIRepository (ns: cozy-system) |
`packages/core/platform/templates/repository.yaml` |
| `letsencrypt-prod`, `letsencrypt-stage`, `selfsigned-cluster-issuer` |
ClusterIssuer |
`packages/system/cert-manager-issuers/templates/cluster-issuers.yaml` |
| `linstorcluster` | LinstorCluster |
`packages/system/linstor/templates/cluster.yaml` |
| `packages.cozystack.io`, `packagesources.cozystack.io` | CRD | label
stamped in `internal/crdinstall/install.go` |

### Bypass

```bash
kubectl label <kind> <name> platform.cozystack.io/no-delete-
kubectl delete <kind> <name>
```

### Why VAP instead of a webhook

Same outcome as #2402 but with zero runtime infrastructure: no
DaemonSet, Service,
TLS certificate, or image. The kube-apiserver evaluates the CEL
`expression: "false"` in-process and denies because `validationActions:
[Deny]`.
`objectSelector` on the binding scopes evaluation to labeled objects
only — every
other DELETE in the cluster is unaffected.

### Release note

```release-note
feat(platform): add deletion-protection guardrail (ValidatingAdmissionPolicy)
blocking DELETE on cozystack platform objects labeled
`platform.cozystack.io/no-delete=true`. Bypass by removing the label first.
Requires Kubernetes 1.30+.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Deletion-protection guardrail: objects labeled
platform.cozystack.io/no-delete=true are denied DELETE requests; removal
of the label allows deletion.

* **Chores**
* Platform migration to v43 backfills the no-delete label onto system
resources and namespaces.

* **Tools**
* Added an "unprotect" teardown utility to remove the no-delete label
for uninstall/disaster recovery.

* **Documentation**
* Helm NOTES now warns when the guardrail is inactive due to missing
cluster API.

* **Tests**
  * New tests and e2e checks validate deletion-protection behavior.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2650?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-27 17:25:58 +05:00
Timofei Larkin 72c4eb9c08 feat(backups): implement Job backup strategy (#1721)
## What

Implements the `strategy.backups.cozystack.io/Job` backup strategy — a
generic, application-agnostic driver that runs a user-supplied
Kubernetes Job to create (and restore) a backup.

## Background

This PR began as a draft written against an earlier shape of the backups
API. The API has since moved to a `BackupClass`-based dispatch model,
which broke the original draft on three counts:

- `BackupJob.Spec.StrategyRef` and `BackupJob.Spec.StorageRef` were
removed; the strategy and its parameters are now resolved from a
`BackupClass` into a `ResolvedBackupConfig`.
- Strategy handling moved from standalone per-strategy reconcilers to a
dispatch model: `BackupJobReconciler.Reconcile` /
`RestoreJobReconciler.Reconcile` resolve the strategy and switch into
per-strategy methods (`reconcileVelero`, `reconcileCNPG`,
`reconcileAltinity`, …).
- The draft's template context (`.Values`, `.Storage`) diverged from the
convention the other strategies adopted, and its `.Storage` half
targeted a storage API that does not exist.

## What changed

- **`reconcileJob` / `reconcileJobRestore`**
(`internal/backupcontroller/jobstrategy_controller.go`): replaces the
standalone `BackupJobStrategyReconciler` with the two dispatch methods
the `BackupJob` / `RestoreJob` reconcilers call. The Job strategy is the
generic counterpart of the Altinity driver — same lifecycle (render
`PodTemplateSpec` → run a one-shot `batch/v1.Job` → poll its terminal
condition → emit a `Backup`), minus the ClickHouse-specific
`applicationRef` gate, so it runs for any application. Shared package
helpers (`renderJobTemplate`, `jobConditionState`,
`getApplicationUnstructured`, status helpers, deterministic Job naming,
controller refs) are reused.
- **Template context** follows the in-tree convention used by the other
drivers: `.Application` (live application object), `.Release.Name` /
`.Release.Namespace`, `.Mode` (`backup` / `restore`), `.Parameters`
(resolved `BackupClass` parameters), and `.Backup` (restore runs only).
Strategy parameters round-trip through the `Backup`'s `driverMetadata`
so a later restore re-renders against the same values.
- **Restore** re-renders the same template with `.Mode == "restore"`.
Cross-namespace restore is unsupported (the RestoreJob target reference
is namespace-local), matching the Altinity driver.
- **`JobSpec.Template` doc + regenerated CRD**: the field doc previously
advertised `.Values`; it now documents the real context, and
`strategy.backups.cozystack.io_jobs.yaml` is regenerated to match (the
change is limited to that field's description).
- **Tests** (`jobstrategy_controller_test.go`): backup-Job creation /
args / labels / status, completion → `Backup` artifact with parameter
round-trip, Job-failure surfacing, missing-strategy terminal failure,
restore into the target namespace, restore completion, and the
parameter-prefix unit. A deliberately generic application kind pins that
the driver never gates on `applicationRef.kind`.

No RBAC changes are required: the controller already holds
`strategy.backups.cozystack.io/*`, `batch/jobs`, `apps.cozystack.io/*`,
and `backups` permissions.

## Notes for reviewers

- Backup and restore share one template, so a strategy author's
`PodTemplateSpec` must branch on `.Mode` to do anything
restore-specific. This matches the existing convention; worth confirming
it's the intended contract for the Job strategy's docs/examples.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Implemented full backup/restore job strategy with deterministic Job
creation, idempotent execution, and status/condition updates; added demo
example scripts and reusable shell helpers for a NATS JetStream
backup/restore workflow.

* **Documentation**
* Clarified Job template rendering context and listed supported template
variables; added README and step-by-step scenario docs for the NATS
example.

* **Tests**
* Added comprehensive unit tests covering job creation,
completion/failure handling, parameter extraction, and unmappable-target
failure cases.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/1721?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-27 14:45:19 +04:00
Aleksei Sviridkin b630706df6 feat(apps): mark stateful-app storageClass fields as immutable (#2639)
## Summary

Annotate `storageClass` as immutable across every stateful chart that
exposes one: clickhouse, foundationdb, harbor, http-cache, kafka (Kafka
+ ZooKeeper), kubernetes (top-level + nodeGroups[]), mariadb, mongodb,
nats, openbao, opensearch, postgres, qdrant, rabbitmq, redis, vm-disk.
The PVC-pinning argument is identical for every chart — Kubernetes fixes
`storageClassName` on each PVC at creation, so editing this field on an
existing resource never migrated data; the schema annotation makes that
contract explicit and lets schema-aware consumers refuse the edit
up-front.

Schema-aware consumers today: the Cozystack UI
(cozystack/cozystack-ui#6) reads the rule from the chart's openAPISchema
and renders the field read-only on edit forms.

This PR is **schema-side only**. The cozystack aggregated apiserver does
not yet evaluate CEL rules in `openAPISchema` (`MariaDB` / `Postgres` /
etc. are application-backed, not CRDs); tracking issue
cozystack/cozystack#2657 covers wiring CEL into
`pkg/registry/apps/application/rest.go::Update`. Until that lands, an
apiserver-level `kubectl patch` that changes `storageClass` is still
accepted — but the underlying StatefulSet PVCs do not move, so there's
no data-corruption risk, only a confusing UX. The UI catches that case
today; the apiserver gate closes it permanently.

## What changes

For each app in the list above:

- `packages/apps/<app>/values.yaml` — gain a `## @immutable` directive
directly above (or below, for fields inside a `@typedef`) the
`storageClass` line.
- Regenerated via `make generate` in each package:
- `api/apps/v1alpha1/<app>/types.go` — kubebuilder `XValidation` marker
on the relevant field(s).
- `packages/apps/<app>/values.schema.json` — `x-kubernetes-validations:
[{rule: "self == oldSelf", message: …}]` on `storageClass`.
  - `packages/apps/<app>/README.md` — regenerated parameter table.
- `packages/system/<app>-rd/cozyrds/<app>.yaml` — the embedded
`openAPISchema` on the ApplicationDefinition reflects the change.

Refactor: `pkg/registry/apps/application/rest.go` factors the inline
schema-build path into a new `buildSpecSchema` helper so it can be
unit-tested without standing up a fake apiserver. Three regression tests
in `pkg/registry/apps/application/rest_spec_schema_test.go` pin the
contract that `x-kubernetes-validations` survives the
v1→internal→structural conversion (test loads the actual MariaDB
ApplicationDefinition fixture on disk, asserts the rule reaches the
structural schema). Empty-input and malformed-JSON paths are also
pinned.

No e2e admission test is added because the apiserver does not yet
enforce the rule — see cozystack/cozystack#2657.

No changelog entry: schema-side annotation is invisible at the apiserver
layer today (UI-only consumer) and the release-cut PR will add the entry
when admission enforcement lands.

## Dependencies

Requires cozystack/cozyvalues-gen#24 to be merged and released, and the
`cozyvalues-gen` binary pin in `.github/workflows/pre-commit.yml` line
50 bumped to the release that includes that PR. Pre-commit CI will fail
until both happen — the stock `v1.3.0` binary silently ignores `##
@immutable` and regenerates artefacts without the rule, producing a
drift against what's committed.

Once that ordering is satisfied, rebase this branch and the pre-commit
job will be green.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* storageClass is now treated as immutable after creation for many
stateful apps (enforced by chart/CRD schema validation).

* **Documentation**
* Added a storage immutability guide explaining the contract and
enforcement.
* Updated chart READMEs and values docs to mark storageClass as
immutable and clarified nodeGroup storage remains intentionally mutable.

* **Tests**
* Added tests validating schema parsing and preservation of immutability
rules.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2639?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-27 10:57:38 +03:00
myasnikovdaniil d3f282cee1 test(e2e): pre-pull platform images via staged-busybox DaemonSet (#2724)
## What this PR does

Pre-pulls every image referenced by `packages/system/kubeovn`,
`packages/system/linstor`, and `packages/system/cert-manager` onto every
node before those HelmReleases install. Cluster-member workloads (OVN
raft, LINSTOR, cert-manager webhook) fail when replicas start at
different times due to per-node image-pull stagger; pre-pulling means
all replicas start with images already cached.

## Mechanics

Renders each chart with `helm template`, then runs a `yq` filter that
walks every PodSpec-shaped object and emits the images of each container
— scoping the output to images the kubelet actually pulls (skips
configmap fields and CRD examples that happen to contain an `image:`
key). Each render is staged through a tmp file so a helm-template
failure trips `set -e` cleanly (cozytest.sh runs `@test` bodies under
`/bin/sh`, which is dash on Ubuntu CI — no `pipefail`).

The pre-pull DaemonSet itself is the hard part: distroless images
(cert-manager and friends) ship no shell and no `/bin/sleep`, so the
obvious `command: ["sleep", "infinity"]` fails with `exec: ENOENT`. The
script stages a statically-linked `busybox:musl` into a shared
`emptyDir` from an initContainer; every prepull container then execs
`/shared/sleep` regardless of what the image itself ships. The glibc
`busybox` tag is unusable — it needs `/lib64/ld-linux-x86-64.so.2` which
distroless lacks.

## Origin

Lifted from #2619 — squash of `test(e2e): pre-pull cert-manager images`
→ POSIX rewrite → cert-manager removal (failed on distroless) →
reintroduction with the busybox helper that solves the distroless
problem. Final state only — the four iterative commits aren't
review-friendly individually.

## Verification

- `sh -n hack/e2e-prepull-images.sh` clean
- All chart-referenced images held Ready under the DaemonSet on local
kind v1.33.1 single-node

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Improved end-to-end test reliability for platform image pre-pulling by
rendering charts separately and consolidating extracted images.
* Simplified and consolidated readiness waits during tenant
configuration for more robust application readiness checks.

* **Chores**
* Enhanced image pre-pulling to reliably handle distroless images using
a staged helper binary and shared volume approach.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2724?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-27 02:55:56 +05:00
Aleksei Sviridkin 5af5962edf feat(apps): mark stateful-app storageClass fields as immutable
Annotate storageClass as immutable across every stateful app whose
chart exposes one — clickhouse, foundationdb, harbor, http-cache,
kafka (Kafka + ZooKeeper), kubernetes (top-level only), mariadb,
mongodb, nats, openbao, opensearch, postgres, qdrant, rabbitmq,
redis, vm-disk. Without coverage across the catalogue the UI would
render storageClass read-only on some apps and editable on others —
a worse mental model than uniform behaviour.

The annotation lands at the schema layer (CEL XValidation rule
'self == oldSelf' on values.schema.json / types.go / embedded
openAPISchema in cozyrds/<app>.yaml). The cozystack aggregated
apiserver does not yet evaluate CEL on application-backed resources;
that is tracked at cozystack/cozystack#2657. The UI
(cozystack/cozystack-ui#6) consumes the marker today.

The PVC-pinning argument is identical for every chart: Kubernetes
fixes a PVC's storageClassName at creation time, so editing this
field on an existing resource never migrated data. The schema
annotation makes that contract explicit.

kubernetes.nodeGroups[name].storageClass is intentionally NOT
annotated: the field is optional and undefaulted, so a strict
'self == oldSelf' rule would block any future attempt to set it on
an existing node group. A richer ratcheting form is needed there —
see docs/storage-immutability.md.

A refactor in pkg/registry/apps/application/rest.go extracts the
inline schema-build path into buildSpecSchema so it can be
unit-tested. Three regression tests in rest_spec_schema_test.go pin
the contract that x-kubernetes-validations survives the
v1->internal->structural conversion (loaded from the actual MariaDB
ApplicationDefinition fixture on disk, not a hand-edited mock).

Each per-app README gains a one-line blockquote pointing to a new
docs/storage-immutability.md that explains the contract and which
consumers enforce it today.

Requires cozystack/cozyvalues-gen with @immutable support — bump the
pre-commit pin in .github/workflows/pre-commit.yml once the release
exists.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-26 23:51:11 +03:00
Myasnikov DaniilandClaude Opus 4.7 12ac2adec9 chore: retrigger CI
Empty commit to fire the Pull Request workflow, which never ran for
this commit because GitHub Actions was down during the original push.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-27 01:24:19 +05:00
Myasnikov DaniilandClaude Opus 4.7 a58c0d193b chore: retrigger CI
Empty commit to fire the Pull Request workflow, which never ran for
3bac5cac because GitHub Actions was down during the original push.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-27 01:07:23 +05:00
Andrei Kvapil 7e92265823 chore(img): refresh Cozystack logo and add icon/stacked variants (#2746)
## Summary

- Replaces `img/cozystack-logo-black.svg` and
`img/cozystack-logo-white.svg` with the updated Cozystack brand mark
(icon + "COZYSTACK" wordmark).
- Adds new layouts so the logo can be used on more surfaces without
re-cropping:
- `cozystack-logo-stacked-{black,white}.svg` — icon stacked over the
wordmark
  - `cozystack-icon-{black,white}.svg` — icon-only mark
- All SVGs are exported on a transparent background, so the same file
works on light and dark surfaces.

## Files

| Path | State | Notes |
|---|---|---|
| `img/cozystack-logo-black.svg` | replaced | horizontal wordmark, black
fill, transparent bg |
| `img/cozystack-logo-white.svg` | replaced | horizontal wordmark, white
fill, transparent bg |
| `img/cozystack-logo-stacked-black.svg` | new | icon over wordmark,
black fill, transparent bg |
| `img/cozystack-logo-stacked-white.svg` | new | icon over wordmark,
white fill, transparent bg |
| `img/cozystack-icon-black.svg` | new | icon only, black fill,
transparent bg |
| `img/cozystack-icon-white.svg` | new | icon only, white fill,
transparent bg |

## Test plan

- [ ] Verify rendering of `cozystack-logo-{black,white}.svg` in the
README on both light and dark GitHub themes.
- [ ] Confirm no other in-repo references to `img/cozystack-logo-*.svg`
need width/height adjustments (existing aspect ratio ~2.8:1 → new ~3:1,
very similar).
2026-05-26 21:22:36 +02:00
Viktoriia Kvapil 6451373251 chore(img): refresh Cozystack logo and add icon/stacked variants
Replace the old "COZYΞTACK" wordmark with the updated brand mark
(icon + "COZYSTACK" wordmark) and add additional layouts:

  - cozystack-logo-{black,white}.svg          horizontal wordmark (replaced)
  - cozystack-logo-stacked-{black,white}.svg  icon over wordmark (new)
  - cozystack-icon-{black,white}.svg          icon only (new)

All variants use a transparent background so they work on both light and
dark surfaces. SVGs are optimized with https://autocrop.cncf.io while
keeping the original padding.

Signed-off-by: Viktoriia Kvapil <bisunviktoria@gmail.com>
2026-05-26 20:54:57 +02:00
Timofei Larkin 727343d259 docs(backups): add NATS Job-strategy example
Add examples/backups/nats/, the first worked example of the generic Job
backup strategy. It backs up a NATS JetStream stream that has no dedicated
driver, using a stock natsio/nats-box image: 'nats stream backup' -> tar ->
curl --aws-sigv4 PUT to the COSI/seaweedfs S3 endpoint, and the reverse for
restore. One PodTemplateSpec serves both directions, branching at runtime on
{{ .Mode }} (the engine templates each string leaf independently and cannot
restructure the Pod per mode).

Mirrors the clickhouse example's structure: numbered scripts (bucket -> NATS
src + seeded stream -> strategy -> BackupClass -> BackupJob -> in-place and
to-copy RestoreJob), shared helpers, run-all.sh, README, and scenario docs.
Like every examples/backups/* dir it is a runnable demo, not CI-gated.

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-05-26 18:06:37 +03:00
Timofei Larkin 5029af6d83 test(backups): harden Job strategy tests
Address review feedback on the Job backup strategy:

- Assert .Application is templated: the rendered backup Job now reads
  .Application.spec.replicas, the load-bearing context key that lets a
  generic backup tool read live app spec (previously only .Release,
  .Mode, .Parameters and .Backup had coverage).
- Pin failure-path messages so a reordered error path can't pass on the
  wrong branch: the missing-strategy test asserts the strategy name, and
  the unmappable-kind tests assert "kind not registered".
- Document why the full Status().Update in the success branches is safe
  (the object is freshly Get() each pass; the StartedAt write requeues).
- Note the fixed batch/v1.Job backoffLimit=2 and link the follow-up to
  make it configurable (#2742).

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-05-26 17:25:42 +03:00
Aleksei Sviridkin adddd4f6e1 feat(dashboard): add cluster-usage RBAC for the new admin page
The cluster-usage admin page in the dashboard reads cluster-scoped
resources (nodes, pods, metrics.k8s.io) to render aggregate utilisation
and a per-node table that includes arbitrary extended resources such as
GPUs. The built-in 'edit' ClusterRole already bound to
cozystack-cluster-admin via cozystack-dashboard-cluster-admin does not
grant cluster-wide pod read or any access to 'nodes', so the dashboard
needs an explicit grant for the page to render anything beyond an error.

The page-side sidebar item is gated by SelfSubjectAccessReview on
'nodes list' — without this binding the gate fails closed and the entry
stays invisible, which is the desired fallback for clusters that have
not adopted the new RBAC yet.

The chart intentionally does not add permissions for deeper sources
(Prometheus, DCGM, HAMi). Those belong in their respective packages or
operator-provided RBAC; the page detects them via discovery and gates
each section with its own per-source review.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-26 17:25:31 +03:00
Timofei LarkinandClaude Opus 4.7 3bac5caac1 fix(platform): remove duplicate test target in Makefile
The deletion-protection commit added a second, identical 'test:' recipe
to the platform Makefile, causing GNU Make to print 'overriding recipe'
warnings on every 'make test' (and thus in helm-unit-tests CI output).
Drop the duplicate; keep the original target.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-05-26 14:35:20 +03:00
Myasnikov Daniil 2ace581117 test(e2e): stage prepull image list to avoid yq pipefail trap
Without pipefail (dash-CI), `yq | hack/e2e-prepull-images.sh` would let a
yq failure produce empty input; prepull-images.sh exits 0 on no-input
and the test passes with pre-pull silently skipped. Stage the yq output
to a tmp file and feed it via stdin so set -e trips on a yq failure.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 16:00:30 +05:00
myasnikovdaniil 2fa15ce044 fix: startupProbes for kubevirt-cdi-operator and grafana-operator (#2725)
## What this PR does

Adds `startupProbe` to two operator deployments whose readiness probes
fail repeatedly during a cold cluster install, generating noise and (for
CDI) blocking dependants on slow leader-election:

- **`kubevirt-cdi-operator`** — leader-election against a slow
`kube-apiserver` (Kamaji control plane bringup) consistently takes >30s
on a cold install. The default readinessProbe with 1s `periodSeconds` /
3 `failureThreshold` fires the first time election times out, the
operator pod gets marked NotReady, and dependants block on it.
- **`grafana-operator`** — bootstrap CRD discovery + Helm-managed
Dashboard reconcile loop spikes startup duration past readinessProbe's
default budget under load.

`startupProbe` gives each operator a longer one-time window to reach
steady state; the regular `readinessProbe` then takes over with its
tight per-second cadence to detect genuine ongoing failures.

## Origin

Both commits lifted unchanged from #2619.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Configuration Updates**
  * Added startup health checks to Grafana operator container
  * Added startup health checks to KubeVirt CDI operator container

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2725?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-26 15:47:56 +05:00
Aleksei Sviridkin edb3eeb011 fix(info): use root-host for Keycloak OIDC issuer URL in tenant kubeconfig (#2704)
## What this PR does

The kubeconfig generated by the `info` package for non-root tenants
pointed `--oidc-issuer-url` at `keycloak.<TENANT>.<ROOT-DOMAIN>`, but
Keycloak's ingress and cert live at `keycloak.<ROOT-DOMAIN>` (built from
`_cluster.root-host`). `kubectl oidc-login` then failed TLS verification
against the nginx-ingress default fake cert, so the auto-generated
kubeconfig was unusable for any non-root tenant.

`$host` was being sourced from `_namespace.host` (tenant-specific
subdomain) with a lookup-based override that tried to read
`tenant-root`'s `spec.values.host`. That override has been dead since
#1787 migrated `tenant-root` to `valuesFrom` (Secret-backed) —
`spec.values` is now always empty, so the override never fires.

Source `$host` from `_cluster.root-host` directly, matching what
`packages/system/dashboard/templates/gatekeeper.yaml` already does for
the same Keycloak realm.

Verified with `helm template tenant1 packages/extra/info -n
tenant-tenant1 --set-string _namespace.host=tenant1.example.org
--set-string _cluster.root-host=example.org ...` — issuer URL is now
`https://keycloak.example.org/realms/cozy` regardless of the
per-namespace host.

### Release note

```release-note
fix(info): use root-host for the Keycloak OIDC issuer URL in tenant kubeconfigs so `kubectl oidc-login` succeeds against the real Keycloak cert on non-root tenants
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated kubeconfig generation to consolidate host configuration
handling, now using a single authoritative source instead of multiple
fallback mechanisms.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2704?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-26 13:41:05 +03:00
Myasnikov Daniil 76fd5d1791 fix(platform): address lllamnyp review on PR #2650
unprotect.sh: add `gitrepository` to the namespaced-kind list so the
platform stays unprotectable when an operator configures
`sourceRef.kind: GitRepository`. Currently only `ocirepository` is
swept, while repository.yaml templates `kind: {{ $sourceRef.kind }}`
unrestricted.

run-migrations.sh: replace the silent skip on a missing migration file
with a hard error. All migrations 1..targetVersion exist in the shipped
image — a gap can only mean a packaging mistake (digest advanced
without the file landing), which previously stalled the cluster at a
stale version while reporting success.

migrations/42: drop the misleading "no 3-way-merge subtlety" claim from
the header. The apply records the no-delete label under the default
field manager's last-applied annotation, so a future label-less apply
by the same field manager would strip it. Document the constraint
instead of denying it.

hack/e2e-install-cozystack.bats: restore `helm.sh/resource-policy: keep`
on the reconstructed cozystack-version ConfigMap in the bypass-path
teardown so cluster state matches the chart template shape
(packages/core/platform/templates/cozystack-version.yaml) on exit.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 15:12:54 +05:00
Myasnikov Daniil 2bed6eb090 fix(platform): do not swallow kubectl errors in unprotect.sh
The blanket `2>/dev/null || true` suppression masked RBAC, wrong-context,
and API-unreachable failures, letting the script print "Done." while
leaving labels in place. Replace with a helper that only tolerates the
"unknown resource type" error (the actual best-effort case — kinds that
don't exist on every cluster, e.g. linstorcluster on a non-LINSTOR
install) and propagates everything else.

Also keep stdout (resource list) and stderr (kubectl warnings) separate
so deprecation warnings can't sneak into the loop variable.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 15:12:54 +05:00
Myasnikov Daniil c2f1bb2eac feat(platform): make unprotect strips no-delete label for teardown
Address review feedback from @lexfrei on packages/core/platform/templates/
deletion-protection.yaml: with this PR every protected object — CRDs,
ClusterIssuers, cozy-system and tenant-root Namespaces, cozystack-version
ConfigMap, cozystack-packages Repository, tenant-root HelmRelease,
LinstorCluster, plus the VAP and Binding themselves — carries the no-delete
label. `helm uninstall cozy-platform` issues DELETE on each, and DELETE is
denied for every one of them, so the chart effectively cannot be uninstalled
without manual `kubectl label` against each object. Disaster recovery and
dev-cluster reset both regress as a result.

Add packages/core/platform/hack/unprotect.sh: enumerates the guarded set
locally (per-kind, not via a single all-resources sweep — some kinds may
not exist on every install, e.g. linstorcluster on a non-LINSTOR cluster,
and kubectl errors on unknown kinds). For cluster-scoped kinds, lists
objects with the label and labels them off one at a time. For namespaced
kinds, lists (namespace, name) pairs via jsonpath and labels each in the
right namespace. Idempotent: a label that is already absent is a no-op.

Wire `make unprotect` in packages/core/platform/Makefile so the operator
runs it before `helm uninstall`. The Makefile target carries a comment
noting this is teardown / disaster-recovery only — after it runs, the
cluster has no guardrail until the next `helm upgrade` re-stamps the
label.

Verified with a stub kubectl in PATH that the script issues the right
get/label calls for both cluster-scoped (e.g. `kubectl label
customresourcedefinition/packages.cozystack.io platform.cozystack.io/
no-delete-`) and namespaced kinds (e.g. `kubectl label --namespace
tenant-root helmrelease tenant-root platform.cozystack.io/no-delete-`).

The template-header reference at templates/deletion-protection.yaml that
forward-links to `make unprotect` was added in the preceding L3 commit;
the recipe it points at now exists.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 15:12:53 +05:00
Myasnikov Daniil 91aae36da3 feat(platform): NOTES.txt warns when no-delete VAP is silently skipped
Address review feedback from @lexfrei on packages/core/platform/templates/
deletion-protection.yaml: on a pre-1.30 cluster the capability gate
(`.Capabilities.APIVersions.Has "admissionregistration.k8s.io/v1/
ValidatingAdmissionPolicy"`) evaluates to false and the policy + Binding
render to nothing. There was no signal in `helm install`/`helm upgrade`
output that the guardrail had been silently skipped, so operators on
older clusters lost the protection without any way to discover that from
the install log. Compounded the integration-test gap @lllamnyp called
out in blocker 3.

Add packages/core/platform/templates/NOTES.txt that renders a multi-line
WARNING whenever the capability is absent: states the guardrail is NOT
active, explains why (1.30+ required), notes that the label is still
stamped on every protected object so the guardrail becomes active
automatically on the next upgrade after a cluster bump, no migration
step required.

Add packages/core/platform/tests/notes_capability_gate_test.yaml: positive
case asserts the warning text + the 1.30 hint + the label string render
on a chart-level `capabilities.apiVersions: []`. Negative case asserts
the warning is absent when admissionregistration.k8s.io/v1/
ValidatingAdmissionPolicy is in the capability list. Verified the
positive case fails when the warning text is removed from NOTES.txt.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 15:12:53 +05:00
Myasnikov Daniil 919a8a9d05 docs(platform): clarify sequential self-protection on no-delete VAP
Address review feedback from @lexfrei on packages/core/platform/templates/
deletion-protection.yaml: both the ValidatingAdmissionPolicy and the
ValidatingAdmissionPolicyBinding carry the no-delete label so each
individually requires the bypass dance, but the labeling is symmetric, not
truly bilateral. Removing the label from the Binding and deleting it first
leaves the VAP labeled but unenforced, so the VAP itself is then deletable
without the bypass — and vice versa. That is by design: the guardrail is
operational guidance, not adversarial defense against an operator with
cluster-admin who is determined to remove it.

Add a paragraph in the template comment header so a future reader does not
waste time wondering why the symmetric self-protection isn't truly
bilateral. Forward-references the whole-chart teardown recipe (make
unprotect) so the comment links to the documented escape hatch.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 15:12:53 +05:00
Myasnikov Daniil 155c011dd5 test(platform): e2e check that no-delete VAP denies + bypass works
Address review feedback from @lllamnyp on packages/core/platform/templates/
deletion-protection.yaml: the VAP contract had no integration coverage.
A unit test that greps the rendered manifest is the wrong shape — the
contract is "the api-server denies a labeled DELETE with the documented
message" and that needs a real apiserver in the loop.

Add a post-install bats test to hack/e2e-install-cozystack.bats that runs
once cozystack is installed on the e2e cluster:

  1. Skips on clusters that pre-date Kubernetes 1.30 (no VAP API).
  2. Asserts the cozystack-version ConfigMap actually carries
     platform.cozystack.io/no-delete=true — precondition, so the deny
     assertion below would not misreport on a regressed binding.
  3. Asserts `kubectl delete configmap cozystack-version -n cozy-system`
     exits non-zero AND its error message contains the documented
     "Deletion blocked: ... platform.cozystack.io/no-delete=true"
     substring AND the `--namespace` bypass hint.
  4. Confirms the ConfigMap survives the rejected delete.
  5. Removes the label, deletes (must succeed), recreates the ConfigMap
     with the original data.version and the label so the cluster ends
     the test in the same state it started.

This single test catches every regression the PR is meant to prevent:
capability gate inverted, binding objectSelector mistyped,
validationActions flipped Deny→Warn, expression flipped false→true,
label-key drift between the binding and the manifests, and the bypass
docs missing --namespace.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 15:12:53 +05:00
Myasnikov Daniil e34eeb0ee7 fix(platform): document --namespace bypass and add VAP unit tests
Address review feedback from @lllamnyp on packages/core/platform/templates/
deletion-protection.yaml:

The bypass instructions in both the template-author-facing comment and the
user-facing CEL message did `kubectl label <kind> <name>` with no
--namespace flag. Several protected objects are namespaced (cozystack-version
ConfigMap, cozystack-packages Repository, tenant-root HelmRelease), so an
operator who copies the documented command verbatim against their default
context gets a "not found" error and concludes that the bypass mechanism is
broken. Updated both call sites to show --namespace <ns> with an explicit
"(omit for cluster-scoped resources)" caveat.

The reviewer also flagged that the VAP contract is otherwise unasserted by
unit tests. Add packages/core/platform/tests/deletion_protection_test.yaml
covering:
  - both the VAP and the Binding carry platform.cozystack.io/no-delete=true
    on themselves (the self-protection contract)
  - failurePolicy is intentionally absent on the VAP
  - the deny message contains the bypass command WITH --namespace and the
    cluster-scoped caveat (regression catch for this fix; verified failing
    when --namespace is dropped from the template)
  - the Binding's objectSelector + validationActions: [Deny] still match
    the no-delete label

Wire packages/core/platform/Makefile's test target to helm unittest so
hack/helm-unit-tests.sh actually runs the new suite in CI (the existing
tests/ files were orphaned without it).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 15:12:53 +05:00
Myasnikov Daniil ecdf3f3a15 fix(platform): drop redundant failurePolicy on no-delete VAP
Address review feedback from @lllamnyp on packages/core/platform/templates/
deletion-protection.yaml: the VAP's validation expression is the literal
`false` — no parameters, no variables, no references — so CEL evaluation
cannot fail and the api-server never has occasion to apply failurePolicy.
The previous `failurePolicy: Fail` line was a no-op masquerading as
operational config.

Drop the field (the default is Fail anyway in admissionregistration.k8s.io/v1)
and document in the template comment why it is intentionally absent, so the
next reader does not re-add it on the assumption that it was forgotten.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 15:12:53 +05:00
Myasnikov Daniil 052f070b63 fix(platform): migration 42 uses a single declarative apply
Address review feedback from @lllamnyp on packages/core/platform/images/
migrations/migrations/42: the previous version did `kubectl label --overwrite`
then `kubectl apply` on a manifest with no labels, relying on `kubectl apply`'s
3-way strategic merge to leave the imperatively-set label intact. That works
in the steady state but is fragile — the contract is "the label is preserved
because it is never in last-applied-configuration" — and it is inconsistent
with the helm template at templates/cozystack-version.yaml, which bakes the
label inline.

Replace the two-step dance with one declarative apply that includes the
deletion-protection label, matching the chart template. One source of truth,
no last-applied-merge subtlety. The script remains idempotent (apply is a
no-op once the ConfigMap matches).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 15:12:53 +05:00
Myasnikov Daniil 5d263a1e25 test(crdinstall): assert no-delete label survives Install + merge
Address review feedback from @lllamnyp on internal/crdinstall/install.go:
the deletion-protection label loop was untested — TestInstall_appliesAllCRDs
only checked that Install returned no error, so a refactor dropping the loop
would have shipped green and silently disabled VAP coverage on every Cozystack
CRD.

- Extend TestInstall_appliesAllCRDs to read each applied CRD back from the fake
  client and assert platform.cozystack.io/no-delete=true is present.
- Add TestInstall_preservesExistingLabels: feeds in a CRD whose manifest already
  carries app.kubernetes.io/managed-by=foo, asserts both that label and the
  deletion-protection label end up on the applied object. Locks down the merge
  contract in install.go:92-101 (the GetLabels()==nil branch is still covered
  by the first test).

Verified the new assertions fail when the label-stamping loop is removed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 15:12:52 +05:00
Myasnikov Daniil c7227bc666 feat(platform): migration 42 backfills no-delete label on cozystack-version
Address review feedback from coderabbitai on
packages/core/platform/templates/cozystack-version.yaml:10:
existing clusters skipped the label because the template's
`lookup`+`if not $configMap` guard never re-renders the ConfigMap on
upgrade (the skip is intentional — it prevents helm from clobbering the
data.version that the migration controller writes from its pre-upgrade
hook).

A one-shot migration is the correct mechanism here:

  - Migration 42 in packages/core/platform/images/migrations/migrations/42
    runs `kubectl label --overwrite configmap cozystack-version
    platform.cozystack.io/no-delete=true` and stamps version=43. Slot 42
    rather than 40 because main already occupies 40 (dashboard CRD
    cleanup) and 41 (kubernetes ephemeralStorage→diskSize rename).
  - Bump migrations.targetVersion 42 -> 43 so the hook fires on the
    next platform upgrade.

Idempotent: --overwrite is a no-op when the label is already at the
target value, MIGRATION_DRY_RUN=1 prints without applying, and the
script aborts before stamping if the kubectl label call fails.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 15:12:52 +05:00
Myasnikov Daniil e5dd5681c3 feat(platform): self-protect the deletion-protection VAP and Binding
Address review feedback from gemini-code-assist on
packages/core/platform/templates/deletion-protection.yaml:15:
label the ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding
themselves with platform.cozystack.io/no-delete=true so the guardrail
guards itself. A direct kubectl delete on either resource now returns
the same denial as deletion of any other protected object; the same
two-step label-then-delete bypass applies when intentional removal is
needed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 14:18:22 +05:00
Myasnikov Daniil 43ef456d84 feat(platform): gate deletion-protection VAP on Kubernetes 1.30+ capability
Address review feedback from gemini-code-assist on
packages/core/platform/templates/deletion-protection.yaml:42:
wrap the ValidatingAdmissionPolicy and Binding in a Helm capability
check so the template stays renderable on clusters that pre-date 1.30
(when ValidatingAdmissionPolicy graduated to GA). The check targets the
specific Kind, not the API group/version — admissionregistration.k8s.io/v1
itself has existed since 1.16 for ValidatingWebhookConfiguration, so the
group/version alone is not a meaningful capability gate.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 14:18:22 +05:00
Myasnikov Daniil 0c17d1870c feat(platform): add deletion-protection guardrail via ValidatingAdmissionPolicy
Block accidental DELETE on critical platform objects (tenant-root,
cozy-system, LinstorCluster, CRDs, ClusterIssuers, cozystack-version
ConfigMap, cozystack-packages OCIRepository, tenant-root HelmRelease)
via a label-scoped ValidatingAdmissionPolicy. No webhook DaemonSet,
Service, TLS, or image — the API server evaluates the CEL expression
in-process. Requires Kubernetes 1.30+.

Bypass for legitimate deletes is two steps:
  kubectl label <kind> <name> platform.cozystack.io/no-delete-
  kubectl delete <kind> <name>

CRDs installed dynamically by the operator get the label stamped in
internal/crdinstall/install.go before the apply call.

Release note:
Added deletion-protection guardrail (ValidatingAdmissionPolicy) blocking
DELETE on cozystack platform objects labeled platform.cozystack.io/no-delete=true.
Bypass by removing the label first.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-26 14:18:22 +05:00
Aleksei Sviridkin 92000f1618 fix(kubernetes): add config_path patch for containerd 2.x (#2723)
What this PR does
This PR adds support for containerd 2.x CRI config_path configuration,
as the CRI plugin section was renamed from io.containerd.grpc.v1.cri to
io.containerd.cri.v1.images.

It also makes the sed patching regexes more resilient by matching any
quotes (single or double) around the plugin names, which fixes the issue
on Ubuntu 24.04 (Noble) where containerd 2.x uses single quotes:
[plugins.'io.containerd.cri.v1.images'.registry].

Release note
fix(kubernetes): add config_path patch for containerd 2.x

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Broadened Kubernetes cluster bootstrap handling to better support
different containerd plugin layouts, improving compatibility when
configuring container registry paths during cluster initialization.
* **Tests**
* Added regression tests and fixtures to validate the containerd
configuration patching logic and integrated the check into the
Kubernetes test target.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2723?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-26 11:04:39 +03:00
Aleksei Sviridkin 3756f5c459 feat(networking): Gateway API support via Cilium (supersedes #2213) (#2470)
## What this PR does

Replaces ingress-nginx as the default ingress layer for cozystack-native
services with **Cilium Gateway API**, materialised per-tenant via a new
`TenantGateway` CRD reconciled by `cozystack-controller`. Supersedes
#2213 / #2208. Stacked on top of #2468.

## Architecture

The chart renders one `gateway.cozystack.io/v1alpha1 TenantGateway` CR
per tenant that owns its own Gateway. The cozystack-controller
reconciles the actual `Gateway`, per-tenant `Issuer`, `Certificate`
(wildcard or per-listener), and HTTP→HTTPS redirect `HTTPRoute` from
there. Helm does not render `Gateway` or `Certificate` directly — that
prevents the Helm-vs-controller race on `Gateway.spec.listeners` that
route-driven listener additions would otherwise cause.

```
┌── TenantGateway CR (rendered by extra/gateway chart for owning tenants)
│
└──▶ cozystack-controller reconciles:
        ├─ Gateway (with dynamic listeners, label-based allowedRoutes)
        ├─ Issuer (per-tenant ACME account)
        ├─ Certificate(s) — wildcard + per-child SANs in DNS-01, per-listener in HTTP-01
        ├─ HTTPRoute (controller-owned http→https redirect)
        ├─ Namespace label patching (cozy-* system namespaces from spec.AttachedNamespaces)
        └─ Watches HTTPRoute / TLSRoute attachments to materialise listeners
```

## Inheritance: tenants attach by default, opt into their own Gateway

A tenant gets a dedicated Gateway (own LB Service, own LB IP, own
Certificate) only when it explicitly asks via
`tenant.spec.gateway=true`. Every other tenant in the tree publishes
through the Gateway of the nearest ancestor that owns one — same shape
as the existing `_namespace.ingress` inheritance.

The `apps/tenant` chart writes a `namespace.cozystack.io/gateway` label
on each tenant namespace, carrying either the tenant's own name (when
owning a Gateway) or the inherited ancestor name (when inheriting). The
Gateway's listener `allowedRoutes.namespaces.selector` is keyed on that
label, so the same selector admits routes from every namespace pointing
at the owner — descendants and cozy-* system namespaces alike.
cozystack-controller patches the same label onto every namespace in
`tgw.Spec.AttachedNamespaces` and garbage-collects labels it wrote when
an entry is removed from the attach list (annotation-gated GC so labels
written by the apps/tenant chart are never stripped).

In DNS-01 mode, the controller extends the parent's wildcard Certificate
with `<child-apex>` + `*.<child-apex>` SANs per inheriting child, and
adds a `*.<child-apex>` HTTPS listener per child apex referencing the
same cert. Without this expansion the parent's single-level wildcard
cannot match a child route's hostname (`harbor.alice.example.org` is two
labels past the parent's `*.example.org`).

Opt-out into own Gateway is the right choice when: a tenant needs its
own LB IP (DNS pinned, firewall rule), the apex is not derived from the
parent (custom `host` value the ancestor's cert can't cover), or the
tenant wants its own ACME account / cert authority. Otherwise leave
`gateway` unset and inherit.

## End-to-end flow

1. Platform: `gateway.enabled=true` toggles cert-manager solver,
switches per-system-app templates from Ingress to HTTPRoute / TLSRoute.
2. Per tenant: explicit `tenant.spec.gateway=true` renders a
`TenantGateway` CR and triggers the controller to materialise the owner
Gateway. Tenants with the field unset inherit through
`_namespace.gateway` propagation in the tenant chart.
3. Adding a published app under any tenant (owner or inheriting) is just
deploying its HTTPRoute. The controller picks up the hostname, adds a
per-listener Certificate (HTTP-01) or extends the wildcard cert's SANs
(DNS-01), and updates the owner Gateway's listeners.

Defaults stay legacy: `gateway.enabled=false`,
`tenant.spec.gateway=false`. Existing clusters upgrade unchanged.

## External IP allocation — mechanism-agnostic

The per-tenant Gateway's auto-created `LoadBalancer` Service draws its
IP from whatever LB allocator the cluster admin has configured at the
platform layer — same shape as ingress-nginx today. The tenant API stays
mechanism-agnostic — no `gatewayIP` field, no allocator-specific
manifest in the tenant chart. Cozystack itself ships MetalLB installed
but does not render any `IPAddressPool` / `L2Advertisement` /
`BGPAdvertisement` from this chart; admins set up the allocator that
suits their environment (MetalLB pool with L2 / BGP, Cilium LB-IPAM with
announcer, robotlb against a cloud provider, or
`Service.spec.externalIPs` pinning).

If a tenant needs a specific address (DNS already pinned, firewall rule,
etc.), the operator pre-allocates it on the admin side: either
pre-create the Service with `loadBalancerIP` set, or hand the tenant a
reference to a named admin-managed pool. Per-Service IP uniqueness is
the allocator's responsibility — same as for any other LoadBalancer
Service.

## Cert mode: HTTP-01 (default) vs DNS-01 (opt-in)

`publishing.certificates.solver` selects how the controller sources TLS
certs.

**HTTP-01 (default)** — per-app HTTPS listener + per-app Certificate.
New apps require zero platform-side config: deploy the HTTPRoute,
controller does the rest. Inheriting children's hostnames pick up
per-listener certs the same way the owner's own apps do.

**DNS-01 (opt-in)** — single wildcard Certificate (`<apex>` +
`*.<apex>`) covering every published app under the apex, plus per-child
SANs (`<child-apex>` + `*.<child-apex>`) for every inheriting tenant.
Pick this for clusters where many apps share the apex and Let's Encrypt
rate limits matter. The DNS provider account must be able to write TXT
records under every apex level the parent serves — for deeply-nested
inherited children that requires zone delegation or a wide-scope
provider credential.

| `publishing.certificates.dns01.provider` | Required
`publishing.certificates.dns01.<provider>` keys |
| --- | --- |
| `cloudflare` (default) | `cloudflare.secretName`,
`cloudflare.secretKey` |
| `route53` | `route53.region`, `route53.secretName` (and `accessKeyID`
if not on IRSA) |
| `digitalocean` | `digitalocean.secretName` |
| `rfc2136` | `rfc2136.nameserver`, `rfc2136.tsigKeyName`,
`rfc2136.secretName` |

The platform chart wires every provider's keys into `_cluster.dns01-*`;
the per-tenant gateway chart and the cluster-wide `cluster-issuers.yaml`
both read them.

## Strengthens admission coverage on apps.cozystack.io/*

The `cozystack-tenant-host-policy` probe in `gateway.bats` initially
looked like a flake — the apply that was supposed to be VAP-rejected
just succeeded. Investigation traced it to the custom REST handler at
`pkg/registry/apps/application/rest.go`. genericapiserver hands every
storage method a callback into the validating admission chain:
`createValidation` for Create, `updateValidation` for Update,
`deleteValidation` for Delete. `Update` was correctly invoking its
callback since 23e399bd, but `Create` and `Delete` accepted the
parameter and never called it — admission was silently bypassed for both
verbs on every `apps.cozystack.io` resource (Tenant, MariaDB, Postgres,
Kubernetes, …).

This PR wires the missing `createValidation` and `deleteValidation`
calls into Create and Delete. Every ValidatingAdmissionPolicy and
ValidatingWebhook against `apps.cozystack.io/*` now fires on all three
verbs as the API contract requires. Per KEP-3488 aggregated APIServers
are responsible for enforcing admission on their own kinds;
cozystack-api is configured to do so via
`genericoptions.NewRecommendedOptions`, the only thing missing was the
explicit `*Validation` callbacks in the custom REST.

## Security — five guards, grouped by what they defend

Tenants in Cozystack interact with the platform exclusively through
`apps.cozystack.io/*` resources (Tenant, Bucket, Kubernetes, …) served
by `cozystack-api`. Tenant RBAC (`cozy:tenant:*` aggregated to a
RoleBinding in the tenant's own namespace) does not grant write access
to `gateway.networking.k8s.io/*`, core `Namespaces`, or
`cozystack.io/Package`. The protections below split into three groups by
who they defend against:

**Tenant-user-input gates.** Layer 4 (`cozystack-tenant-host-policy`).
`Tenant.spec.host` is the user-supplied field that surfaces as a
security boundary at the hostname layer; gated on every Create / Update
via `cozystack-api`'s admission chain.

**Defense-in-depth.** Layers 1, 2, 5, 7. These do not protect against
tenant-user input (tenants don't hold the relevant RBAC). They guard
against bugs in cozystack-controller / Flux, supply-chain compromise of
an app chart that emits Gateway API resources, and confused-deputy
mistakes by a cluster admin. Fail-closed via `failurePolicy: Fail` +
`validationActions: [Deny]`.

The five layers themselves:

1. **Listener `allowedRoutes` label selector** —
`namespace.cozystack.io/gateway = <owner-tenant-name>`
(controller-written for cozy-* via `AttachedNamespaces`, Helm-written
for tenant namespaces via the tenant chart, both feeding into the same
Gateway selector).
2. **`cozystack-gateway-hostname-policy`** — VAP on `Gateway`
CREATE/UPDATE. Restricts listener hostnames to the namespace's
`namespace.cozystack.io/host` apex.
4. **`cozystack-tenant-host-policy`** — VAP on `Tenant` CREATE/UPDATE.
5. **`cozystack-namespace-host-label-policy`** — VAP on core `Namespace`
CREATE/UPDATE. Immutability gate on the host label.
7. **`cozystack-route-hostname-policy`** — VAP on `HTTPRoute` and
`TLSRoute` CREATE/UPDATE. Closes the hostname-hijack vector across
namespaces under inheritance — a child cannot claim a hostname outside
its own apex.

The previous Layer 3 (`cozystack-gateway-attached-namespaces-policy`)
and Layer 6 (render-time `fail` in cozystack-basics) banned `tenant-*`
entries in `publishing.gateway.attachedNamespaces` on the assumption
that the attach list was the hijack vector. Under inheritance the attach
surface is the label selector, not the static list; the hijack vector is
closed by Layers 4/5/7 independently. Both Layer 3 and Layer 6 are
dropped — kept in numbering above for continuity with the earlier review
threads.

## Foreign-takeover guards

Five reconcile paths refuse to silently rewrite a pre-existing object
that shares the controller-derived name but carries no `OwnerReference`
back to the TenantGateway: `Gateway`, redirect `HTTPRoute`, per-tenant
`Issuer`, wildcard `Certificate`, per-listener `Certificate`. An
operator who hand-pinned a Certificate or Issuer at the controller's
derived name (private CA, manual cert pinning) gets an explicit
`Ready=False/ReconcileError` condition instead of having their config
silently destroyed and the resource re-issued from a different ACME
account.

The namespace-label patching path applies the same discipline:
cozystack-controller only writes / strips
`namespace.cozystack.io/gateway` on namespaces it annotates with
`cozystack.io/gateway-attached-by`. Labels written by the apps/tenant
chart (no annotation) are never touched, so inheritance for tenant
namespaces survives every reconcile.

## Tests

- **Go controller tests** — 50+ cases in
`internal/controller/tenantgateway/`: idempotency, OwnerReference
cascade, multi-parentRef status, HTTP-01↔DNS-01 mode-transition cert
lifecycle, foreign-takeover refusals (Gateway / HTTPRoute / Issuer /
Certificate / per-listener Certificate), DNS-01 provider matrix
(cloudflare / route53 / digitalocean / rfc2136), HTTPS Kinds=[HTTPRoute]
restriction, fail-closed CEL on missing host label, status condition
wiring. Plus, for inheritance: label-based allowedRoutes selector shape,
namespace label patch + garbage-collect via owner annotation, Helm-owned
label safety, DNS-01 wildcard SAN expansion per inheriting child,
per-child wildcard listener materialisation.
- **Admission-chain tests** —
`pkg/registry/apps/application/rest_admission_test.go` pins
`createValidation`, `updateValidation`, `deleteValidation` paths via
sentinel callbacks.
- **helm-unittest** across `extra/gateway` (TenantGateway CR rendering,
both modes), `core/platform` (DNS-01 wiring), `apps/tenant`
(gateway-effective decision matrix including inheritance),
`system/cozystack-basics` (three VAPs + Layer 7 VAP + tenant-root
namespace label including the conditional
`namespace.cozystack.io/gateway: tenant-root` + cozystack-values-secret
`_namespace.gateway` propagation), `system/cert-manager-issuers` (full
DNS-01 provider matrix), `system/cozystack-api` and `system/dashboard`
(Ingress↔TLSRoute/HTTPRoute toggle), `system/monitoring` (Grafana /
Alerta Ingress↔HTTPRoute toggle).
- **e2e bats** — `hack/e2e-apps/gateway.bats` (17 cases) covering
GatewayClass Accepted, Gateway Programmed + LB Service, HTTPRoute
Accepted with status, VAP rejects cross-tenant hostname hijack, VAP
rejects `tenant.spec.host` writes from non-trusted SA
(regression-detector for the admission-chain fix), VAP rejects
`namespace.cozystack.io/host` label changes from non-trusted SA,
route-hostname VAP rejects HTTPRoute hostnames outside the apex, and
three new cases for inheritance — Package admission accepts `tenant-*`
in `attachedNamespaces` (regression guard for the dropped ban), child
tenant without explicit gateway inherits `_namespace.gateway` from a
Gateway-owning parent in lockstep across the namespace label and the
cozystack-values Secret, and the cross-namespace attach end-to-end where
a child tenant's HTTPRoute reaches Accepted on the parent's Gateway via
the inheritance label.
- **CI wiring** — `make test-controllers` target invoked from
`pull-requests.yaml` runs `go test ./internal/...`.

## Backward compatibility

The only always-on change is the flip of `cilium.envoy.enabled` +
`cilium.gatewayAPI.enabled` to `true` (extra `cilium-envoy` DaemonSet,
~100 MB RAM/node idle). Everything else is gated behind
`gateway.enabled=false` / `tenant.spec.gateway=false`. Existing clusters
see no behavioural change until an operator explicitly opts in.

The cozystack-api admission-chain fix is unconditional. Existing
operators who do not define any VAP / admission webhook on
`apps.cozystack.io` resources see no observable change. Operators who DO
have their own webhooks/VAPs against these kinds will start seeing them
fire on Create and Delete (as intended) — review your policies before
merging if you maintain custom admission for `apps.cozystack.io/*`.

Operators who set `publishing.gateway.attachedNamespaces` to include
`tenant-*` entries used to hit a render-time fail or an admission
rejection — now the apply succeeds and the listed tenant namespaces
simply pick up the gateway-attach label alongside the cozy-* system
namespaces. This matches the inheritance model and was kvaps' explicit
ask in the May 25 review.

## Release note

```release-note
- Adds Gateway API support via Cilium, materialised per-tenant via the new `gateway.cozystack.io/v1alpha1 TenantGateway` CRD reconciled by `cozystack-controller`. Opt-in via `publishing.gateway.enabled=true` at the platform level; per-tenant opt-in via `tenant.spec.gateway=true` for dedicated Gateway / LB IP, otherwise the tenant inherits the nearest ancestor's Gateway via a label-based `allowedRoutes` selector (same shape as the existing `_namespace.ingress` inheritance). Includes HTTP-01 (default) and DNS-01 (opt-in, with cloudflare / route53 / digitalocean / rfc2136 providers) cert solvers; DNS-01 extends the parent's wildcard Certificate with per-child apex SANs and adds a `*.<child-apex>` listener per inheriting tenant. Tenant API is mechanism-agnostic for IP allocation — the LoadBalancer Service draws its IP from the cluster's admin-configured allocator (MetalLB / Cilium LB-IPAM / robotlb / externalIPs).
- Strengthens admission coverage on `apps.cozystack.io/*` resources: `cozystack-api` now invokes `createValidation` and `deleteValidation` callbacks on Create and Delete (Update was already wired). Operators with custom ValidatingAdmissionPolicies or webhooks targeting `apps.cozystack.io/*` will see them fire on all three verbs as the API contract requires; review your policies before merging if you maintain such admission.
```
2026-05-26 10:29:30 +03:00
Aleksei Sviridkin c85e473523 test(e2e): inheritance attach asserts controller writes, not Cilium Accepted
The previous e2e check waited for
`HTTPRoute.status.parents[0].conditions[Accepted]=True` after
applying a child tenant's HTTPRoute against the parent's Gateway.
That condition is set by Cilium after the listener's TLS Secret
becomes Ready — and in the e2e cluster every fresh certificate fails
to issue (LE prod refuses every `.example.org` hostname by policy,
visible across alerta/grafana/dashboard/seaweedfs-s3 cluster certs
in cozyreport). So Cilium never set Accepted, the route timed out
3/3 retries, and the failure was environmental — not a regression
in the inheritance code path.

The contract the test exists to pin is cozystack-controller's
behaviour, which has two observable writes:

1. The parent Gateway grows a per-listener HTTPS entry for the
   child route's hostname (proves collectHostnameClaims saw the
   route through the inheritance label and reconcileGateway
   appended the listener).
2. A per-listener Certificate object appears in the parent
   namespace with the route's hostname in spec.dnsNames (proves
   reconcilePerListenerCertificates ran for the same hostname).

Both writes are deterministic and depend only on the inheritance
label + the HTTPRoute existing — not on ACME, not on Cilium TLS
readiness. The test now polls those two objects directly within
shorter, scenario-appropriate timeouts (120s for the listener, 60s
for the Certificate). The route Accepted check is dropped with an
explanation in the test comment so a future cluster with a working
ACME doesn't get a misleading pre-condition added back. The
namespace inheritance label is also verified up front as the
precondition collectHostnameClaims's filter reads.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-25 23:51:00 +03:00
Aleksei Sviridkin 320172137e fix(tenantgateway): include inheriting child namespaces in hostname claim collection
The HTTP-01 inheritance flow deadlocked at the route-Accepted step:
the apps/tenant chart correctly labelled the child namespace with
`namespace.cozystack.io/gateway=<owner>`, the parent Gateway's
allowedRoutes selector matched by label, but `collectHostnameClaims`
filtered routes by the static set `{tgw.Namespace} ∪ Spec.AttachedNamespaces`
— so a child route in a namespace reached purely via the inheritance
label was silently dropped. No per-listener Certificate, no per-app
HTTPS listener, no hostname match → `Accepted=False,
Reason=NoMatchingListenerHostname` indefinitely.

Caught by the new `child tenant's HTTPRoute attaches to parent's
Gateway via inheritance label` e2e bats — failed 3/3 retries on the
CI run for `afc7765e6`. The Go controller-level
TestReconcile_HTTP01CollectsHostnamesFromInheritingChildNamespaces
test pins the same contract at a faster feedback loop: an HTTPRoute
in a namespace labelled `namespace.cozystack.io/gateway=<owner>` but
NOT listed in Spec.AttachedNamespaces must still cause the
controller to render a per-listener HTTPS listener for its hostname.

The fix extends the `allowed` set in `collectHostnameClaims` to
also include namespaces matched by the gateway label selector, so the
hostname-claim collector and the Gateway's allowedRoutes selector
agree on which namespaces can attach. DNS-01 mode is unaffected — it
returns nil from `collectHostnameClaims` immediately and uses the
separate `collectInheritingChildApexes` path for SAN expansion.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-25 21:43:51 +03:00
myasnikovdaniil 2cb16d5ae6 refactor(build): standardize image tagging to fix concurrent PR push conflicts (#2711)
## What this PR does

Concurrent PR builds were racing in the OCIR registry. Every package
Makefile tagged its images via `settag`, which on any non-release commit
collapsed to `:latest` — so every image across every PR was pushed to
the same tag, and parallel builds got 409s on the manifest PUT. Two
outliers (`metallb`, `redis-operator`) used hardcoded version tags and
collided across PRs unconditionally.

This PR replaces `settag` and the `TAG=latest` fallback in
`hack/common-envs.mk` with a single `image-tags` macro driven by three
env vars:

| Env | Meaning |
|---|---|
| `IMAGE_TAG` | Build-unique tag, always pushed |
| `PUBLISH_VERSIONED` | Also push `:<component-version>` (when provided)
|
| `PUBLISH_FLOATING` | Also push `:latest` |

Wired into the two workflows:

- **`pull-requests.yaml`** — `IMAGE_TAG=pr-<N>-<sha>`, both `PUBLISH_*`
off. Every PR build pushes a unique tag set; no collisions, no `:latest`
move.
- **`tags.yaml`** — `IMAGE_TAG=<ref_name>`, both `PUBLISH_*` on. Release
output preserved (component versions + `:latest`) plus a uniform
`:<git-tag>` handle on every image for traceability.
- **Local** — defaults to `IMAGE_TAG=dev`, no flags, `PUSH=0`.

All 30 package Makefiles converted to `$(call
image-tags,<repo>,<versioned-tag>)`. Image consumers reference digests
pinned into `values.yaml` by each Makefile, so the tag is decorative for
cluster-side pulls — the build-unique tag is sufficient for E2E.

Also drops the redundant `:<component-version>-<git-tag>` form (cilium,
linstor, matchbox, ubuntu-container-disk, kubevirt-*,
cluster-autoscaler). Nothing in the repo or downstream values.yaml
consumed it; the same information is preserved via the new `:<git-tag>`
plus the existing `:<component-version>` tags on release.

`ubuntu-container-disk` builds a separate image per Kubernetes version
in a loop, so its primary tag became `:<k8s-ver>-<IMAGE_TAG>` to prevent
loop iterations within a single build from pushing to the same tag.

Verified by dry-running `make -n image` across representative packages
(simple, dual-tag, skopeo, flux-push, loop) under all four contexts (PR
/ release / main / dev) — tag sets are unique and behave as expected.

### Release note

```release-note
refactor(build): PR CI now pushes images under a unique `pr-<N>-<sha>` tag per build instead of `:latest`. Concurrent PR builds no longer race in the registry. Release builds keep the existing `:<component-version>` and `:latest` tags and additionally publish a uniform `:<git-tag>` handle across every image. The redundant `:<component-version>-<git-tag>` tag (cilium, linstor, matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) is no longer published — image references in values.yaml are digest-pinned and were the only legitimate consumers.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Centralized and simplified container image tagging across build
workflows.
* PR builds now use unique image tags derived from PR metadata for
better isolation.
* Added configurable flags to control publishing of versioned and
floating tags, making release tagging behavior more predictable.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2711?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-25 22:16:57 +05:00