Address review feedback from lexfrei on migrations/43: the version stamp
did a label-less `kubectl create configmap | kubectl apply` under the same
field manager migration 42 used, stripping the platform.cozystack.io/no-delete
label and dropping cozystack-version out of the deletion-protection guardrail on
every upgrade through this migration. Replace it with a labeled heredoc manifest
mirroring migration 42 so the label survives the stamp.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Re-pull the vendored flux-operator and flux-instance charts from 0.48.0 to the
latest v0.50.0 (operator-version bumps only: OpenShift v4.21 compat, OLM base
image, Kubernetes client deps — no Flux distribution change). The kubernetesEnvs
and networkPolicy patches re-apply cleanly. distribution.version stays 2.8.x, so
the tenant Flux toolkit auto-tracks the latest 2.8 patch (currently 2.8.8) at
reconcile. The management flux-aio module is already at its latest published
build (2.8.0); no newer 2.8.x module exists upstream yet.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The imageType default was switched to split in values.yaml/values.schema.json/
RD schema but make generate was not run, leaving the +kubebuilder:default marker
in api/apps/v1alpha1/foundationdb/types.go at "unified" and the README param
table sized for the old value — failing the pre-commit generate check (exit 123).
Regenerated with cozyvalues-gen v1.5.0.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- kafka: assert WorkloadMonitor reads .Values.kafka.replicas /
.Values.zookeeper.replicas (regression for the phantom .Values.replicas),
and that enableServiceLinks lives under entityOperator.template.pod.
- tenant: pin that no swept HelmRelease carries upgrade.force (would have
caught the missed gateway.yaml).
- seaweedfs-db: baseline render cover for the new split-out chart.
- kafka and seaweedfs-db had no `test` Makefile target, so their suites never
ran in CI; add one to each.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
tenant-rd now carries release.cozystack.io/helm-install-timeout alongside
kubernetes-rd (its parent chart bootstraps the seaweedfs-db CNPG cluster, whose
first reconcile exceeds flux's default wait budget). Update the comment that
claimed only kubernetes-rd carries it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Building from unpinned common-instancetypes HEAD on every `make update` silently
churns the catalog. Worse, the v1.3.0 upstream removal dropped the gn1.* (GPU
NVIDIA) instancetypes that existing GPU VMs and tenant Kubernetes GPU workers
still reference — vm-instance/templates/vm.yaml hard-fails the render when a
referenced instancetype is absent.
Pin the source to a commit SHA (not a tag: the multi-arch preferredArchitecture
fields exist only on main past v1.6.0, so a tag would regress them) and
re-append the gn1.* objects from a Cozystack-local extra-instancetypes.yaml the
regen copies into templates/. The EOL centos.7*/centos.stream8* preferences
stay dropped (release-noted). Regenerate the vm-instance and kubernetes README
instancetype/preference tables to match the shipped catalog.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Pre-SSA the chart nested imageType under automationOptions, where Flux v2.7
silently dropped it, so the operator always used its own default (split).
Flux v2.8 SSA now delivers the field, so the chart's previous "unified" default
would trigger an in-place image-type migration on the first upgrade of every
existing cluster. Default to "split" to match the value clusters effectively
ran; operators can set "unified" to migrate deliberately.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The flux-v2.8 sweep removed `upgrade.force: true` from the sibling tenant
HelmReleases (etcd, info, ingress, monitoring, seaweedfs) for helm-controller
v1.5 SSA compatibility, but gateway.yaml was missed. Complete the sweep so no
tenant HR keeps the flag that conflicts with Server-Side Apply.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
run-migrations.sh loops `seq $CURRENT $((TARGET-1))`, so with targetVersion: 43
a cluster at version 42 runs only migration 42 — migration 43 (the seaweedfs-db
Cluster adoption introduced by the split) is never iterated, and the
missing-file guard never fires because 43 is out of range. The migration is
written as "43 -> 44" and stamps version=44. Without it the new seaweedfs-db
HelmRelease hits a Helm ownership conflict on the existing Cluster and the
seaweedfs-system upgrade can prune it (filer metadata loss). Set targetVersion
to 44 so migration 43 runs, matching the migration-42 precedent.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The two external-dns bats tests already poll for the user-facing ${name}
HelmRelease before calling kubectl wait, but they immediately wait on the
${name}-system HR without the same backstop. The system HR is materialised
asynchronously by the controller, so kubectl wait can race and exit non-zero
when the object does not exist yet. Apply the same existence poll pattern
to both occurrences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The vendored flux-operator chart ships a NetworkPolicy on port 9080 gated
by web.enabled + web.networkPolicy.create (default true). Cozystack already
restricts ports 8080/8081/9080 via a CiliumClusterwideNetworkPolicy patched
into the same chart, so the upstream policy is redundant if a user opts in
to web.enabled: true. Pin networkPolicy.create to false to keep the Cilium
policy as the single source of truth.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Splits the `seaweedfs-system` HelmRelease into two: the CNPG
`Cluster/seaweedfs-db` moves to its own HR (`seaweedfs-db`), and the
application HR (`seaweedfs-system`) `dependsOn` it. The new HR uses Flux
v2 `HelmRelease.spec.healthCheckExprs` with a CEL expression on
`Cluster.status.conditions[type=Ready]` plus `waitStrategy.name:
poller`, so its `Ready=True` only flips after the postgres primary is
actually serving connections — not just after `helm install` applied the
`Cluster` CR.
This eliminates the `seaweedfs-filer` CrashLoopBackOff race on a fresh
tenant install. With Cilium `kubeProxyReplacement: true`, socket-LB
returns `EPERM` from `connect(2)` to ClusterIPs with no Ready endpoints.
Pre-split, the filer StatefulSet scheduled concurrently with the CNPG
bootstrap (~55–70s of unavailable postgres), each `connect()` failed
`EPERM`, kubelet exponential restart backoff pushed past the e2e bats
`kubectl wait hr/seaweedfs-system --timeout=2m` window, and the
"Configure Tenant and wait for applications" test failed.
## Stacked on #2602
This PR depends on **#2602** (Flux v2.8.0 upgrade) — `healthCheckExprs`
is a helm-controller v1.5.0 feature. GitHub base set to `flux-bump-v048`
so the diff cleanly shows only this PR's changes; will auto-rebase onto
`main` once #2602 merges.
## Commits
1. **`refactor(seaweedfs)`** — the split itself. Squash of the three
iterative commits from the consolidated branch (#2619): the initial
split, a `dependsOn` merge-collision fix (collapses unconditional
`seaweedfs-db` dependency with the namespace-guarded ingress dependency
into one list), and the `healthCheckExprs` three-predicate form (route
CNPG's `ClusterIsNotReady` bootstrap condition to `inProgress`, reserve
`failed` for other `Ready=False` reasons, lean on the HR `timeout: 10m`
as the real backstop).
2. **`fix(tenant)`** — bump parent HR `spec.timeout` from 5m → 15m so
the serialized `seaweedfs-db → seaweedfs-system → seaweedfs` chain has
enough wall-clock budget. Two complementary changes: the static
`tenant-root` HR (created by `cozystack-basics`) gets its `spec.timeout`
bumped directly; sub-tenants (Tenant CRs whose HR is built by
`cozystack-api` from the `tenant` ApplicationDefinition) get the
`release.cozystack.io/helm-install-timeout=15m` annotation that
`kubernetes-rd` already uses to opt into the longer budget.
3. **`test(e2e)`** — consolidate two redundant child-HR waits in
`hack/e2e-install-cozystack.bats` into one budgeted assert. The second
`kubectl wait` (`hr/monitoring hr/seaweedfs-system --timeout=2m`) was a
residue of a `flux-reconcile-force` workaround removed in #2509;
`monitoring`'s Ready is already implied by `hr/tenant-root` Ready and
`seaweedfs-system`'s by `hr/seaweedfs` Ready, so the line was
tautologically true by the time it ran.
## Migration
`packages/core/platform/images/migrations/migrations/42` adopts existing
`Cluster/seaweedfs-db` resources into the new release on upgrade —
rewrites `meta.helm.sh/release-name: seaweedfs-system → seaweedfs-db`
and stamps `helm.sh/resource-policy: keep` so the `seaweedfs-system`
upgrade (which no longer renders the Cluster) does not delete it during
the transition. Renumbered from the consolidated branch's slot 40 to fit
fresh main (last migration is 41). `targetVersion` bumped 42 → 43.
The new HR's `upgrade.force: true` was dropped to match #2602's
platform-wide cleanup (Helm v4 in helm-controller v1.5.0 rejects
combining `--force-replace` with SSA's `--force-conflicts`).
## Verification
- `helm template` renders `seaweedfs-db` HR cleanly with
`healthCheckExprs`, `waitStrategy: poller`, and `upgrade: { remediation:
{ retries: -1 } }` (no `force: true`).
- Migration 42 syntax-checks under `sh -n`.
- All four PR commits GPG-signed + DCO-signed-off.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* SeaweedFS database is now available as a separate, independently
configurable component with customizable replicas, storage size, and
storage class options.
* **Chores**
* Restructured database configuration from the main application into a
dedicated component for improved modularity and initialization order.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2601?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The flux-bump-v048 base branch removed upgrade.force from every other
HelmRelease in 7f94182a87 because helm-controller v0.48 refuses to
combine Helm's --force replace with server-side apply's --force-conflicts
("cannot use force conflicts and force replace together"). This refactor
re-introduced the field on the seaweedfs-system HR by accident; CI E2E
fails with the same error in a loop until the seaweedfs subtree times
out the tenant install. Drop the line so the upgrade path matches every
other HR on the base branch.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
values.yaml declared an unused Resources typedef while
templates/database.yaml hardcoded cpu/memory limits and requests. The
wrapper chart at packages/extra/seaweedfs/templates/seaweedfs-db.yaml
already passes db.resources through cozy-lib.resources.defaultingSanitize
into the sub-chart values, so the user-facing db.resources and
db.resourcesPreset on the outer chart silently dropped on the way in.
Drop the unused typedef, accept .Values.db.resources as a free-form
object (the wrapper already sanitizes it into the K8s nested
limits/requests shape that CNPG Cluster expects), and wire it through
to the Cluster manifest. Defaults are preserved verbatim so helm
template output is byte-identical for a no-override install.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
PR #2619 split seaweedfs into a serialized chain
(seaweedfs-db -> seaweedfs-system -> seaweedfs) that needs ~5 min of
wall-clock to install: CNPG bootstrap, master quorum, then the wrapper.
With the prior 5m wait budget on the parent tenant HR, helm-controller
would time out before the children reach Ready, rollback would delete
the child HRs, and the upgrade would retry in a loop. The e2e bats step
that patches the Tenant CR and waits 60s for child HRs to exist landed
in the rollback gap and timed out.
Two complementary changes:
- tenant-root is created statically by cozystack-basics, so its
HelmRelease spec.timeout is bumped 5m -> 15m directly.
- Sub-tenants (Tenant CRs) get their HR built by cozystack-api from
the tenant ApplicationDefinition. Add the
release.cozystack.io/helm-install-timeout=15m annotation that
kubernetes-rd already uses to opt the kind into the longer budget.
Keeps both values in lockstep and documents the coupling inline.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit cc9da3a80a)
Splits the seaweedfs-system HelmRelease in two so the CNPG
Cluster/seaweedfs-db lives in its own HR (seaweedfs-db), and the
application HR (seaweedfs-system) dependsOn it. The new HR uses Flux v2
HelmRelease.spec.healthCheckExprs with a CEL expression on
Cluster.status.conditions[type=Ready] plus waitStrategy.name: poller, so
its Ready=True only flips after the postgres primary is actually serving
connections — not just after helm install applied the Cluster CR.
This eliminates the seaweedfs-filer CrashLoopBackOff race on a fresh
tenant install. With Cilium kubeProxyReplacement: true, socket-LB
returns EPERM from connect(2) to ClusterIPs with no Ready endpoints.
Pre-split, the filer StatefulSet scheduled concurrently with the CNPG
bootstrap (~55–70s of unavailable postgres), each connect() failed
EPERM, kubelet exponential restart backoff pushed past the e2e bats
'kubectl wait hr/seaweedfs-system --timeout=2m' window, and the
'Configure Tenant and wait for applications' test failed.
healthCheckExprs uses Flux's three-predicate form: route
ClusterIsNotReady (the CNPG bootstrap condition) to `inProgress` so the
HR keeps polling, keep `failed` for other Ready=False reasons, and rely
on the HR `timeout: 10m` as the real backstop for a genuinely stuck
cluster. A plain Ready=False predicate would flag the HR failed within
seconds of creating the Cluster CR.
Migration 42 adopts existing Cluster/seaweedfs-db resources into the new
release on upgrade by rewriting meta.helm.sh/release-name and stamping
helm.sh/resource-policy: keep so the seaweedfs-system upgrade (which no
longer renders the Cluster) does not delete it during the transition.
The application HR's dependsOn collapses the db dependency (unconditional)
with the existing ingress dependency (guarded by `if eq $ingress
.Release.Namespace` so sub-tenants that inherit ingress from a parent
namespace don't deadlock on a non-existent local HR) into one list.
Requires Flux v2.8.x (helm-controller v1.5.0+) for healthCheckExprs.
Provided by the parent PR #2602 in this chain.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two timing bugs unmasked by Flux v2.8 kstatus, with the same root shape:
the bats steps assumed a side-effect ordering ("by the time X is true,
the parent HR's helm install is done") that held under v2.7's faster
dispatch but breaks under v2.8 where helm-controller walks kstatus on
every applied resource before flipping the HR Ready.
run-kubernetes.sh:
After every addon HR (cilium, coredns, csi, vsnap-crd, ingress-nginx)
reaches Ready, the remediation guard reads `.status.history` on the
parent kubernetes-${test_name} HR. The parent's helm install can still
be Reconciling at that point because kstatus is finishing its full
sweep; `.status.history` is empty until install completes, so the
guard exits 1 with "empty .status.history on a Ready HR". kubernetes-
latest masked this because the ouroboros assertions add ~10m of
buffer; kubernetes-previous (no ouroboros) hit the guard with the
parent still installing. Add an explicit
kubectl wait hr/kubernetes-${test_name} --for=condition=ready
before the guard.
vminstance.bats:
The test waited 120s for `kubectl get vmi vm-instance-$name` to
return a VMI with an assigned IP, then waited for the parent HR Ready
separately afterwards. Under v2.7 the parent HR's helm install
applied the VirtualMachine CR fast enough that kubevirt had created
the VMI before the 120s budget expired. Under v2.8 the dispatch is
slower and the VMI poll returned NotFound for its entire budget.
Reorder: wait for HR Ready first (which implies the VirtualMachine
CR is applied + kstatus-ready), then poll for the VMI with IP.
Loosen the VMI poll budget from 120s to 5m for nested-virt slow paths.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit c664b868bc)
Flux helm-controller v1.5.0 (shipped in Flux v2.8) replaced the legacy
Helm readiness logic with kstatus-based health checking, which polls
Deployment/StatefulSet/Service status and waits for kstatus to assess
rollout-ready. This typically adds 10-30s of latency between resource
apply and HR condition Ready=True compared to v2.7.
Our per-app bats tests had `kubectl wait hr/... --for=condition=ready
--timeout=` budgets of 20s, 30s, 60s, 100s — sized for v2.7's faster
readiness flip and previously masked by the 3x retry on Run E2E tests
(dropped in #2558). Under v2.8 with no retry, every app that runs after
the first few hit its HR wait timeout exactly, surfacing as 13 of 16
test failures on PR #2619.
Standardise on 5m for HR Ready across all app tests. Tight enough to
catch genuine bring-up failures within the per-app job budget, loose
enough to absorb kstatus dispatch jitter on a management cluster that
is concurrently reconciling multiple tenant-test HRs.
Post-HR-Ready checks for app-specific resources (StatefulSet/Service/
PVC/operator CRs) remain — for CR-based apps kstatus only observes
that the chart applied, not that the operator finished. Audit of
which of those are actually redundant under v2.8 kstatus is queued
as a follow-up.
Also bumps the 10s "kamajicontrolplane appears" wait in
run-kubernetes.sh to 2m, and the kubernetes-addon HR Ready loop from
1m to 5m for the same reason.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5ef02ee7b3)
Each of the 19 child HR templates in packages/apps/kubernetes/templates/
helmreleases/ added the parent kubernetes-application HR as a dependsOn
entry, guarded by a `lookup "helm.toolkit.fluxcd.io/v2" "HelmRelease"`
on the parent so it only stamped after the parent already existed.
Under Flux 2.7 (helm-controller v1.4.x), the parent HR's Ready=True flips
after helm install returns, regardless of any child HRs it just applied,
so the guard was a no-op for first install and a soft ordering hint for
subsequent reconciles.
Under Flux 2.8 (helm-controller v1.5.0), the default waitStrategy is
`poller` -- the parent HR now waits via kstatus for every applied
resource, including the 19 child HRs, to be Ready=True before flipping
its own Ready. The child HRs in turn cannot become Ready until they have
a working `super-admin.svc` kubeConfig and the lookup-stamped dependsOn
back-edge resolves -- which only happens after the parent HR is Ready.
Deadlock: parent waits on child, child waits on parent.
Removing the parent dependsOn from each child breaks the cycle. The
children start reconciling immediately; helm-controller's existing retry
loop covers the brief window before the parent's helm install lands the
admin-kubeconfig secret. The cross-child ordering (cilium -> cert-manager,
prometheus-operator-crds -> metrics-server, etc.) stays intact via the
unconditional sibling entries.
Five files (the *-crds and vsnap-crd HRs) had only the parent reference
under dependsOn; remove \`dependsOn:\` entirely in those. Cilium had one
conditional gateway-api-crds entry under it; pull the \`dependsOn:\` key
inside that conditional so it disappears when gateway-API is off rather
than rendering as a null array.
This is option B from issue #2642 task 1.
Verified:
- \`helm template\` clean with default values-ci.yaml and with every addon
enabled (cert-manager, ingress-nginx, monitoring-agents, velero, fluxcd,
gateway-api, gpu-operator, hami, ouroboros).
- No remaining \`- name: {{ .Release.Name }}\` dependsOn entries in
packages/apps/kubernetes/templates/helmreleases/.
- \`helm unittest\` passes (123/123).
Refs: #2642
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 6119aafcb5)
The packages/system/vertical-pod-autoscaler chart's vpa-for-vpa.yaml template
emits an HR in cozy-vpa-for-vpa that the parent helm install applies and waits
on. That nested HR had `dependsOn: monitoring-agents`, which never resolves:
parent VPA helm install
waits for nested vpa-for-vpa HR Ready
waits for monitoring-agents HR Ready
waits for cozystack.vertical-pod-autoscaler PackageSource Ready
waits for parent VPA HR Ready <-- back to start
Symptom observed on PR #2619 e2e (cozyreport):
Helm upgrade failed for release cozy-vertical-pod-autoscaler/
vertical-pod-autoscaler with chart cozy-vertical-pod-autoscaler@...:
timeout waiting for: [HelmRelease/cozy-vpa-for-vpa/vpa-for-vpa
status: 'InProgress']
The recommender's /health-check readiness probe is served by the binary
itself and doesn't require Prometheus. Removing the dependsOn lets the
nested HR become Ready quickly; the recommender will log connection
errors against vmselect until monitoring-agents installs later in the
boot sequence, then start producing recommendations normally.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 08433dc02e)
When a disk entry has no bus set (the schema default), the previous
template rendered `disk:` with no value — YAML parses that as
`disk: null` and KubeVirt's virtualmachines-mutator webhook denies the
request with: "spec.template.spec.domain.devices.disks.disk in body
must be of type object: null".
Restructure the disk loop so the disk/cdrom value is always a valid
object: `disk: {}` when bus is empty, `disk: {bus: <x>}` when set.
Mirrors how `cloudinitdisk` already does it in the same template.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 0f3d2fa3a6)
(cherry picked from commit e2c6b7b60b)
Flux v2.8 / helm-controller v1.5.0 embeds Helm v4, which renamed --force
to --force-replace and rejects combining it with SSA's --force-conflicts.
Existing HelmReleases with spec.upgrade.force: true fail every upgrade with
"cannot use force conflicts and force replace together", looping
upgrade -> rollback -> upgrade and blocking dependants. Observed on PR #2619
CI for tenant-root/{ingress-nginx-system,seaweedfs-db}, tenant-test/info,
and others.
Per Flux docs, upgrade.force has been a no-op since v2.8 anyway, so removing
it is safe on green clusters and unblocks broken ones.
Also fix packages/apps/kafka/templates/workloadmonitor.yaml: it referenced
.Values.replicas (no such field; values.yaml has kafka.replicas and
zookeeper.replicas). Flux v2.7 silently dropped the null; v2.8 SSA forwards
it, and the WorkloadMonitor CRD rejects "spec.replicas: null" as not
type:integer, which blocks every kafka-test install.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 7e286cf8e1)
Same class of bug as the foundationdb misplaced-field fixes. The kafka
chart had:
entityOperator:
template:
pod:
metadata: { labels: { ... } }
spec: ← child of template, not of pod
enableServiceLinks: false
The Strimzi Kafka CRD's `entityOperator.template` has no `spec` key.
`enableServiceLinks` is a direct field of `template.pod` (alongside
metadata, affinity, tolerations, etc.). Flux v2.7's permissive
client-side patch silently dropped the misplaced field; v2.8's strict
Server-Side Apply rejects:
failed to create typed patch object (Kind=Kafka):
.spec.entityOperator.template.spec: field not declared in schema
Move `enableServiceLinks` under `template.pod` where the operator
actually reads it, and fix the inconsistent metadata.labels indentation
(was 9/11 spaces instead of 10/12) while we're here.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 447f1dcbf1)
The chart nested `faultDomain`, `imageType`, `labels`, and
`minimumUptimeSecondsForBounce` inside `automationOptions`, but the
FoundationDBCluster CRD schema defines all four as direct children of
`spec`. Flux v2.7 (helm-controller v1.4.3) used permissive client-side
patch and silently dropped the unknown fields; v2.8 (helm-controller
v1.5.0) does strict Server-Side Apply and rejects:
failed to create typed patch object: .spec.automationOptions.<field>:
field not declared in schema
This was a latent bug from day one — none of those fields ever actually
configured anything. Moving them one level up applies the values to
where the operator looks for them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
`make update` stripped `persistent: true` from upstream's
`preferredTPM: { persistent: true }`, leaving the parent key with a null value.
KubeVirt v1.6's CRD tolerated it; v1.8 (the operator version we bumped to in
6af0bf71) enforces structural schema and rejects null:
VirtualMachineClusterPreference "windows.11" is invalid:
spec.devices.preferredTPM: Invalid value: "null": spec.devices.preferredTPM
in body must be of type object: "null"
Stop stripping the line and regenerate the templates from upstream
common-instancetypes. Persistent TPM state falls back to the cluster default
StorageClass with RWO access mode when `vmStateStorageClass` is unset on the
KubeVirt CR, per pkg/storage/backend-storage/backend-storage.go.
Drive-by from the regen:
- Removed (upstream EOL): centos.7*, centos.stream8*
- Added: debian, oraclelinux, legacy, linux*, fedora.s390x, rhel.{9,10}.s390x,
windows.{xp,7,7.virtio,2k3,2k8,2k8.virtio,2k12,2k12.virtio}, plus new
cx1.*1gi and d1 instancetype variants.
Also regenerates packages/apps/vm-instance instanceProfile enum
(values.schema.json / README.md) from the refreshed preferences.yaml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Re-render internal/fluxinstall/manifests/fluxcd.yaml via
`make update` in packages/core/flux-aio (timoni bundle build against
oci://ghcr.io/stefanprodan/modules/flux-aio at the latest tag) so
the cozystack-operator binary embeds Flux 2.8.0 instead of 2.7.3.
Auto-syncs the helm-controller image into fluxcd-tenants.yaml.
Embedded versions now match the chart bump in the prior commit:
flux: v2.7.3 -> v2.8.0
helm-controller: v1.4.3 -> v1.5.0 (enables healthCheckExprs)
source-controller: v1.7.3 -> v1.8.0
kustomize-controller: v1.7.2 -> v1.8.0
notification-controller:v1.7.4 -> v1.8.0
source-watcher: v2.0.2 -> v2.1.0
Verified by rebuilding cmd/cozystack-operator and running `strings`
on the binary -- the expected v1.5.0 / v1.8.0 / v2.1.0 image strings
are now embedded in place of the v1.4.x / v1.7.x ones.
Structure preserved:
- single AIO Deployment named "flux" in namespace cozy-fluxcd
- all 5 controllers + source-watcher as containers in one pod
- custom tolerations from flux-aio.cue intact
- Namespace pod-security label still "privileged" via the
Makefile yq post-processing step
Net diff is -3664 lines (mostly CRD schema simplification in 2.8).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Bump flux-operator and flux-instance vendored charts from v0.33.0 to
v0.48.0, moving the Flux distribution from 2.7.x to 2.8.x. This brings
helm-controller v1.5.x, which exposes spec.install.healthCheckExprs on
HelmRelease — a prerequisite for adding real readiness gating to
dependency-target packages (postgres, mongodb, kafka, etc.).
The new flux-operator chart defaults web.enabled=true; override it to
false in packages/system/fluxcd-operator/values.yaml so the Flux Status
Web UI stays opt-in. Operators wanting it can set web.enabled=true and
configure web.config / web.configSecretName / web.ingress per the
upstream chart values.
Patches reapplied:
- kubernetesEnvs.diff: matched cleanly (offset shift only)
- networkPolicy.diff: extended to also restrict port 9080 (web UI port);
hunk count updated 20 -> 22
Pre-flight: no references to the v0.39-removed --disable-wait-interruption
flag anywhere in packages/ or internal/.
Verified locally:
- helm template renders both packages without errors
- web.enabled=false (default): no web Deployment/Service/NetworkPolicy
- web.enabled=true: renders web NetworkPolicy on port 9080
- CiliumClusterwideNetworkPolicy lists 8080/8081/9080
- FluxInstance distribution.version: "2.8.x", all 7 components present
- Cozystack kustomize patches (concurrent, requeue, storage-adv-addr,
events-addr) intact
Notable upstream changes between v0.33 and v0.48:
- Flux 2.8.x with HelmRelease inventory format change (auto-migrated)
- Reliability fixes for stuck FluxInstance/ResourceSet on health-check
cancellation (v0.39, v0.46)
- New CLI: flux-operator migrate, diff, patch instance, distro mirror
- ResourceSetInputProvider: GitLab Environments, GitHub App, Gitea/Forgejo
- CVE-2026-23990 (Web UI OIDC impersonation bypass) — N/A while web is off
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Bundles four independent e2e improvements lifted from #2619:
1. **`test(e2e): fold ouroboros assertions into kubernetes-latest`** —
moves the standalone `hack/e2e-apps/ouroboros.bats` (which provisioned
its own full Kamaji control plane, ~25m) into the existing
`kubernetes-latest` run as an optional addon. The assertions cover
ouroboros HR Ready, controller pod Running, `Ingress → coredns-custom`
rewrite-line injection, and end-to-end DNS resolution from inside the
tenant. Reuses the existing tenant cluster instead of building a second
one.
2. **`test(e2e): wait for tenant CSI HR Ready before NFS test`** —
explicit `kubectl wait hr/kubernetes-${test_name}-csi
--for=condition=ready` ahead of the NFS pod. Otherwise the NFS pod
schedules while the `kubevirt-csi-node` DaemonSet is still rolling out,
eats ~1m on `FailedAttachVolume` retries, and trips the pod-Succeeded
wait.
3. **`test(e2e): bump NFS pod wait to 10m to fit slow-path chain`** —
was 5m. With dependencies properly serialized (per #2), 5m was tight on
slow runners; 10m gives the chain genuine room without masking real
bugs.
4. **`test(e2e): set keycloakInternalUrl when enabling OIDC`** — the
dashboard gatekeeper (oauth2-proxy) does OIDC discovery against the
keycloak Service URL; without an explicit `keycloakInternalUrl`, it
falls back to the public ingress URL which isn't routable from inside
the cluster yet during install.
## Origin
All four commits lifted unchanged from #2619.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Integrated Ouroboros validations into the main Kubernetes end-to-end
flow (removed separate standalone Ouroboros test)
* Added hairpin-NAT and in-tenant DNS resolution checks for Ouroboros
scenarios
* Improved Keycloak OIDC test by configuring an in-cluster internal URL
* Increased various timeouts and readiness waits and added tenant CSI
readiness gating to improve test reliability
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2728?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds `cert-manager` as a `dependsOn` entry to the five PackageSources
whose charts apply `cert-manager.io/v1` `Certificate` CRs and need the
webhook ready before their first reconcile. Without this, the helm
install renders Certificate manifests, the cert-manager webhook isn't
admitting requests yet (cold cluster), the apply fails, and
helm-controller retries until the webhook eventually comes up — burning
~2-3 minutes per affected package on cold installs.
The five PackageSources: `capi-operator`, `cozystack-engine`,
`ingress-nginx`, `kamaji`, and `rabbitmq-operator` (the platform
`cozy-system` set with webhook-cert consumers; not the apps layer).
In `cozystack-engine`, the dependency is added to both the `default` and
`oidc` variants — the `oidc` variant installs the same `cozystack-api`
and `lineage-controller-webhook` components that render Certificate
resources, so it had the same race.
## Origin
First commit lifted unchanged from #2619. Second commit (`oidc` variant)
added in response to review feedback.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Chores**
* Updated package source dependencies across multiple components to
ensure proper installation sequencing.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2726?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds a new `ClusterRole` `cozystack-dashboard-cluster-usage` and its
`ClusterRoleBinding` under `packages/system/dashboard/templates/`, plus
the matching helm-unittest fixture. The role grants the
`cozystack-cluster-admin` group cluster-wide read on `nodes`, `pods`
(get/list/watch) and on `metrics.k8s.io` `nodes` (get/list).
Companion to the upcoming UI work in cozystack/cozystack-ui#15 — a new
admin page `Console → Administration → Cluster Usage` that renders
cluster-scoped resource utilisation and a per-node table including
arbitrary extended resources (GPUs and other accelerators). The
page-side sidebar item is gated by a `SelfSubjectAccessReview` on `nodes
list`; without this binding the gate fails closed and the entry stays
invisible, which is the desired fallback for clusters that have not
adopted the new RBAC yet.
The built-in `edit` `ClusterRole` already bound to
`cozystack-cluster-admin` via `cozystack-dashboard-cluster-admin` does
not grant cluster-wide pod read or any access to `nodes`, so an explicit
grant is required for the page to render anything beyond an error.
This chart intentionally does not add permissions for deeper sources
(Prometheus, DCGM, HAMi). Those belong in their respective packages or
in operator-provided RBAC; the page detects them via discovery and gates
each section with its own per-source review.
### Screenshots
No UI in this PR — RBAC only.
### Release note
```release-note
feat(dashboard): grant cluster-admin read access to nodes, pods and metrics.k8s.io for the new Cluster Usage admin page
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Dashboard cluster usage monitoring now includes configured role-based
access control with appropriate read permissions for nodes, pods, and
metrics data.
* **Tests**
* Added comprehensive test suite to validate the dashboard's role-based
access control configuration, including verification of proper
permission bindings and authorization rules.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2743?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Both \${test_name} and \${component} are literal-safe today (no
whitespace, no glob characters), so this is a style-only change — but
quoting variable expansions in kubectl arguments matches the rest of
the file and removes a future-foot-gun if a test name ever grows a
hyphen-y suffix that interacts oddly with completion or globbing.
Scoped to the two HelmRelease wait lines flagged by CodeRabbit and
echoed by the human reviewer; the machinedeployment wait on the
preceding line keeps its existing style.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
\`local k8s_version=\$(yq ...)\` masks the exit status of the command
substitution — the assignment to a local variable always succeeds, so
a failing yq is swallowed. Declare the local first, then assign, so
non-zero yq exits propagate via shell errexit semantics if the script
ever runs under \`set -e\`.
Pure shell hygiene; no behavioral change on the success path. Surfaced
by CodeRabbit and confirmed by the human reviewer as non-blocking but
worth applying.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The previous commit (3c81261b) folded the ouroboros hairpin-NAT
reconciliation assertions into the kubernetes-latest tenant under
run-kubernetes.sh, but did not remove the standalone bats file.
The e2e workflow auto-discovers every hack/e2e-apps/*.bats and dispatches
test-apps-<name> via the wildcard rule in packages/core/testing/Makefile.
Leaving ouroboros.bats in the tree meant the assertions ran twice: once
on the standalone ~25m Kamaji-backed tenant and again as the folded
addon on the kubernetes-latest tenant — the opposite of the consolidation
the fold was intended to deliver.
The folded assertions in hack/e2e-apps/run-kubernetes.sh cover the same
surface (HR Ready, controller pod Running, coredns-custom rewrite-line
injection, and end-to-end DNS resolution from inside the tenant), so
the standalone file is dead weight after the fold.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The dashboard gatekeeper (oauth2-proxy) does OIDC discovery against the
external host at container startup. The e2e sandbox uses
host=example.org which does not resolve, so the auth-proxy crashloops:
Get "https://keycloak.example.org/realms/cozy/.well-known/openid-configuration":
dial tcp: lookup keycloak.example.org: no such host
Under Flux v2.8 kstatus the gatekeeper Deployment then flips to
'Failed' (rather than the pre-v2.8 'InProgress') and stalls the
cozy-dashboard/dashboard HelmRelease, which in turn blocks
cozy-fluxcd/flux-plunger via dependsOn. Previous runs masked this with
an unrelated ImagePullBackOff on cozy-dashboard-console; with the
console image fixed the OIDC discovery failure now surfaces.
packages/core/platform/values.yaml:194-198 already declares
authentication.oidc.keycloakInternalUrl. When set, the dashboard
chart (packages/system/dashboard/templates/gatekeeper.yaml:59-66)
renders --skip-oidc-discovery plus explicit redeem/jwks/validate/
backend-logout URLs pointing at the in-cluster keycloak Service.
Browser-facing redirect-url and login-url stay on the external host
(they are not exercised by the e2e flow).
The patch sets the value alongside oidc.enabled in the same
kubectl-patch so the configmap and gatekeeper rollout happen in one
reconcile pass.
Observed on CI run 26142986969 (PR #2619).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5915a579ee)
Previous run failed kubernetes-latest (and then kubernetes-previous via
cascade) on the 5m budget for nfs-test-pod to reach Succeeded. The
chain is host linstor prime PVC + CDI import + tenant CSI attach +
busybox image pull + containerd CreateContainer (which occasionally
stalls past kubelet's 2m CRI deadline and retries with name-reservation
conflict). ~40s on the happy path, 6m+ in observed worst case.
When kubernetes-latest times out, its tenant cluster (2x 20Gi worker
PVCs) leaks because set -e exits before any cleanup. kubernetes-
previous then runs, adds another 40Gi, and the host tenant-quota
(100Gi) is exhausted -- every downstream app test (mariadb, mongo,
postgres, qdrant, redis, openbao, vminstance) fails with
ErrCreatingPVCPrime / exceeded quota.
Bumping the wait to 10m gives the slow path room. Headroom costs
nothing on green runs since the pod is busybox echo.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit c5cc7b60b8)
kubernetes-previous fails on the NFS pod step. From cozyreport + job log:
- Pod scheduled to tenant worker
- 7x FailedAttachVolume over 35s: CSINode does not contain driver
csi.kubevirt.io (kubevirt-csi-node DaemonSet still rolling out)
- Attach succeeds at T+1m08s after the driver registers
- busybox pull queued ~1m25s, completes at T+2m48s
- CreateContainer in containerd then stalls past kubelet's CRI
deadline; retry trips "failed to reserve container name ... is
reserved for <prev>" -- the 5m pod-Succeeded wait then expires.
Root cause: run-kubernetes.sh creates the NFS PVC+Pod before asserting
kubernetes-${test_name}-csi is Ready. That HR lives in the parent
cluster but installs cozy-kubevirt-csi-node into the tenant via
kubeConfig.secretRef; helm-controller walks kstatus on the tenant
DaemonSet using the tenant kubeconfig. HR Ready in parent therefore
implies kubevirt-csi-node Pods Ready on every tenant worker, which
implies csi.kubevirt.io is published in CSINode. The existing wait at
line 354 runs AFTER the NFS test, so it does not gate the race.
Move the wait earlier with a 10m budget. Post-NFS sibling waits stay
as is, so component coverage is unchanged.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 394b1b2b9a)
The standalone hack/e2e-apps/ouroboros.bats provisioned a full Kamaji-
backed tenant Kubernetes cluster (control plane + 2 CAPI workers + base
addons) just to flip `addons.ouroboros.enabled=true` on top, then ran
~15m of hairpin-NAT reconciliation assertions. Cluster shape was
identical to what hack/e2e-apps/run-kubernetes.sh already provisions for
kubernetes-latest, so the second Kamaji bringup was pure duplication —
roughly 25-45m of CI wall-time per run.
Add an `enable_ouroboros` 4th parameter to `run_kubernetes_test()`. When
"true", inject the ouroboros addon block into the Kubernetes CR's
addons map and run the post-cluster assertions inline:
- Wait for the ouroboros HR Ready in the host namespace.
- Verify coredns-custom ConfigMap exists in tenant kube-system.
- Wait for the ouroboros controller pod Ready.
- Apply an Ingress with a TLS host in tenant default namespace.
- Poll coredns coredns-custom for the rewrite line (5m budget).
- Resolve the hairpin host from inside the tenant via a netshoot pod
and assert it points at the ouroboros-proxy ClusterIP.
- Clean up dnscheck pod + Ingress.
kubernetes-latest passes "true" (gets ouroboros coverage); kubernetes-
previous leaves it empty (control-plane-only smoke remains).
(The standalone ouroboros.bats was removed in the preceding commit
alongside an unrelated foundationdb chart fix — the git rm staging had
already happened when the foundationdb commit landed.)
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit b5c9c2ac90)
The oidc variant installs system/cozystack-api and
system/lineage-controller-webhook, both of which render
cert-manager.io/v1 Certificate resources. Add cozystack.cert-manager to
the oidc variant's dependsOn so it matches the default variant and
webhook-cert consumers do not race the cert-manager controller on cold
installs.
Address review feedback from gemini-code-assist and coderabbitai on
packages/core/platform/sources/cozystack-engine.yaml.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Five PackageSources whose charts apply cert-manager Certificate CRs and
mount the resulting Secret on a webhook Deployment lacked an explicit
dependsOn: cozystack.cert-manager. Under Flux v2.8 helm-controller v1.5.0
kstatus, the dependent Deployment trips progressDeadlineSeconds (10m
default) when cert-manager controller isn't running yet, and the HR is
failed early. Repeatedly observed on PR #2619 with capi-operator,
kamaji, rabbitmq-operator, cozystack-api (cozystack-engine), and
ingress-nginx all cascading to InstallFailed at ~12-15m into a fresh
install.
The mariadb-operator, kubeovn-webhook, snapshot-controller, etcd-operator,
postgres-operator, mongodb-operator, victoria-metrics-operator, linstor,
and linstor-scheduler PackageSources already had the dependency, which
is why they weren't in the failure set.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 0fc0796b19)
## What this PR does
Adds a `ValidatingAdmissionPolicy` that blocks `DELETE` on critical
platform
objects labeled `platform.cozystack.io/no-delete=true`. Native,
in-process CEL
evaluation — no webhook DaemonSet, no Service, no TLS, no image to
maintain.
Requires Kubernetes 1.30+.
This is an alternative to the closed#2402 (admission webhook approach).
### Protected objects (labeled in this PR)
| Resource | Kind | Source |
|---|---|---|
| `cozy-system` | Namespace |
`packages/core/installer/templates/cozy-system-{namespace,labels}.yaml`
|
| `tenant-root` | Namespace |
`packages/system/cozystack-basics/templates/tenant-root.yaml` |
| `tenant-root` | HelmRelease (ns: tenant-root) | same |
| `cozystack-version` | ConfigMap (ns: cozy-system) |
`packages/core/platform/templates/cozystack-version.yaml` |
| `cozystack-packages` | OCIRepository (ns: cozy-system) |
`packages/core/platform/templates/repository.yaml` |
| `letsencrypt-prod`, `letsencrypt-stage`, `selfsigned-cluster-issuer` |
ClusterIssuer |
`packages/system/cert-manager-issuers/templates/cluster-issuers.yaml` |
| `linstorcluster` | LinstorCluster |
`packages/system/linstor/templates/cluster.yaml` |
| `packages.cozystack.io`, `packagesources.cozystack.io` | CRD | label
stamped in `internal/crdinstall/install.go` |
### Bypass
```bash
kubectl label <kind> <name> platform.cozystack.io/no-delete-
kubectl delete <kind> <name>
```
### Why VAP instead of a webhook
Same outcome as #2402 but with zero runtime infrastructure: no
DaemonSet, Service,
TLS certificate, or image. The kube-apiserver evaluates the CEL
`expression: "false"` in-process and denies because `validationActions:
[Deny]`.
`objectSelector` on the binding scopes evaluation to labeled objects
only — every
other DELETE in the cluster is unaffected.
### Release note
```release-note
feat(platform): add deletion-protection guardrail (ValidatingAdmissionPolicy)
blocking DELETE on cozystack platform objects labeled
`platform.cozystack.io/no-delete=true`. Bypass by removing the label first.
Requires Kubernetes 1.30+.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Deletion-protection guardrail: objects labeled
platform.cozystack.io/no-delete=true are denied DELETE requests; removal
of the label allows deletion.
* **Chores**
* Platform migration to v43 backfills the no-delete label onto system
resources and namespaces.
* **Tools**
* Added an "unprotect" teardown utility to remove the no-delete label
for uninstall/disaster recovery.
* **Documentation**
* Helm NOTES now warns when the guardrail is inactive due to missing
cluster API.
* **Tests**
* New tests and e2e checks validate deletion-protection behavior.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2650?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
Implements the `strategy.backups.cozystack.io/Job` backup strategy — a
generic, application-agnostic driver that runs a user-supplied
Kubernetes Job to create (and restore) a backup.
## Background
This PR began as a draft written against an earlier shape of the backups
API. The API has since moved to a `BackupClass`-based dispatch model,
which broke the original draft on three counts:
- `BackupJob.Spec.StrategyRef` and `BackupJob.Spec.StorageRef` were
removed; the strategy and its parameters are now resolved from a
`BackupClass` into a `ResolvedBackupConfig`.
- Strategy handling moved from standalone per-strategy reconcilers to a
dispatch model: `BackupJobReconciler.Reconcile` /
`RestoreJobReconciler.Reconcile` resolve the strategy and switch into
per-strategy methods (`reconcileVelero`, `reconcileCNPG`,
`reconcileAltinity`, …).
- The draft's template context (`.Values`, `.Storage`) diverged from the
convention the other strategies adopted, and its `.Storage` half
targeted a storage API that does not exist.
## What changed
- **`reconcileJob` / `reconcileJobRestore`**
(`internal/backupcontroller/jobstrategy_controller.go`): replaces the
standalone `BackupJobStrategyReconciler` with the two dispatch methods
the `BackupJob` / `RestoreJob` reconcilers call. The Job strategy is the
generic counterpart of the Altinity driver — same lifecycle (render
`PodTemplateSpec` → run a one-shot `batch/v1.Job` → poll its terminal
condition → emit a `Backup`), minus the ClickHouse-specific
`applicationRef` gate, so it runs for any application. Shared package
helpers (`renderJobTemplate`, `jobConditionState`,
`getApplicationUnstructured`, status helpers, deterministic Job naming,
controller refs) are reused.
- **Template context** follows the in-tree convention used by the other
drivers: `.Application` (live application object), `.Release.Name` /
`.Release.Namespace`, `.Mode` (`backup` / `restore`), `.Parameters`
(resolved `BackupClass` parameters), and `.Backup` (restore runs only).
Strategy parameters round-trip through the `Backup`'s `driverMetadata`
so a later restore re-renders against the same values.
- **Restore** re-renders the same template with `.Mode == "restore"`.
Cross-namespace restore is unsupported (the RestoreJob target reference
is namespace-local), matching the Altinity driver.
- **`JobSpec.Template` doc + regenerated CRD**: the field doc previously
advertised `.Values`; it now documents the real context, and
`strategy.backups.cozystack.io_jobs.yaml` is regenerated to match (the
change is limited to that field's description).
- **Tests** (`jobstrategy_controller_test.go`): backup-Job creation /
args / labels / status, completion → `Backup` artifact with parameter
round-trip, Job-failure surfacing, missing-strategy terminal failure,
restore into the target namespace, restore completion, and the
parameter-prefix unit. A deliberately generic application kind pins that
the driver never gates on `applicationRef.kind`.
No RBAC changes are required: the controller already holds
`strategy.backups.cozystack.io/*`, `batch/jobs`, `apps.cozystack.io/*`,
and `backups` permissions.
## Notes for reviewers
- Backup and restore share one template, so a strategy author's
`PodTemplateSpec` must branch on `.Mode` to do anything
restore-specific. This matches the existing convention; worth confirming
it's the intended contract for the Job strategy's docs/examples.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Implemented full backup/restore job strategy with deterministic Job
creation, idempotent execution, and status/condition updates; added demo
example scripts and reusable shell helpers for a NATS JetStream
backup/restore workflow.
* **Documentation**
* Clarified Job template rendering context and listed supported template
variables; added README and step-by-step scenario docs for the NATS
example.
* **Tests**
* Added comprehensive unit tests covering job creation,
completion/failure handling, parameter extraction, and unmappable-target
failure cases.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/1721?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Annotate `storageClass` as immutable across every stateful chart that
exposes one: clickhouse, foundationdb, harbor, http-cache, kafka (Kafka
+ ZooKeeper), kubernetes (top-level + nodeGroups[]), mariadb, mongodb,
nats, openbao, opensearch, postgres, qdrant, rabbitmq, redis, vm-disk.
The PVC-pinning argument is identical for every chart — Kubernetes fixes
`storageClassName` on each PVC at creation, so editing this field on an
existing resource never migrated data; the schema annotation makes that
contract explicit and lets schema-aware consumers refuse the edit
up-front.
Schema-aware consumers today: the Cozystack UI
(cozystack/cozystack-ui#6) reads the rule from the chart's openAPISchema
and renders the field read-only on edit forms.
This PR is **schema-side only**. The cozystack aggregated apiserver does
not yet evaluate CEL rules in `openAPISchema` (`MariaDB` / `Postgres` /
etc. are application-backed, not CRDs); tracking issue
cozystack/cozystack#2657 covers wiring CEL into
`pkg/registry/apps/application/rest.go::Update`. Until that lands, an
apiserver-level `kubectl patch` that changes `storageClass` is still
accepted — but the underlying StatefulSet PVCs do not move, so there's
no data-corruption risk, only a confusing UX. The UI catches that case
today; the apiserver gate closes it permanently.
## What changes
For each app in the list above:
- `packages/apps/<app>/values.yaml` — gain a `## @immutable` directive
directly above (or below, for fields inside a `@typedef`) the
`storageClass` line.
- Regenerated via `make generate` in each package:
- `api/apps/v1alpha1/<app>/types.go` — kubebuilder `XValidation` marker
on the relevant field(s).
- `packages/apps/<app>/values.schema.json` — `x-kubernetes-validations:
[{rule: "self == oldSelf", message: …}]` on `storageClass`.
- `packages/apps/<app>/README.md` — regenerated parameter table.
- `packages/system/<app>-rd/cozyrds/<app>.yaml` — the embedded
`openAPISchema` on the ApplicationDefinition reflects the change.
Refactor: `pkg/registry/apps/application/rest.go` factors the inline
schema-build path into a new `buildSpecSchema` helper so it can be
unit-tested without standing up a fake apiserver. Three regression tests
in `pkg/registry/apps/application/rest_spec_schema_test.go` pin the
contract that `x-kubernetes-validations` survives the
v1→internal→structural conversion (test loads the actual MariaDB
ApplicationDefinition fixture on disk, asserts the rule reaches the
structural schema). Empty-input and malformed-JSON paths are also
pinned.
No e2e admission test is added because the apiserver does not yet
enforce the rule — see cozystack/cozystack#2657.
No changelog entry: schema-side annotation is invisible at the apiserver
layer today (UI-only consumer) and the release-cut PR will add the entry
when admission enforcement lands.
## Dependencies
Requires cozystack/cozyvalues-gen#24 to be merged and released, and the
`cozyvalues-gen` binary pin in `.github/workflows/pre-commit.yml` line
50 bumped to the release that includes that PR. Pre-commit CI will fail
until both happen — the stock `v1.3.0` binary silently ignores `##
@immutable` and regenerates artefacts without the rule, producing a
drift against what's committed.
Once that ordering is satisfied, rebase this branch and the pre-commit
job will be green.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* storageClass is now treated as immutable after creation for many
stateful apps (enforced by chart/CRD schema validation).
* **Documentation**
* Added a storage immutability guide explaining the contract and
enforcement.
* Updated chart READMEs and values docs to mark storageClass as
immutable and clarified nodeGroup storage remains intentionally mutable.
* **Tests**
* Added tests validating schema parsing and preservation of immutability
rules.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2639?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Pre-pulls every image referenced by `packages/system/kubeovn`,
`packages/system/linstor`, and `packages/system/cert-manager` onto every
node before those HelmReleases install. Cluster-member workloads (OVN
raft, LINSTOR, cert-manager webhook) fail when replicas start at
different times due to per-node image-pull stagger; pre-pulling means
all replicas start with images already cached.
## Mechanics
Renders each chart with `helm template`, then runs a `yq` filter that
walks every PodSpec-shaped object and emits the images of each container
— scoping the output to images the kubelet actually pulls (skips
configmap fields and CRD examples that happen to contain an `image:`
key). Each render is staged through a tmp file so a helm-template
failure trips `set -e` cleanly (cozytest.sh runs `@test` bodies under
`/bin/sh`, which is dash on Ubuntu CI — no `pipefail`).
The pre-pull DaemonSet itself is the hard part: distroless images
(cert-manager and friends) ship no shell and no `/bin/sleep`, so the
obvious `command: ["sleep", "infinity"]` fails with `exec: ENOENT`. The
script stages a statically-linked `busybox:musl` into a shared
`emptyDir` from an initContainer; every prepull container then execs
`/shared/sleep` regardless of what the image itself ships. The glibc
`busybox` tag is unusable — it needs `/lib64/ld-linux-x86-64.so.2` which
distroless lacks.
## Origin
Lifted from #2619 — squash of `test(e2e): pre-pull cert-manager images`
→ POSIX rewrite → cert-manager removal (failed on distroless) →
reintroduction with the busybox helper that solves the distroless
problem. Final state only — the four iterative commits aren't
review-friendly individually.
## Verification
- `sh -n hack/e2e-prepull-images.sh` clean
- All chart-referenced images held Ready under the DaemonSet on local
kind v1.33.1 single-node
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Improved end-to-end test reliability for platform image pre-pulling by
rendering charts separately and consolidating extracted images.
* Simplified and consolidated readiness waits during tenant
configuration for more robust application readiness checks.
* **Chores**
* Enhanced image pre-pulling to reliably handle distroless images using
a staged helper binary and shared volume approach.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2724?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Annotate storageClass as immutable across every stateful app whose
chart exposes one — clickhouse, foundationdb, harbor, http-cache,
kafka (Kafka + ZooKeeper), kubernetes (top-level only), mariadb,
mongodb, nats, openbao, opensearch, postgres, qdrant, rabbitmq,
redis, vm-disk. Without coverage across the catalogue the UI would
render storageClass read-only on some apps and editable on others —
a worse mental model than uniform behaviour.
The annotation lands at the schema layer (CEL XValidation rule
'self == oldSelf' on values.schema.json / types.go / embedded
openAPISchema in cozyrds/<app>.yaml). The cozystack aggregated
apiserver does not yet evaluate CEL on application-backed resources;
that is tracked at cozystack/cozystack#2657. The UI
(cozystack/cozystack-ui#6) consumes the marker today.
The PVC-pinning argument is identical for every chart: Kubernetes
fixes a PVC's storageClassName at creation time, so editing this
field on an existing resource never migrated data. The schema
annotation makes that contract explicit.
kubernetes.nodeGroups[name].storageClass is intentionally NOT
annotated: the field is optional and undefaulted, so a strict
'self == oldSelf' rule would block any future attempt to set it on
an existing node group. A richer ratcheting form is needed there —
see docs/storage-immutability.md.
A refactor in pkg/registry/apps/application/rest.go extracts the
inline schema-build path into buildSpecSchema so it can be
unit-tested. Three regression tests in rest_spec_schema_test.go pin
the contract that x-kubernetes-validations survives the
v1->internal->structural conversion (loaded from the actual MariaDB
ApplicationDefinition fixture on disk, not a hand-edited mock).
Each per-app README gains a one-line blockquote pointing to a new
docs/storage-immutability.md that explains the contract and which
consumers enforce it today.
Requires cozystack/cozyvalues-gen with @immutable support — bump the
pre-commit pin in .github/workflows/pre-commit.yml once the release
exists.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Empty commit to fire the Pull Request workflow, which never ran for
this commit because GitHub Actions was down during the original push.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Empty commit to fire the Pull Request workflow, which never ran for
3bac5cac because GitHub Actions was down during the original push.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## Summary
- Replaces `img/cozystack-logo-black.svg` and
`img/cozystack-logo-white.svg` with the updated Cozystack brand mark
(icon + "COZYSTACK" wordmark).
- Adds new layouts so the logo can be used on more surfaces without
re-cropping:
- `cozystack-logo-stacked-{black,white}.svg` — icon stacked over the
wordmark
- `cozystack-icon-{black,white}.svg` — icon-only mark
- All SVGs are exported on a transparent background, so the same file
works on light and dark surfaces.
## Files
| Path | State | Notes |
|---|---|---|
| `img/cozystack-logo-black.svg` | replaced | horizontal wordmark, black
fill, transparent bg |
| `img/cozystack-logo-white.svg` | replaced | horizontal wordmark, white
fill, transparent bg |
| `img/cozystack-logo-stacked-black.svg` | new | icon over wordmark,
black fill, transparent bg |
| `img/cozystack-logo-stacked-white.svg` | new | icon over wordmark,
white fill, transparent bg |
| `img/cozystack-icon-black.svg` | new | icon only, black fill,
transparent bg |
| `img/cozystack-icon-white.svg` | new | icon only, white fill,
transparent bg |
## Test plan
- [ ] Verify rendering of `cozystack-logo-{black,white}.svg` in the
README on both light and dark GitHub themes.
- [ ] Confirm no other in-repo references to `img/cozystack-logo-*.svg`
need width/height adjustments (existing aspect ratio ~2.8:1 → new ~3:1,
very similar).
Replace the old "COZYΞTACK" wordmark with the updated brand mark
(icon + "COZYSTACK" wordmark) and add additional layouts:
- cozystack-logo-{black,white}.svg horizontal wordmark (replaced)
- cozystack-logo-stacked-{black,white}.svg icon over wordmark (new)
- cozystack-icon-{black,white}.svg icon only (new)
All variants use a transparent background so they work on both light and
dark surfaces. SVGs are optimized with https://autocrop.cncf.io while
keeping the original padding.
Signed-off-by: Viktoriia Kvapil <bisunviktoria@gmail.com>
Myasnikov Daniil
