admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-09 14:02:56 +09:00
Code Issues Packages Projects Releases Wiki Activity
5,081 Commits 568 Branches 294 Tags
e615ca925385a9e86c8a3e516bb345a583843883
Commit Graph
5081 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ivan Okhotnikov e615ca9253 chore(kubernetes): drop remaining nodeGroups residue from docs and fixtures 
Remove the orphaned kubernetes.nodeGroups doc comment, the stale kubelet
reservation / nodeGroups storageClass prose in the README, and the inert
nodeGroups blocks in the test value fixtures.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:56:43 +03:00
Ivan Okhotnikov 4619fb3d0c fix(kubernetes-nodes): make unpin failures visible and scope its RBAC 
Check the resource-policy annotation before removing it so a native pool
is a clean no-op while a real annotate failure fails the Job (retries
bounded, logs retained) instead of silently re-introducing the keep
leak. Scope the MD/MHC/WorkloadMonitor Role rules with resourceNames.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:56:43 +03:00
Ivan Okhotnikov fe958cb9ec test(kubernetes): provision e2e workers via KubernetesNodes 
The chainsaw harness created the Kubernetes CR with spec.nodeGroups,
which the control-plane-only chart now ignores, so no MachineDeployment
was created and the suites timed out. Create a KubernetesNodes CR named
<cluster>-md0 (roles: [ingress-nginx]) so the same MachineDeployment is
produced and the MD/ingress waits pass; this also gives the split its
first e2e coverage.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:56:43 +03:00
Ivan Okhotnikov 3d8617a02a fix(kubernetes): do not hard-fail when the ingress addon lacks a pool 
The render-time fail flipped existing degraded clusters from Ready to
Failed on upgrade, blocking all subsequent control-plane changes.
Restore the prior non-breaking behavior (the DaemonSet schedules nothing
if no ingress-roled node exists) and document the pool requirement in
NOTES.txt instead of enforcing it at render time.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:56:43 +03:00
Ivan Okhotnikov a76d947517 test(kubernetes-nodes): drop parity script from make test 
The cross-chart parity script was removed (the parent no longer renders
pools to diff against); the render snapshot under helm unittest is the
drift guard now and already runs in CI.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:56:43 +03:00
Ivan Okhotnikov 87b41b5fa8 chore(kubernetes): drop stale nodeGroups references from fixtures and docs 
Remove dead nodeGroups blocks from the CI values fixtures and point the
storage-immutability doc at the kubernetes-nodes per-pool storageClass.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:56:43 +03:00
Ivan Okhotnikov 7070cb0fe7 feat(kubernetes-nodes): unpin adopted worker objects on delete 
Migration 52 pins adopted worker objects with helm.sh/resource-policy=keep
for the transition window. Left in place it leaks the objects when an
adopted pool is deleted. Add a pre-delete hook that strips the annotation
from the release's own worker objects so Helm deletes them cleanly;
runs on delete only (never on upgrade, so it cannot race the parent
re-render) and is a no-op for natively-created pools.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:56:43 +03:00
Ivan Okhotnikov 82074cb9db fix(kubernetes): regenerate deepcopy after nodeGroups removal 
types.go dropped NodeGroups/NodeGroup/GPU/Kubelet/NodeHealthCheck but
the generated deepcopy still referenced them, so the api module failed
to build. Regenerated with controller-gen.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:56:43 +03:00
Ivan Okhotnikov a8286a3ac0 fix(platform): harden worker-pool adoption migration 
Anchor the KubevirtMachineTemplate sweep to the 6-hex content-hash
suffix so a pool name that is a prefix of a sibling (md0 vs md0-large)
no longer mis-adopts the sibling's KMT and deadlocks the migration.
Adopt the worker objects before creating the child HelmRelease so
helm-controller cannot start the child install into un-adopted objects
and fail Helm ownership validation. Emit the same install/upgrade
strategy (RetryOnFailure) and maxHistory cozystack-api sets, add a
pre-flight release-name length check, and document the rollback path.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:56:43 +03:00
Ivan Okhotnikov e96ea2b0f3 test(kubernetes-nodes): replace cross-chart parity with a render snapshot 
The parent no longer renders pools to diff against, so the one-shot
extraction parity check is replaced by a self-contained golden snapshot.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:55:15 +03:00
Ivan Okhotnikov 1e5ba15f6d test(kubernetes): update tests for the control-plane-only chart 
Drop pool-render tests; assert no pool objects render; add ingress-nginx
guard tests.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:54:25 +03:00
Ivan Okhotnikov 4bd79942f3 feat(kubernetes): require an ingress-nginx pool when the ingress addon is enabled 
With pools moved out of the chart the ingress-nginx addon can no longer
rely on an auto-provisioned md0; fail the render with a clear message
when addons.ingressNginx.enabled and no pool carries roles: [ingress-nginx].

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:54:25 +03:00
Ivan Okhotnikov 31336f0e18 feat(kubernetes)!: remove nodeGroups from the Kubernetes API 
Worker pools are now managed via KubernetesNodes resources; existing
pools are adopted by migration 52.

BREAKING CHANGE: the Kubernetes CR no longer accepts spec.nodeGroups.
Create KubernetesNodes resources for worker pools instead.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:54:25 +03:00
Ivan Okhotnikov 92b9785daa refactor(kubernetes): remove worker node pools from the chart 
The KubevirtMachineTemplate/MachineDeployment/MachineHealthCheck/worker
WorkloadMonitor, the preserve-old-KMT block, the talos-reconcile Job and
its RBAC now live in the kubernetes-nodes chart. The kubernetes chart
renders the control plane only.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:54:24 +03:00
Ivan Okhotnikov 86ecedb3c0 feat(platform): adopt worker pools into kubernetes-nodes releases (migration 52) 
Pre-upgrade migration for the Phase 2 kubernetes chart split. For every
parent kubernetes-<cluster> HelmRelease it creates one child
kubernetes-nodes-<cluster>-<pool> HelmRelease and re-annotates the
existing worker objects onto it with helm.sh/resource-policy=keep, so
the now control-plane-only parent upgrade does not prune the tenant's
running workers. Byte-identical child render (PR-2a golden parity) keeps
the KubevirtMachineTemplate content-hash stable, so no worker VM churns.
Idempotent and strict (exit 1 on any failure). Bumps targetVersion to 53
(contiguous with main; rebase to 53/54 if another 52 lands first).

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:52:05 +03:00
Ivan Okhotnikov b2e1707632 fix(kubernetes-nodes): drop dead ephemeralStorage guard, clarify parity scope 
The ephemeralStorage guard could never fire (the pool group is built from a
fixed set of chart values that never includes ephemeralStorage), so it read
as protection that is not there — remove it. Note in the parity script header
that only the four pool objects are compared, not the talos-reconcile Job.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:11 +03:00
Ivan Okhotnikov 326dbe0a5b test(kubernetes-nodes): cover GPU and kubelet pool shapes in render parity 
Extend the golden-parity matrix beyond the resources-sized pool to also
diff a GPU pool and a kubelet-override pool against the parent render, so
future drift in those branches (which would roll the content-hash-named
KubevirtMachineTemplate) is caught in CI.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:11 +03:00
Ivan Okhotnikov f21365c4f0 fix(kubernetes-nodes): address review nits in guard comment and NOTES 
Reference the adoption migration by role (Phase 2b) instead of a number
that does not exist on this branch, and correct the NOTES operational-note
count.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:11 +03:00
Ivan Okhotnikov 9e2f7cc953 test(kubernetes-nodes): run render parity in make test 
Wire tests/render-parity.sh into the chart's test target so CI verifies
the pool objects stay byte-identical to the parent render.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov 3d10a1c431 fix(kubernetes-nodes): keep applied TalosConfigTemplate byte-identical to the parent 
Restore the inline machineconfig comments (nameservers, GPU node labels)
inside the reconcile Job's data block so the applied TalosConfigTemplate
matches the parent's, avoiding drift on adoption.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov 73be7032a3 fix(kubernetes-nodes): drop unbacked cluster option source 
The cluster field referenced x-cozystack-options source 'kubernetes',
which has no provider (cozystack-api returns NotFound), so the dashboard
dropdown 404s. Drop the annotation so the field is a plain text input;
a real provider listing Kubernetes CRs is a follow-up.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov fe32fd8e58 fix(kubernetes-nodes): scope KMT preservation by release and guard foreign pool names 
Replace the collision-prone name-prefix match (pool 'gpu' matched
sibling 'gpu-large-<hash>') with a release-name annotation scope, and
add a render-time guard that fails cleanly when a pool's MachineDeployment
already exists under a different release (e.g. naming a pool after a
parent nodeGroup such as the default md0). Document the naming caveat in
NOTES.txt.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov 1e8d44995c fix(kubernetes): scope worker KMT preservation to the release 
The preserve-old-KubevirtMachineTemplate block re-emitted any KMT owned
by the CAPI Cluster and referenced by a MachineSet. A kubernetes-nodes
pool's KMT lives in the same namespace under the same Cluster, so the
parent would re-emit it with a foreign meta.helm.sh/release-name and its
next upgrade would fail Helm ownership validation. Restrict the block to
KMTs whose release-name annotation matches the parent release.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov e7947dd0e7 feat(platform): register kubernetes-nodes application 
Add the kubernetes-nodes-rd package (ApplicationDefinition), the
cozystack.kubernetes-nodes-application PackageSource (kubevirt variant,
kubernetes-nodes + kubernetes-nodes-rd components), and wire it into the
iaas bundle. Mirrors the vm-instance/vm-disk registration precedent.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov 277b32306c test(kubernetes-nodes): add unit tests and golden-parity check 
Unit tests cover naming/selectors, clusterName wiring, autoscaler
annotations, omitted replicas, bootstrap TCT reference, KMT hash name,
worker WorkloadMonitor, and the empty-cluster / mis-prefixed-release
render guards. render-parity.sh diffs the pool objects against the
parent kubernetes chart and asserts byte-identity.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov f842c0a17e feat(kubernetes-nodes): generate API types for KubernetesNodes 
Generated by cozyvalues-gen + controller-gen from values.yaml.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Ivan Okhotnikov fd14126bea feat(kubernetes-nodes): add worker node pool chart 
Introduce the kubernetes-nodes application chart: one HelmRelease per
worker node pool, attached to a parent Kubernetes cluster by name
convention (kubernetes-<cluster>). Renders the KubevirtMachineTemplate,
MachineDeployment, MachineHealthCheck, worker WorkloadMonitor and the
talos-reconcile Job with its RBAC, extracted from the monolithic
kubernetes chart. Pool objects are byte-identical to the parent's render
for the same group, so a later adoption never churns live worker VMs.

This is the additive half of the Phase 2 split (community#8); the parent
kubernetes chart is left unchanged.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
Andrey Kolkov aef9e201c7 fix(ci): make workflow cancellation effective (#3401) 
## What this PR does

- allows superseded PR workflow runs to cancel `finalize` and `E2E
Tests` instead of continuing because of job-level `always()`
- makes `update-website-docs` cancellation-responsive while preserving
its behavior when changelog generation fails or is skipped
- retains step-level `always()` conditions for reports and sandbox
cleanup

### Screenshots

Not applicable.

### Release note

```release-note
fix(ci): ensure superseded and manually canceled workflows stop long-running jobs
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Prevented release and validation-related jobs from running when a
workflow is cancelled.
* Updated pull request check, end-to-end testing, and stable
documentation update conditions to avoid executing after cancellation.
* Kept existing success-gating so follow-up tasks still require
prerequisite steps to complete successfully.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-21 20:52:02 +04:00
Andrey Kolkov 64097dc455 fix(platform): forward backupStorage overrides to the backupstrategy-controller Package (#3333) 
## What this PR does

Fixes #3245.

The documented admin override for the `cozy-default` backup S3
coordinates — `spec.components.backupstrategy-controller.values` on the
`Package cozystack.cozystack-platform`
(`docs/operations/backup-classes.md`, "Admin overrides for
`cozy-default`") — had no effect: the `cozystack-platform` PackageSource
exposes exactly one component (`platform`), so the operator silently
ignored the block, and the platform chart rendered the child `Package
cozystack.backupstrategy-controller` via the `.default` helper with no
components argument, so nothing forwarded `backupStorage` either. Admins
running against an external or TLS S3 had no working supported path to
set `backupStorage.endpoint` / `region` / `systemSecretName`.

This PR wires the override through the platform chart, the same way
`bundles/iaas.yaml` forwards kubevirt and gpu-operator values:

- `packages/core/platform/values.yaml` gains a top-level `backupStorage:
{}` value.
- `templates/bundles/system.yaml` renders it into the emitted `Package
cozystack.backupstrategy-controller` as
`components.backupstrategy-controller.values.backupStorage`, from where
the cozystack operator merges it into the `backupstrategy-controller`
HelmRelease over the chart defaults. When `backupStorage` is unset the
components block is omitted and the emitted Package is byte-identical to
the historical render.
- `docs/operations/backup-classes.md` now documents the working path
(`spec.components.platform.values.backupStorage`) and explicitly calls
out the two plausible-looking paths that do not work (the old documented
one, and patching the child Package directly, which the platform
helm-reconcile reverts).
- A new helm-unittest suite
(`packages/core/platform/tests/bundles_backupstorage_wiring_test.yaml`)
is the behavioural regression test: it proves the unset case emits no
components block and the set case lands the values on the child Package,
on both `isp-full` and `isp-full-generic`.

Verified with `make test` in `packages/core/platform` (22 suites, 91
tests, all green). `make show` requires a live cluster (`cozyhr` +
`lookup` in `templates/repository.yaml`), so the unittest render of
`templates/bundles/system.yaml` is the offline verification path.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#622
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Walking the trigger map against the diff: the change to
`packages/core/platform/values.yaml` reaches **cozystack/website** twice
— `content/en/docs/next/operations/configuration/platform-package.md`
(hand-written table of `spec.components.platform.values.*` keys gains
`backupStorage`) and
`content/en/docs/next/operations/services/backup-classes.md` (mirrors
the broken override example this PR corrects). The follow-up is
cozystack/website#622, which also carries the superseded website#547
content and covers both next/ and v1.4/ docs, marked not to merge before
this PR lands; its v1.4 copy assumes this fix is backported to
release-1.4. No other downstream repository restates the touched files:
the templates change is chart-internal, no app `values.schema.json`,
CRD, variant list, `hack/` file, or namespace changed.

### Release note

```release-note
fix(platform): the documented admin override for the cozy-default backup S3 coordinates now works — set spec.components.platform.values.backupStorage on the cozystack.cozystack-platform Package and the platform chart forwards it to the backupstrategy-controller component (previously the documented spec.components.backupstrategy-controller path was silently ignored)
```
 2026-07-21 19:30:38 +04:00
myasnikovdaniil 3ad3c2e611 fix(kubernetes): move kubevirt-csi-driver base off quay.io/centos to almalinux (#3400) 
## What this PR does

`packages/apps/kubernetes/images/kubevirt-csi-driver/Dockerfile` pinned
its runtime stage to `quay.io/centos/centos:stream9` by digest. Quay
serves **only currently-tagged manifests** — when a rolling tag moves,
the superseded digest immediately returns `MANIFEST_UNKNOWN`, with no
grace period.

`stream9` rolls roughly every six days, so the pin has already died
twice:

| Date | Fix | Digest pinned | Status today |
|---|---|---|---|
| Jul 15 | #3303 | `0c9db821…` → `a93e3316…` | `0c9db821` gone |
| Jul 21 | #3386 | `a93e3316…` → `3714c89f…` | `a93e3316` gone |

Each death broke `make build` on `main` and on every PR triggering a
full rebuild. Because the root `build` target is serial, a failure in
`packages/apps/kubernetes` (4th of 29) aborts the remaining packages and
leaves the shared build cache unwritten.

Repinning cannot fix this: the freshly pinned digest is always the
current live tag, so it is next in line to be retired. This is not
specific to CentOS — the same probe against
`quay.io/fedora/fedora:latest` and `quay.io/centos/centos:stream10`
found every historical digest unfetchable.

This PR keeps the digest pin and changes the *source* to a registry that
provably retains superseded manifests, making the pin durable rather
than decorative. Renovate policy is unchanged — `pinDigests` stays on,
and `almalinux:9` is an ordinary rolling tag on a retaining registry
exactly like the `golang`, `alpine` and `debian` bases already in the
grouped digest PR.

### Verification

Both bases were run locally with the Dockerfile's exact install line
(`dnf install -y e2fsprogs xfsprogs nfs-utils`):

| Binary | CentOS Stream 9 | AlmaLinux 9 |
|---|---|---|
| `mkfs.ext4` | `/usr/sbin/mkfs.ext4` | `/usr/sbin/mkfs.ext4` |
| `mkfs.xfs` | `/usr/sbin/mkfs.xfs` | `/usr/sbin/mkfs.xfs` |
| `blkid` | `/usr/sbin/blkid` | `/usr/sbin/blkid` |
| `mount` | `/usr/bin/mount` | `/usr/bin/mount` |
| `umount` | `/usr/bin/umount` | `/usr/bin/umount` |
| `fsck.ext4` | `/usr/sbin/fsck.ext4` | `/usr/sbin/fsck.ext4` |
| `fsck.xfs` | `/usr/sbin/fsck.xfs` | `/usr/sbin/fsck.xfs` |
| `mount.nfs` | `/usr/sbin/mount.nfs` | `/usr/sbin/mount.nfs` |

Every path `k8s.io/mount-utils` shells out to is identical. Package
versions:

| Package | CentOS Stream 9 | AlmaLinux 9 |
|---|---|---|
| `e2fsprogs` | 1.46.5-8.el9 | 1.46.5-8.el9 |
| `xfsprogs` | 6.4.0-7.el9 | 6.4.0-7.el9 |
| `nfs-utils` | 2.5.4-44.el9 | 2.5.4-42.el9_8 |
| `util-linux` | 2.37.4-26.el9 | 2.37.4-25.el9 |
| `glibc` | 2.34-274.el9 | 2.34-270.el9_8 |

Same upstream versions throughout; the el9 release-number lag is
expected since AlmaLinux is downstream of RHEL 9 while CentOS Stream 9
is upstream. `e2fsprogs` and `xfsprogs` — the two that govern
`mkfs`/`fsck` behaviour — are identical NVRs. `glibc` is not linked at
all: the driver is built `CGO_ENABLED=0`.

The pinned digest is the OCI image index covering `amd64`, `arm64`,
`ppc64le` and `s390x` — the same architecture set as the previous base.

Retention was verified empirically rather than assumed: four superseded
`docker.io/library` digests taken from this repo's own git history
(`golang:1.26`, `golang:1.26-alpine`, `alpine:3.24`, `alpine:3.22`) all
still resolve after their tags moved.

### Scope

This addresses the immediate recurring breakage only. The broader
concern in #3305 — that any retired digest blocks the entire platform
build, with no advance warning and no fast path to repin — is not solved
here and remains open.

### Release note

```release-note
fix(kubernetes): move the kubevirt-csi-driver runtime base from quay.io/centos/centos:stream9 to docker.io/library/almalinux:9. Quay retires superseded manifests as soon as a rolling tag moves, which broke builds twice in six days; AlmaLinux 9 provides an equivalent RHEL 9 userspace on a registry that retains them, so the digest pin remains valid over time. No functional change to the driver.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated the container runtime environment to use AlmaLinux 9 instead
of CentOS Stream 9.
* Kubernetes CSI driver behavior and startup configuration remain
unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-21 19:24:11 +05:00
Myasnikov Daniil a4b5b9d57b docs(release): update workflow cancellation condition 
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 18:41:45 +05:00
Myasnikov Daniil 792951b9c6 fix(ci): make workflow cancellation effective 
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 17:57:48 +05:00
Myasnikov Daniil 14af87296f fix(kubernetes): move kubevirt-csi-driver base off quay.io/centos to almalinux 
quay.io serves only currently-tagged manifests: when a rolling tag moves,
the superseded digest immediately returns MANIFEST_UNKNOWN. The pinned
`centos:stream9` digest has died twice in six days (#3303, #3386), each
time breaking `make build` on main and on every PR that triggers a full
rebuild.

Repinning is not a fix -- the newly pinned digest is always the live tag,
so it is next in line to be retired. Move the runtime stage to a registry
that provably retains superseded manifests instead, which makes the digest
pin durable rather than decorative. Pinning policy is unchanged.

Verified equivalent: after `dnf install -y e2fsprogs xfsprogs nfs-utils`,
both bases provide mkfs.ext4, mkfs.xfs, blkid, mount, umount, fsck.ext4,
fsck.xfs and mount.nfs at identical paths. e2fsprogs and xfsprogs are
identical NVRs. The driver is built CGO_ENABLED=0, so the base libc is not
linked. The pinned digest is the OCI index covering amd64, arm64, ppc64le
and s390x -- the same architecture set as the previous base.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 17:56:11 +05:00
myasnikovdaniil 741a5acc4a fix(opensearch-operator): use RollingUpdate maxSurge=0 instead of Recreate (#3319) 
## What this PR does

Fixes a Helm-upgrade failure in the `opensearch-operator` chart that
blocks the new release-upgrade E2E lane (added in #3276).

Upgrading from a chart that shipped no explicit strategy (v1.5.x) to a
v1.6.0-rc build fails validation immediately, helm-controller retries
it, and the HelmRelease never becomes Ready:

```
Helm upgrade failed for release cozy-opensearch-operator/opensearch-operator:
Deployment.apps "opensearch-operator-controller-manager" is invalid:
spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is Recreate
```

Observed in the ["Upgrade E2E Test" job of run
29480724982](https://github.com/cozystack/cozystack/actions/runs/29480724982)
— `UpgradeFailed ... (x19 over 11m)`, with the HelmRelease never
reaching Ready.

### Root cause

#3040 (commit db797b1d) set the single-replica operator Deployment to
`strategy.type: Recreate` — a reasonable goal: with leader election
gated off on one replica, a default `RollingUpdate` briefly runs two
uncoordinated managers (`maxSurge`) during a rollout, and `Recreate`
closes that window.

The problem surfaces only on **upgrade from an older chart, under
server-side apply**:

- A v1.5.x release shipped no strategy at all, so the apiserver
**defaulted** `spec.strategy.rollingUpdate` to `25%/25%`, and the
`helm-controller` field manager never owned that block. Its
`managedFields` entry covers `f:replicas`, `f:selector` and
`f:template`, but not `f:strategy`.
- helm-controller applies **server-side** by default
(`Install.ServerSideApply` defaults to true, `Upgrade.ServerSideApply`
to `auto`; `internal/operator/package_reconciler.go` sets neither), and
SSA does not remove a field the applier never owned merely because the
new intent omits it.
- So the v1.6.0-rc manifest's `type: Recreate` merges onto a live object
that still carries the defaulted `rollingUpdate`, and the apiserver
rejects the result with `FieldValueForbidden`.

The **client-side** path is unaffected, which is why this surfaces
through helm-controller's SSA but not a local `helm upgrade`:
`DeploymentSpec.Strategy` carries `patchStrategy:"retainKeys"`
(`k8s.io/api@v0.34.1/apps/v1/types.go:395`), so a 3-way merge emits
`{"spec":{"strategy":{"$retainKeys":["type"],"type":"Recreate"}}}`,
which merges to a valid `{"strategy":{"type":"Recreate"}}` and does
clear the block.

This is a genuine product upgrade bug, not a test-harness issue — #3276
only adds the upgrade lane that exposes it; it does not touch
`opensearch-operator`.

> An earlier revision of this PR attributed the failure to Helm's
client-side 3-way merge. That was wrong, and the correction is
@lexfrei's — see [his
review](https://github.com/cozystack/cozystack/pull/3319#pullrequestreview-4713156867).
The mechanism above is now reproduced end to end (below).

### The fix

Keep `type: RollingUpdate` on the single-replica path and set
`rollingUpdate.maxSurge: 0 / maxUnavailable: 1` instead of switching to
`Recreate`:

- `maxSurge: 0` caps total pods at `replicas`, so the deployment
controller must scale the old manager down before it can scale the new
one up — unlike the default 25% surge, which starts a second manager
while the first is still fully live.
- This is **not** `Recreate`'s exclusivity, and the PR no longer claims
it is. `rolloutRecreate` gates scale-up on `oldPodsRunning()`, which
counts non-terminal `Status.Phase`, so `Recreate` waits for the old pod
to be gone. `rolloutRolling` has no such gate: `NewRSNewReplicas`
derives `currentPodCount` from `GetReplicaCountForReplicaSets`, which
sums `rs.Spec.Replicas` (desired, not live). The ordering that is
guaranteed is over ReplicaSet desired counts, not pod lifecycles, so a
draining old manager can still overlap the new pod within
`terminationGracePeriodSeconds` (10s here).
- Because `strategy.type` never becomes `Recreate`, the forbidden merge
cannot occur on any apply path. It also recovers anyone already on an rc
build with `Recreate`, since `Recreate -> RollingUpdate` is permitted.
- This matches the idiom already vendored in this repo:
`packages/system/kubeovn/charts/kube-ovn/templates/central-deploy.yaml:11-15`
uses `RollingUpdate` with `maxSurge: 0 / maxUnavailable: 1`.

Multi-replica installs (`manager.replicaCount > 1`) are unchanged: they
keep the default `RollingUpdate` with leader election enabled.

The change is carried in `patches/leaderElection.diff` so it survives a
chart re-vendor via `make update`; the patch applies to pristine
upstream 2.8.0 with no fuzz and reproduces the committed strategy block
exactly.

### Alternatives considered

| Option | Why not |
| --- | --- |
| Keep `Recreate`, but make the chart own `rollingUpdate` first (ship an
explicit block, then switch `type` in a later release) | This is the
only option that preserves real exclusivity, but it needs two releases
and only works once every install has reconciled the intermediate one.
Disproportionate for a single-replica operator whose residual overlap is
a draining manager. |
| Pre-upgrade hook / Job that patches or deletes the Deployment strategy
before the upgrade | Heavy: needs a kubectl image, a ServiceAccount and
RBAC to `patch`/`delete` Deployments, and hook ordering; runs on every
upgrade even when unneeded; adds standing attack surface —
disproportionate for a scalar field transition. |
| `helm.sh/resource-policy` / force-replace | `resource-policy: keep`
only governs deletion on uninstall (irrelevant). Helm has no
per-resource "force recreate" annotation; `--force` / HelmRelease
`spec.upgrade.force` is global and disruptive and not controllable from
within the chart. |
| Plain revert to default `RollingUpdate` (25%/25%) | Fixes the
rejection but starts a surge pod while the old manager is still fully
live — the window #3040 set out to close. `maxSurge: 0` narrows it to a
draining old pod without leaving `RollingUpdate`. |

### Verification

The mechanism and the fix were reproduced on kind v1.33.1 (a bare
apiserver is enough — the failure is defaulting + SSA + validation):

| Path | Result |
| --- | --- |
| SSA install of the v1.5.x manifest (no strategy) | apiserver defaults
`rollingUpdate: 25%/25%`; `managedFields` shows `helm-controller` does
not own `f:strategy` |
| SSA upgrade to `type: Recreate` | fails with the exact production
error above |
| Client-side apply of the same pair | succeeds; `rollingUpdate` is
cleared, live strategy becomes `{"type":"Recreate"}` |
| SSA upgrade to `maxSurge: 0` (this PR) | succeeds |
| SSA of `maxSurge: 0` onto a live `Recreate` Deployment (rc recovery) |
succeeds |
| Rollout under `maxSurge: 0` vs `Recreate` | `maxSurge: 0`: two pods
coexisted within 1s (old Terminating, new ContainerCreating).
`Recreate`: one pod, waited the full drain. This is the evidence for the
"not exclusivity" wording above. |

### Note on regression coverage

`tests/leader_election_test.yaml` was never executed:
`hack/helm-unit-tests.sh` runs a package's suite only when the package
Makefile defines a `test:` target, and this one did not — so the
coverage the previous revision of this PR claimed did not exist. This PR
adds the target (matching `packages/system/etcd-operator/Makefile`),
which revives the suite; the repo-wide runner now picks the package up
and a revert to `Recreate` fails it. Thanks to @lexfrei for catching
that.

The apiserver-side merge that produces the invalid object still needs a
live cluster, so the behavioural regression test remains the
release-upgrade E2E lane in #3276 (which this fix unblocks).

### Follow-ups filed

- #3324 — `packages/system/ouroboros` has the same latent shape
(`Recreate` gated on `controller.mode == "external-dns"`, while
`controller.mode` defaults to `coredns`). Not fixed here, and it likely
needs a different fix, since that mode genuinely depends on exclusivity.
- #3325 — `make update` does not reproduce this package's vendored
template: three hand-edits are not captured in `patches/`, and one of
them breaks rendering when `manager.extraEnv` is set. Pre-existing on
`main`; an earlier revision of this body claimed the regeneration was
exact, which was wrong.

### Downstream repositories

- [x] No downstream repository is affected by this change

<!-- Walked the trigger map in docs/agents/contributing.md against the
diff: the change touches only a vendored packages/system operator chart
template + its patch + its helm-unittest suite + the package Makefile
test target. No packages/apps or packages/extra add/rename/remove, no
values.schema.json / enum / default change, no node contract, no
annotations/labels, no hack/ or ApplicationDefinition change.
opensearch-k8s-operator is a third-party upstream (not a cozystack
downstream), and the fix lives in patches/leaderElection.diff so it
survives make update. -->

### Release note

```release-note
fix(opensearch-operator): fix a Helm upgrade failure on single-replica installs. Upgrading a release that predates an explicit Deployment strategy failed under server-side apply, because switching strategy.type to Recreate does not clear the apiserver-defaulted spec.strategy.rollingUpdate block that the chart never owned, and the merged object is rejected. The operator now uses RollingUpdate with maxSurge=0/maxUnavailable=1, which keeps the transition legal on every apply path and recovers releases already stuck on Recreate.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Updated OpenSearch Operator controller rollout behavior for
single-replica installs to use a safer rolling update configuration.
* Leader election is now applied only when multiple replicas are
configured, reducing the chance of conflicting controller activity.
* **Tests**
* Expanded Helm unittest assertions for the controller deployment
rollout strategy and leader-election behavior for single- and
multi-replica scenarios.
* **Chores**
  * Added a `make test` target to run Helm unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-21 17:17:03 +05:00
myasnikovdaniil 960d853706 feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) (#3273) 
## What

Adds typed `waitStrategy` + `healthCheckExprs` fields so a package/app
can declare the Flux helm-controller v1.5.0 kstatus health gate for the
custom resource it wraps — making `HelmRelease.Ready` mean "the CR is
actually healthy" instead of "helm applied the CR". This is the
operator/apiserver foundation of the kstatus epic (#2642); the API
surface was reviewed and approved by the API owner in #3263.

Commits (per the review's "isolated step-0" request):
1. `build(deps)` — the dependency bump (see below).
2. `feat(api)` — the fields, plumbing, tests, and regenerated code.
3. `fix(api)` — match the openapi code-generator pin to the 0.35 runtime
(fixes an aggregated-apiserver crash the bump surfaced).
4. `fix(api)` — declare `OpenAPIModelName` on the apps types so SSA
resolves catalog kinds.
5. `docs(api)` — note that `disableWait` silently defeats
`healthCheckExprs`.

## API surface

Typed fields on **both** HelmRelease-building paths:
- `ApplicationDefinition.spec.release` (apiserver / catalog apps →
`convertApplicationToHelmRelease`)
- `ComponentInstall` (operator / platform packages →
`buildHelmReleaseSpec`)

```go
// WaitStrategy maps to HelmReleaseSpec.WaitStrategy.Name — a deliberate scalar
// simplification of the upstream {name} object. One of poller|legacy.
WaitStrategy string
// HealthCheckExprs maps to HelmReleaseSpec.HealthCheckExprs (the upstream
// kustomize.CustomHealthCheck type verbatim).
HealthCheckExprs []kustomize.CustomHealthCheck
```

`config.ResolveWaitStrategy` is shared by both builders and **couples
the default**: `healthCheckExprs` are only evaluated under the `poller`
strategy, so when expressions are set and no strategy is given, the
generated HelmRelease defaults to `poller`. A package that sets only
`healthCheckExprs` is therefore self-contained, independent of the
controller's global default.

## The dependency bump (commit 1)

The `WaitStrategy` / `kustomize.CustomHealthCheck` /
`HelmReleaseSpec.HealthCheckExprs` types do not exist below
`helm-controller/api` v1.5.0 — the runtime is already v1.5.0 (shipped by
#2602), so this closes a Go-types-vs-runtime skew. It is **not** an
isolated bump: v1.5.x requires **k8s 0.35 + controller-runtime 0.23**,
cascading the k8s stack 0.34 → 0.35 (and the go directive to 1.26).

cozystack replaces `k8s.io/apimachinery` with a fork carrying the
still-open upstream `ConvertToVersion` patch (k/k#135537). That single
commit rebases cleanly onto apimachinery v0.35.0; published at
`cozystack/apimachinery@release-1.35-pr135537`, and `go.mod` pins its
pseudo-version. **The repo builds with zero source changes under the new
stack.**

> Maintainer note: please confirm the `release-1.35-pr135537` fork
branch naming, or fold it into the `cozystack` branch as you prefer —
the pin can be updated to match.

## Not included (conscious YAGNI)

`disableWait` is intentionally omitted. The only known need (the
kubernetes tenant-addon deadlock) was resolved child-side, and
`waitStrategy: legacy` is not equivalent (legacy still waits
Helm-v3-style; `disableWait` skips waiting entirely). Add it later when
a concrete leaf needs a true no-gate escape hatch.

## Tests

- `config.TestResolveWaitStrategy` — the couple-the-default logic
(unset+exprs→poller; explicit legacy honored; unset+no-exprs→nil).
- Operator + apiserver cases mirror each other
(`TestBuildHelmReleaseSpecHealthCheckExprs` /
`TestConvertApplicationToHelmRelease_HealthCheckExprs`).
- `go build ./...` clean, `go vet` clean, `gofmt` clean, codegen
regenerated and committed.

The 4 backupstrategy CRDs regenerate with benign k8s-0.35 PodSpec
description drift (DynamicResourceAllocation alpha→stable, toleration
Lt/Gt, resizePolicy note) — no functional change.

## Follow-up

Phase 3 (per-leaf adoption — postgres/kafka/mongodb/etc. declaring their
CEL via these fields) builds on this. The full CRD→CEL catalog is
maintained in the epic's readiness audit.

Refs #2642

```release-note
feat(api): add `waitStrategy` and `healthCheckExprs` to ApplicationDefinition and ComponentInstall, so a HelmRelease reports Ready only once the custom resource it wraps is actually healthy
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added configurable CEL health-check expressions for application
releases and package components.
* Added `waitStrategy` support (`poller` and `legacy`) with automatic
`poller` selection when health checks are configured and no strategy is
set.
* Propagated these settings into generated Helm releases, affecting
readiness/wait behavior.

* **Documentation**
* Updated Application and PackageSource schemas to include
`healthCheckExprs` and `waitStrategy`.
* Extended related Kubernetes CRD documentation, including projected pod
certificate annotations and workload reference fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-21 17:02:37 +05:00
Andrey KolkovandClaude Fable 5 7d117679d4 docs(backups): document bucketName's two modes, cover isp-hosted in the wiring test 
Branch-review on this PR: for provisionBucket: false the
backupstrategy-controller.bucketName helper returns
.Values.backupStorage.bucketName verbatim as the real S3 bucket name
and every strategy CR plus the Velero BSL bakes it in, so the
external-S3 section must require updating bucketName (the Secret's
bucketName key alone is not read by the strategies), and the knob
table must state the two-mode semantics instead of the COSI-only one.
Also route the external-S3 instruction through the platform component
override instead of naming raw chart values, and add the isp-hosted
variant case to the forwarding test (92 tests green).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 0cd3fe20b8 docs(backups): credentials are projected for RestoreJob reconciles too 
The projector runs from both the BackupJob and RestoreJob controllers
(restorejob_controller.go), not only before BackupJobs — keep the
monorepo doc aligned with the website copy (cozystack/website#622).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 93de6adc04 docs(backups): fix HelmRelease interval, FDB API group, BSL wait syntax, leader-election phrasing 
Same corrections as the website copy (cozystack/website#622 round 4):
the re-render interval is 5m via the operator's helmrelease-interval
flag (not a 10-minute Flux default), the FoundationDB backup CR group
is apps.foundationdb.org, the BSL readiness check uses valid kubectl
wait --for=jsonpath syntax, and the projector's first synchronous round
is gated on leader election rather than pod readiness.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 a5dff22c51 docs(backups): fix PromQL examples and bucket-name phrasing, document namespace knob and bslEnabled gap 
Keep the monorepo doc aligned with the website copy
(cozystack/website#622 branch-review round 3): full
cozystack_backup_credentials_projection_* metric names with range
vectors in the alerting examples, the intro no longer conflates the
Bucket CR name with the COSI-assigned S3 bucket name, the knob table
gains the backupStorage.namespace row, and the external-S3 section
states that the Velero BSL derives from the same backupStorage block
while velero.bslEnabled is not carried by the override path.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 b4c1e44fd5 docs(backups): namespace the flux reconcile example 
The backupstrategy-controller HelmRelease lives in
cozy-backup-controller (packages/core/platform/sources/
backupstrategy-controller.yaml), so the un-namespaced command fails
out of the box.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 e74c8cb459 docs(backups): pin the placeholder-credentials upgrade note to v1.5 
Same note as the website copy (cozystack/website#622 review round 2):
name the explicit version boundary instead of 'pre-PR / After this PR',
so the two documents cannot diverge — the empty-string defaults and
useSystemBucket shipped in release-1.5.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 1422f8b78a fix(platform): forward backupStorage overrides to the backupstrategy-controller Package (#3245) 
The documented admin override for the cozy-default backup S3
coordinates (spec.components.backupstrategy-controller.values on the
cozystack.cozystack-platform Package) had no effect: that PackageSource
exposes only the platform component, so the operator silently ignored
the block, and nothing in the platform chart forwarded backupStorage to
the child Package.

Wire it the same way bundles/iaas.yaml forwards kubevirt / gpu-operator
values: a new top-level backupStorage value on the platform chart is
rendered into the emitted cozystack.backupstrategy-controller Package
CR as components.backupstrategy-controller.values.backupStorage, from
where the cozystack operator merges it into the
backupstrategy-controller HelmRelease over the chart defaults. When
backupStorage is unset the emitted Package is unchanged.

Update docs/operations/backup-classes.md to the working override path
(spec.components.platform.values.backupStorage) and call out the two
plausible-looking paths that do not work. Add a helm-unittest suite
covering the no-op default and the forwarding on isp-full and
isp-full-generic.

Fixes #3245

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-21 14:47:08 +04:00
Andrey Kolkov e73ad68a85 fix(backups): support cnpg-operator with barman plugin (#3313) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->


## What this PR does

Fixes #3300. Migrates CNPG Postgres backups off the deprecated native
`spec.backup.barmanObjectStore` (removed in CNPG 1.29; the
`barman-cloud-*` binaries it needs are absent from the `standard` image
variant keycloak-db pins, and the bare system-flavor tags apps/postgres
pins are deprecated upstream) onto the [CloudNativePG Barman Cloud
plugin](https://github.com/cloudnative-pg/plugin-barman-cloud), across
every Postgres backup surface: `apps/postgres`, `system/keycloak`, and
the `cozy-default-cnpg` BackupClass strategy driver. The chart renders
`spec.plugins` + a `barmancloud.cnpg.io/ObjectStore` CR; the
backup-controller driver SSA-applies an `ObjectStore` and patches
`spec.plugins`; `Backup`/`ScheduledBackup` use `method: plugin`. The
`postgres-operator` chart vendors and co-locates the
`plugin-barman-cloud` operator (with its cert-manager-issued mTLS) so
CNPG can discover it.

One platform-level change is required for the plugin to run and ships
with this PR (see the upgrade notes in
`docs/operations/backup-classes.md` and the release note below): the
CloudNativePG operator is bumped to 1.28.1 (also fixing an operator/CRD
version skew that broke all backups). The other prerequisite — the
LINSTOR scheduler admission webhook stripping
`initContainers[].restartPolicy` cluster-wide, which broke the plugin's
restartable sidecar — is already fixed on `main` (f4aef7305 re-vendors
`linstor-scheduler` to upstream chart 0.3.1 / extender v0.3.6, which
keeps the webhook and stops stripping unknown pod fields), so after the
rebase this PR no longer touches `linstor-scheduler`; just do not hold
that package back on an older version when rolling out the plugin.

Supersedes #3306 (the keycloak `system`-image workaround): keycloak-db
now backs up via the plugin on the `standard` image, so that stopgap is
no longer needed.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: docs/operations/backup-classes.md changed; a website mirror
follow-up is warranted (not yet opened).
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): migrate CNPG Postgres backups to the CloudNativePG Barman Cloud plugin

Postgres backups (apps/postgres, system/keycloak, and the cozy-default-cnpg BackupClass strategy) now use the CloudNativePG Barman Cloud plugin (spec.plugins + a barmancloud.cnpg.io/ObjectStore, method: plugin) instead of the deprecated native spec.backup.barmanObjectStore (removed in CNPG 1.29; the barman-cloud-* binaries it needs are absent from the standard image variant keycloak-db pins, and the bare system-flavor tags apps/postgres pins are deprecated upstream). The barman-cloud plugin operator is installed alongside the CNPG operator.

One platform-level change rides along and requires operator attention on upgrade:
- The CloudNativePG operator is bumped to 1.28.1 (fleet-wide minor upgrade); this also fixes an operator-newer-than-its-CRDs skew that made the operator report "instance manager was restarted during backup" and fail every backup. The vendored chart ships the CNPG CRDs as ordinary release manifests (crds.create), so helm upgrade updates them with the operator; the skew came from the image.tag pin outrunning the chart's CRDs, which this PR removes.
- The plugin's restartable sidecar requires the linstor-scheduler admission webhook that no longer strips initContainers[].restartPolicy (upstream chart 0.3.1 / extender v0.3.6, shipped separately on main); do not hold the linstor-scheduler package back on an older version when rolling out the plugin.
```

Close #3300, #3246

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Switched PostgreSQL CNPG backups/restores to the plugin-based Barman
Cloud workflow, using dedicated ObjectStore resources for backup and
recovery.
* Added the Barman Cloud CNPG-I plugin chart (deployment, TLS, RBAC,
leader election).
* **Bug Fixes**
* Improved backup/restore attachment messaging when the CNPG Cluster
isn’t ready yet.
* Preserves server-name behavior to avoid WAL-archive prefix issues;
clarified endpoint CA handling.
* **Changes**
* Helm integration no longer uses the legacy
`spec.backup.barmanObjectStore` path; updated gating, credential
projection, and ObjectStore-related RBAC/cleanup.
* Removed the linstor-scheduler admission webhook; updated
CloudNativePG/operator versions and CRDs.
* **Tests**
* Updated controller and Helm-unittest checks for the plugin flow;
added/extended end-to-end backup/restore coverage.
* **Documentation**
* Expanded upgrade/migration and backup documentation; updated backup
example materials and comments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-21 14:45:52 +04:00
myasnikovdaniil cc66e22952 fix(ci): mirror and retag images whose digest is embedded in the values tag (#3375) 
## What's broken

`collect_refs` in both `hack/nightly-mirror.sh` and
`hack/promote-retag.sh` recognises three values shapes. The one most
package Makefiles actually write is not among them:

```yaml
image:
  repository: <registry>/linstor-csi
  tag: main@sha256:<digest>      # digest embedded in the tag
```

Package Makefiles set `.image.tag` to `"$(IMAGE_TAG)@$(digest)"` in a
single `yq` call rather than maintaining a separate `digest` key. That
matches neither the `{repository, tag, digest}` rule (no `digest` key)
nor, usefully, the single-string rule — that one matches the bare tag
value, which carries no repository, so `ref_repo()` reduces
`main@sha256:...` to the repository `main` and the ownership filter
drops it.

Being dropped is silent. Both scripts then report success having simply
never seen those refs.

The same emit-then-drop hits a second class, found in review: refs whose
registry host does not live inside `repository` at all.
`keycloak-operator` splits it into a sibling `registry` key, and
`kubeovn` keeps it in the document-level `global.registry.address` with
a bare `repository: kubeovn`. Both are built and pushed to `$REGISTRY`
by cozystack — `kubeovn` by the `cozystack/kubeovn-chart` wrapper repo,
whose own `make image` writes all three keys — and both were emitted
host-less and discarded as third-party.

## Impact

**Nightly (`nightly-mirror.sh`) — this is the one that broke.** The
image is never copied to GHCR, *and* the unconditional host rewrite at
the end of the script still repoints its repository at the destination
registry. The result is a published reference to an image that was never
pushed there.

Confirmed against the `:nightly` tag on GHCR — the two packages using a
recognised shape are present, and eight images across six packages using
this shape are absent:

| shape | packages | `:nightly` |
|---|---|---|
| `{repository,tag,digest}` | cilium | present |
| single-string full ref | cozystack-operator | present |
| `{repository, tag:<t>@sha256}` | kamaji, kilo, linstor-csi,
piraeus-server, linstor-gui, metallb-controller, metallb-speaker,
redis-operator | **absent** |

This surfaced as the first full nightly run failing at install: the
`e2e-image-prepuller` DaemonSet never became ready (0/3, 10-minute
timeout) because `linstor-csi` and `piraeus-server` 404 on GHCR. The
prepuller only templates kubeovn/linstor/cert-manager, so it exposed two
of the eight; the other six would have failed later at HelmRelease
install.

**Promotion (`promote-retag.sh`) — same blind spot, milder
consequence.** This retags within a single registry, so the digests
already exist and digest-pinned installs still resolve — there is no
dangling reference and no 404. What is missing is the `:<version>` tag
(and `:latest` under `MOVE_LATEST=1`) for those images, so a promoted
release does not carry the tags it claims to.

The registry shows how long that has been true for the two host-split
images. `keycloak-operator` carries exactly one tag, `latest`, though
the package has shipped since 2024-11. `kubeovn`'s newest
cozystack-versioned tag is `v1.14.11-v0.38.8`, from before the 1.0
versioning switch — while `cilium`, which uses an already-recognised
shape, carries `v1.5.0` through `v1.6.0-rc.2`. Neither has received a
tag from any 1.x release.

Measured against the committed tree, the dry-run selects **19 unique
repositories before this change and 29 after** — the eight digest-in-tag
images plus `keycloak-operator` and `kubeovn`, nothing lost.

### What the nightly does *not* recover

Four of the ten are not in the root `Makefile`'s `build` target: `kilo`,
`redis-operator`, `keycloak-operator` and `kubeovn`. Their baked refs
keep the committed GHCR host, so the `SRC_REGISTRY` filter still drops
them and their `:nightly` tags remain absent after this PR. Installs are
unaffected — those digests already resolve on GHCR. The nightly recovers
the six OCIR-built images; promotion recovers all ten.

## Why this was invisible

Latent since `nightly-mirror.sh` was added — the nightly had never
completed a full run, so its mirror step was never exercised end to end.

Both test suites were green throughout. `nightly-mirror_test.bats`'s
fixture covered the single-string, `{repository,tag,digest}` and
platformSource shapes — a layout only cilium actually uses — and omitted
the dominant one. `promote-retag_test.bats` asserted "at least one
cozystack-owned image is selected", which is satisfied by the shapes
that already worked and cannot detect an omission.

Two comments actively pointed the wrong way, and are corrected here. The
header presented its list as the shapes the build writes, which reads as
exhaustive; it now says the list is empirical, names what an audit found
outside it, and says to re-run that audit when a package is added. The
ownership-filter comment named `kilo`, `kube-ovn` and
`keycloak-operator` as third parties "not pushed to `$REGISTRY`" — all
three are pushed there, and that wrong comment is part of why the
missing tags went unnoticed.

Both fixtures are extended here. The promotion test names concrete
packages rather than asserting a count or a generic pattern, since
either would drift back to proving nothing.

## Type guards

Both digest-in-tag rules check `.tag` is a string before `test()`.
Without it a non-string tag aborts the whole expression (`cannot match
with !!int, can only match strings`), and since these invocations
swallow stderr and status, that abort would silently discard **every**
ref of this shape in the same file — reintroducing this very bug in any
file that happens to carry an unquoted `tag: 1.24` next to a cozystack
image. Rule 1 is immune because its `select(tag == "!!str")` already
precedes its `test()`.

`registry`, `repository` and `digest` are guarded too, against a
narrower failure. Those keys are concatenated rather than `test()`ed,
and yq coerces int, bool, null and seq on `+` — each merely yields a ref
the ownership filter discards. Only a `!!map` aborts, raising `!!str ()
cannot be added to a !!map`, which takes the file's whole expression
with it. So that guard is insurance against a nested-map typo in a
vendored chart, not against unquoted scalars. It is a `select()`, so an
offending map drops only itself and its neighbours still resolve.

No non-string value of either kind exists in the tree today, so both are
prevention rather than an active fix.

## Testing

- Both suites pass: `nightly-mirror_test.bats` 2/2,
`promote-retag_test.bats` 4/4.
- Every new assertion was mutation-tested — removing the rule under test
turns it red. Verified individually for the shape-3 registry join, the
shape-2 registry join, the `global.registry.address` rule, and the
`repository` type guard. Also verified that a rule matching the bare tag
instead of the repository (the subtle wrong fix) fails, which is why the
assertions check the full source ref rather than only the destination.
- `promote-retag.sh --dry-run` against the real committed tree: 19 → 29
unique repositories, no losses.
- An audit of every `@sha256:` digest under `packages/*/*/values.yaml`
against the selector output accounts for all 35: 30 are selected, and
the remaining five are third-party
(`ghcr.io/lexfrei/{kuberture,ouroboros}`, `docker.io/clastix/kubectl`,
`ghcr.io/kvaps/...`) or sit outside `$REGISTRY`
(`ghcr.io/cozystack/ingress-nginx-with-protobuf-exporter/*`).
- `shellcheck` clean on both scripts at default severity and `-S style`.

## Ordering note for reviewers

Worth landing before the next release candidate is cut, rather than
treating this as routine CI cleanup.

`promote-rc.yaml` builds the release branch from the rc's staging
branch, and `pull-requests-release.yaml` runs `./hack/promote-retag.sh`
from that merged tree — so the copy of the script that executes at
promotion is the one frozen into the rc tag's tree, not whatever is on
the default branch at promotion time. An rc cut before this merges will
carry the old script and silently skip those retags when promoted.

The nightly half is also what unblocks the full-suite e2e lane, which
cannot currently publish a working install closure to GHCR.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff: it touches only `hack/nightly-mirror.sh`, `hack/promote-retag.sh`
and their two `.bats` suites. No `packages/`, no chart, no
`values.schema.json`, no `ApplicationDefinition`, no CRD or RBAC. The
`cozystack/ccp` trigger covers moving or renaming files under `hack/`
and changing what a make target does; neither applies — the anchor files
it gates on (`hack/package.mk`, `hack/common-envs.mk`) are untouched and
no make target changes behaviour.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(ci): mirror and retag every cozystack-owned image, including those whose digest is embedded in the values `tag` and those whose registry host is stored outside `repository`. Ten images across eight packages were silently skipped: the nightly published references that 404 at install, and promoted releases did not carry the `:<version>` tags they claim.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved nightly mirroring and retag promotion to recognize
digest-pinned image references across more YAML value shapes, including
digests embedded in the `tag` field and optional registry wrappers.
* Expanded support for chart-global image definitions and safer parsing
to avoid silently skipped references.
* Strengthened selection logic to exclude third-party images from copy
plans.
* **Tests**
* Expanded dry-run coverage with additional fixture cases for split
maps, `tag`-embedded digests, numeric-tag variants, and more “owned”
image assertions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
v1.6.0-rc.4 		2026-07-21 14:59:48 +05:00
myasnikovdaniil 1b777d8077 fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver (#3386) 
## What this PR does

`quay.io/centos/centos:stream9` is a **rolling tag** — upstream deletes
the previously-pinned digest whenever it publishes a new `stream9`
build. The digest re-pinned in ce4b263ca (`sha256:a93e3316…`) has now
itself been deleted upstream, so every `kubevirt-csi-driver` image build
fails:

```
#3 ERROR: quay.io/centos/centos:stream9@sha256:a93e3316…: not found
ERROR: failed to solve: … failed to resolve source metadata … not found
make: *** [Makefile:43: image-kubevirt-csi-driver] Error 1
```

This breaks the **Build packages/apps/kubernetes** job on `main` and on
every open PR (e.g. observed on #3383).

Re-pins to the current `stream9` manifest-list digest
`sha256:3714c89f1903dac5a5e1eb6c02d84189ff6d962a0c18df01feba0a5406856926`,
resolved via `docker buildx imagetools inspect
quay.io/centos/centos:stream9`.

> Note: this is the second recurrence of the same class of breakage
(ce4b263ca fixed the first). Because `stream9` is a rolling tag, a
pinned digest will keep rotting; a follow-up could track it via renovate
or pin a date-stamped/immutable tag instead. Out of scope here — this
just unblocks CI.

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated the container build to use a refreshed CentOS Stream 9 base
image.
  * No end-user-facing functionality or behavior changed.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-21 13:47:53 +05:00
Andrey KolkovandClaude Opus 4.8 9c3bae012a fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver 
quay.io/centos/centos:stream9 is a rolling tag; upstream deletes the pinned
digest whenever it publishes a new stream9 build. The digest re-pinned in
ce4b263ca (sha256:a93e3316...) has now itself been deleted, so every
kubevirt-csi-driver image build fails with:

  quay.io/centos/centos:stream9@sha256:a93e3316...: not found
  failed to resolve source metadata ...
  make: *** [image-kubevirt-csi-driver] Error 1

breaking the "Build packages/apps/kubernetes" job on main and every PR.

Re-pin to the current stream9 manifest-list digest
(sha256:3714c89f1903dac5a5e1eb6c02d84189ff6d962a0c18df01feba0a5406856926),
resolved via `docker buildx imagetools inspect quay.io/centos/centos:stream9`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-21 10:40:17 +04:00
Myasnikov Daniil 292f8ec65a fix(ci): type-guard the concatenated keys in the ref selectors 
Address non-blocking review feedback from lexfrei on
hack/nightly-mirror.sh:81 and hack/promote-retag.sh:84.

`registry`, `repository` and `digest` are joined into a ref by string
concatenation with no type guard, so a non-string value could abort the
whole per-file expression. Because collect_refs swallows both stderr and
status, that abort silently discards every co-located owned ref — the
same per-file blast radius the `tag` guard already prevents.

The mechanism differs from `tag`'s, though, and the narrower reading is
worth recording. These keys are never fed to test(), so the "cannot
match with !!int" abort does not apply to them: yq coerces int, bool,
null and seq on `+`, and each of those merely yields a ref the
ownership filter discards. Only a !!map actually aborts, raising
"!!str () cannot be added to a !!map". The guard is therefore narrow
insurance against a nested-map typo in a vendored chart, not protection
against unquoted scalars.

Implemented as select() rather than a test(), so an offending map drops
only itself and its neighbours in the same file still resolve.

The nightly fixture gains a file pairing a map-typed `repository` with
an owned ref, asserting the latter is still mirrored; removing the
guard turns it red. Selection over the committed tree is unchanged at
30 repositories.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 11:06:47 +05:00
Myasnikov Daniil acd88bdd00 fix(ci): collect image refs whose host is outside repository 
Address review feedback from lexfrei and IvanHunters on
hack/promote-retag.sh:84 and hack/nightly-mirror.sh:81.

The shape-3 rule assumed the registry host lives inside `repository`.
Two cozystack-built images do not satisfy that and were emitted
host-less, so the ownership filter dropped them — the same silent
emit-then-drop this PR fixes for the other shapes:

  - keycloak-operator splits the host into a sibling `registry` key
    ({registry: ghcr.io, repository: cozystack/cozystack/keycloak-
    operator, tag: latest@sha256:...}), and its own Makefile writes
    exactly that layout.
  - kubeovn keeps the host in the document-level
    global.registry.address with a bare `repository: kubeovn`. It is
    built and pushed to $REGISTRY by the cozystack/kubeovn-chart
    wrapper, whose `make image` writes all three keys.

Neither has received a 1.x release tag: on GHCR keycloak-operator
carries only `latest`, and kubeovn's newest cozystack-versioned tag is
v1.14.11-v0.38.8, from before the 1.0 versioning switch — while cilium,
which uses an already-covered shape, has v1.5.0 through v1.6.0-rc.2.

Shapes 2 and 3 now rejoin an optional `registry` sibling, and a new
shape-4 rule handles the chart-global layout. An audit of every
@sha256: digest under packages/*/*/values.yaml against the selector
output confirms the remaining five misses are third-party or sit
outside $REGISTRY. Dry-run over the committed tree: 28 -> 30
repositories, exactly these two gained, nothing lost.

Also correct two comments that overstated coverage. The header no
longer presents its list as the shapes the build is limited to — the
list is empirical and charts are vendored — and the ownership-filter
comment no longer names kilo, kube-ovn and keycloak-operator as
third parties not pushed to $REGISTRY. All three are pushed there, and
that wrong comment is part of why the missing tags went unnoticed.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 11:04:21 +05:00
Andrey KolkovandClaude Fable 5 b897b53107 test(postgres): fix S3 CA secret name in backup roundtrip e2e 
The backup roundtrip e2e copied the seaweedfs S3 CA from a Secret named
seaweedfs-system-ca-cert in tenant-root, a name captured off a dev
cluster. On current main the seaweedfs chart pins
fullnameOverride: seaweedfs, so the cert-manager CA Certificate's secret
is seaweedfs-ca-cert; the old name is absent and the first CI run of the
e2e failed at step 05a with 'secrets "seaweedfs-system-ca-cert" not
found'.

Default S3_CA_SECRET to seaweedfs-ca-cert and, when that Secret is
absent, auto-discover the seaweedfs CA Certificate (the seaweedfs-labelled
one with spec.isCA=true) and read its secretName, so a future
fullnameOverride change does not silently break the copy again. Fail with
a clear message when neither resolves. README and chainsaw comment
updated to match.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-20 22:31:27 +04:00
Myasnikov Daniil 770314b3cb fix(ci): retag images whose digest is embedded in the values tag 
promote-retag.sh carries the same collect_refs blind spot the previous commit
fixes in nightly-mirror.sh, and nightly-mirror.sh's own comment asserted the
two were "identical to" each other -- which is how the gap stayed invisible
once one of them was fixed.

Measured against the committed tree, the dry-run selects 19 repositories
before this change and 27 after. The eight it was missing are exactly the set
that broke the nightly: kamaji, kilo, linstor-csi, piraeus-server,
linstor-gui, metallb-controller, metallb-speaker, redis-operator. Nothing is
lost from the previous selection.

The consequence differs from the nightly case and is milder. This retag is
within a single registry, so the digests already exist and every digest-pinned
install still resolves -- there is no dangling reference and no 404. What is
missing is the :<version> tag (and :latest under MOVE_LATEST=1) for those
eight images, so a promoted release does not carry the tags it claims to, and
anything resolving a cozystack component by version tag rather than by digest
does not find it. Nothing in the promotion detects this: the script reports
success having simply never seen those refs.

The existing test could not catch it. It asserts "at least one cozystack-owned
image is selected" and that nothing third-party leaks in -- both satisfied by
the shapes that already worked, which is why eight omissions sat under a green
suite. It now names concrete shape-3 packages, since a count or a generic
pattern would drift back to proving nothing. Verified the assertion fails
against the unfixed script.

Same type guard on `.tag` as the previous commit, for the same reason: without
it an unquoted numeric tag anywhere in a file silently discards every shape-3
ref in that file, and the swallowed stderr leaves no trace.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-20 19:07:04 +05:00