The doc still claimed the kubernetes-nodes per-pool storageClass is intentionally
not immutable and undefaulted. This branch annotated it immutable (self == oldSelf
XValidation, @immutable in values.yaml) and it defaults to replicated, and the
cluster field is immutable too. Rewrite the paragraph to describe both fields,
add kubernetes-nodes to the covered-apps list, and drop the now-stale
top-level-only caveat on the kubernetes entry.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The 53-char overflow branch exists because exiting the migration deadlocks the
platform pre-upgrade hook for every tenant. The equivalent deadlock via name
validity was unguarded: the old nodeGroups map key was never schema-constrained
(map[string]NodeGroup, no propertyNames pattern), so a stored parent HR can
carry a key like 'My_Pool' whose own render already failed at apply time. The
migration then built child_name=kubernetes-nodes-<cluster>-My_Pool, kubectl
apply rejected the RFC-1123-invalid HelmRelease name, the script exited 1 under
set -e, and every subsequent platform upgrade re-ran and re-failed.
Validate child_name against the RFC-1123 label pattern alongside the length
check and route failures through the same warn + pin + skip path. Factor the
pin-and-skip logic shared by both guards into pin_pool_objects. Cover the new
branch in migration-54-adopt-path.bats (the fake kubectl apply now rejects an
invalid metadata.name, so the case fails without the guard).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The unpin Job carried only helm.sh/hook-delete-policy: before-hook-creation,
and its comment claimed a successful run is torn down with the release. Helm
does not track hook resources as release members, so after helm uninstall the
succeeded Job and its pod remained in the tenant namespace indefinitely, one
per deleted pool name per namespace — before-hook-creation only fires on a
next pre-delete of a same-named release, which never comes for a deleted pool.
Add spec.ttlSecondsAfterFinished: 86400 so a finished Job is garbage-collected
(a failed run still keeps its logs for a day), and correct the comment.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
assertParentVersion failed the pool render whenever the pool minor differed
from the parent KamajiControlPlane minor in either direction. A worker minor
lagging the control plane is supported (n-3 upstream) and is the normal state
during a rolling upgrade, where the parent Kubernetes CR is bumped first and
each pool follows. The symmetric check flipped every pool HelmRelease of a
cluster to render-failure the moment the parent was bumped, blocking all pool
operations until each pool version was hand-edited.
Fail only when the pool minor is ahead of the parent minor (semverCompare on
normalized major.minor.0), matching the stated rationale, and cover all four
directions (ahead / lag / equal / no parent) with a helm-unittest suite driven
by a mocked KamajiControlPlane.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The Phase-2 worker-pool adoption migration was renumbered 52 -> 54 during
the rebase onto main (main added migrations 52 linstor-scheduler and 53
seaweedfs). Rename the adoption bats tests and their testdata directory,
and update the kubernetes-nodes pre-delete hook comment and the
kubernetes README so they reference migration 54. The bats MIG path
pointed at migrations/52 (now the linstor migration), so without this the
adoption unit tests would exercise the wrong migration.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Address review on #3315:
- README breaking-changes: worker pools move to KubernetesNodes (nodeGroups
removed, adopted by migration 52); the parent Kubernetes CR's WorkloadsReady
is now control-plane-only; ingressNginx needs an explicit ingress pool; do
not edit Kubernetes CRs mid platform-upgrade.
- Note that the apiserver ClusterIP certSAN is a best-effort extra — workers
dial by DNS, so a pool-less control plane is fine without it; the merge-patch
array-replace predates the split.
- Confirm the pool VM pod is intentionally attributed to the parent Kubernetes
application (byte-parity + lineage/quota under the owning cluster).
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Address review on #3315:
- Mark cluster and storageClass immutable (self == oldSelf): both feed object
names / the KubevirtMachineTemplate content hash, so editing them on a live
pool would rename everything and delete the running worker VMs.
- Validate the pool's Kubernetes minor against the parent cluster's
KamajiControlPlane at render (skipped offline / when the parent is absent),
so a pool created after the split cannot join workers ahead of the apiserver.
- Note in the snapshot suite header that the instancetype-sized default is not
pinned offline (lookup nil); the e2e exercises it.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Address review on #3315:
- Name-length overflow no longer exits 1 (which would deadlock the platform
pre-upgrade hook for every tenant). Instead pin the pool's worker objects
with resource-policy=keep, skip the child HR, warn loudly, and continue.
- Copy only images.kubectl into the child values, not the parent's whole
images map (waitForKubeconfig/talosCsrSigner are undeclared in the schema).
- Note that the pre-upgrade window is not quiet (a parent HR edit mid-upgrade
renders the old chart and self-heals once the new artifact lands).
- Cover the adoption path directly: annotate keep + release-name, idempotent
skip, unexpected-owner refusal, and the 6-hex KMT anchor rejecting a sibling
pool's template.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Drives migration 52 against a fake kubectl and asserts the adopted
child HelmRelease carries the tenant's non-default talos.version,
version and images (plus group fields and storageClass fallback). Fails
against the pick(o;...) filter-parameter bug that dropped those keys and
rolled live worker VMs; passes against the pick($o;...) fix.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The adoption jq used a filter-parameter helper `def pick(o; ks)`; inside
the `reduce` the `.` context is the accumulator, so `o | has($k)` tested
the accumulator, not the source object, and `pick(.; ["version","talos",
"images"])` deterministically returned {}. The child HelmRelease was
therefore created without the tenant's talos/version, so Helm rendered
the chart defaults; since talos.version feeds the KubevirtMachineTemplate
content-hash, the KMT name changed and the MachineDeployment rolled every
live worker VM on upgrade — the exact churn this migration must avoid.
Caught by e2e on dev3 (reproduced on 2/2 real clusters). Switch pick to
a $-parameter so it is evaluated once at the call site.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The chart split removed the parent pool-render suites and the cross-chart
parity script, leaving the GPU device block, kubelet-reservation math and
maxUnhealthy IntOrString branches (all still in nodegroup.yaml /
talos-reconcile-job.yaml) uncovered. Add GPU and kubelet snapshot cases
so drift renaming the content-hashed KubevirtMachineTemplate fails CI,
and port the behavioral + failure-path assertions (GPU nodeLabels,
kubelet reserved/effective invariants and quantity rejections, MHC
integer/percentage quoting) to the child chart.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The chainsaw harness created a KubernetesNodes CR but never reclaimed it,
leaking the pool CR, its child HelmRelease, release Secret and
WorkloadMonitor into the shared tenant-test namespace and breaking
rerun-in-same-sandbox (the surviving child HR left the MachineDeployment
un-rendered on retry). Delete kubernetesnodeses in all three cleanup
paths, before the parent CR so the pre-delete unpin hook runs.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Remove the orphaned kubernetes.nodeGroups doc comment, the stale kubelet
reservation / nodeGroups storageClass prose in the README, and the inert
nodeGroups blocks in the test value fixtures.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Check the resource-policy annotation before removing it so a native pool
is a clean no-op while a real annotate failure fails the Job (retries
bounded, logs retained) instead of silently re-introducing the keep
leak. Scope the MD/MHC/WorkloadMonitor Role rules with resourceNames.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The chainsaw harness created the Kubernetes CR with spec.nodeGroups,
which the control-plane-only chart now ignores, so no MachineDeployment
was created and the suites timed out. Create a KubernetesNodes CR named
<cluster>-md0 (roles: [ingress-nginx]) so the same MachineDeployment is
produced and the MD/ingress waits pass; this also gives the split its
first e2e coverage.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The render-time fail flipped existing degraded clusters from Ready to
Failed on upgrade, blocking all subsequent control-plane changes.
Restore the prior non-breaking behavior (the DaemonSet schedules nothing
if no ingress-roled node exists) and document the pool requirement in
NOTES.txt instead of enforcing it at render time.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The cross-chart parity script was removed (the parent no longer renders
pools to diff against); the render snapshot under helm unittest is the
drift guard now and already runs in CI.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Remove dead nodeGroups blocks from the CI values fixtures and point the
storage-immutability doc at the kubernetes-nodes per-pool storageClass.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Migration 52 pins adopted worker objects with helm.sh/resource-policy=keep
for the transition window. Left in place it leaks the objects when an
adopted pool is deleted. Add a pre-delete hook that strips the annotation
from the release's own worker objects so Helm deletes them cleanly;
runs on delete only (never on upgrade, so it cannot race the parent
re-render) and is a no-op for natively-created pools.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
types.go dropped NodeGroups/NodeGroup/GPU/Kubelet/NodeHealthCheck but
the generated deepcopy still referenced them, so the api module failed
to build. Regenerated with controller-gen.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Anchor the KubevirtMachineTemplate sweep to the 6-hex content-hash
suffix so a pool name that is a prefix of a sibling (md0 vs md0-large)
no longer mis-adopts the sibling's KMT and deadlocks the migration.
Adopt the worker objects before creating the child HelmRelease so
helm-controller cannot start the child install into un-adopted objects
and fail Helm ownership validation. Emit the same install/upgrade
strategy (RetryOnFailure) and maxHistory cozystack-api sets, add a
pre-flight release-name length check, and document the rollback path.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The parent no longer renders pools to diff against, so the one-shot
extraction parity check is replaced by a self-contained golden snapshot.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
With pools moved out of the chart the ingress-nginx addon can no longer
rely on an auto-provisioned md0; fail the render with a clear message
when addons.ingressNginx.enabled and no pool carries roles: [ingress-nginx].
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Worker pools are now managed via KubernetesNodes resources; existing
pools are adopted by migration 52.
BREAKING CHANGE: the Kubernetes CR no longer accepts spec.nodeGroups.
Create KubernetesNodes resources for worker pools instead.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The KubevirtMachineTemplate/MachineDeployment/MachineHealthCheck/worker
WorkloadMonitor, the preserve-old-KMT block, the talos-reconcile Job and
its RBAC now live in the kubernetes-nodes chart. The kubernetes chart
renders the control plane only.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Pre-upgrade migration for the Phase 2 kubernetes chart split. For every
parent kubernetes-<cluster> HelmRelease it creates one child
kubernetes-nodes-<cluster>-<pool> HelmRelease and re-annotates the
existing worker objects onto it with helm.sh/resource-policy=keep, so
the now control-plane-only parent upgrade does not prune the tenant's
running workers. Byte-identical child render (PR-2a golden parity) keeps
the KubevirtMachineTemplate content-hash stable, so no worker VM churns.
Idempotent and strict (exit 1 on any failure). Bumps targetVersion to 53
(contiguous with main; rebase to 53/54 if another 52 lands first).
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The ephemeralStorage guard could never fire (the pool group is built from a
fixed set of chart values that never includes ephemeralStorage), so it read
as protection that is not there — remove it. Note in the parity script header
that only the four pool objects are compared, not the talos-reconcile Job.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Extend the golden-parity matrix beyond the resources-sized pool to also
diff a GPU pool and a kubelet-override pool against the parent render, so
future drift in those branches (which would roll the content-hash-named
KubevirtMachineTemplate) is caught in CI.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Reference the adoption migration by role (Phase 2b) instead of a number
that does not exist on this branch, and correct the NOTES operational-note
count.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Wire tests/render-parity.sh into the chart's test target so CI verifies
the pool objects stay byte-identical to the parent render.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Restore the inline machineconfig comments (nameservers, GPU node labels)
inside the reconcile Job's data block so the applied TalosConfigTemplate
matches the parent's, avoiding drift on adoption.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The cluster field referenced x-cozystack-options source 'kubernetes',
which has no provider (cozystack-api returns NotFound), so the dashboard
dropdown 404s. Drop the annotation so the field is a plain text input;
a real provider listing Kubernetes CRs is a follow-up.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Replace the collision-prone name-prefix match (pool 'gpu' matched
sibling 'gpu-large-<hash>') with a release-name annotation scope, and
add a render-time guard that fails cleanly when a pool's MachineDeployment
already exists under a different release (e.g. naming a pool after a
parent nodeGroup such as the default md0). Document the naming caveat in
NOTES.txt.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The preserve-old-KubevirtMachineTemplate block re-emitted any KMT owned
by the CAPI Cluster and referenced by a MachineSet. A kubernetes-nodes
pool's KMT lives in the same namespace under the same Cluster, so the
parent would re-emit it with a foreign meta.helm.sh/release-name and its
next upgrade would fail Helm ownership validation. Restrict the block to
KMTs whose release-name annotation matches the parent release.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Add the kubernetes-nodes-rd package (ApplicationDefinition), the
cozystack.kubernetes-nodes-application PackageSource (kubevirt variant,
kubernetes-nodes + kubernetes-nodes-rd components), and wire it into the
iaas bundle. Mirrors the vm-instance/vm-disk registration precedent.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Unit tests cover naming/selectors, clusterName wiring, autoscaler
annotations, omitted replicas, bootstrap TCT reference, KMT hash name,
worker WorkloadMonitor, and the empty-cluster / mis-prefixed-release
render guards. render-parity.sh diffs the pool objects against the
parent kubernetes chart and asserts byte-identity.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Introduce the kubernetes-nodes application chart: one HelmRelease per
worker node pool, attached to a parent Kubernetes cluster by name
convention (kubernetes-<cluster>). Renders the KubevirtMachineTemplate,
MachineDeployment, MachineHealthCheck, worker WorkloadMonitor and the
talos-reconcile Job with its RBAC, extracted from the monolithic
kubernetes chart. Pool objects are byte-identical to the parent's render
for the same group, so a later adoption never churns live worker VMs.
This is the additive half of the Phase 2 split (community#8); the parent
kubernetes chart is left unchanged.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
## What this PR does
- allows superseded PR workflow runs to cancel `finalize` and `E2E
Tests` instead of continuing because of job-level `always()`
- makes `update-website-docs` cancellation-responsive while preserving
its behavior when changelog generation fails or is skipped
- retains step-level `always()` conditions for reports and sandbox
cleanup
### Screenshots
Not applicable.
### Release note
```release-note
fix(ci): ensure superseded and manually canceled workflows stop long-running jobs
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Prevented release and validation-related jobs from running when a
workflow is cancelled.
* Updated pull request check, end-to-end testing, and stable
documentation update conditions to avoid executing after cancellation.
* Kept existing success-gating so follow-up tasks still require
prerequisite steps to complete successfully.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Fixes#3245.
The documented admin override for the `cozy-default` backup S3
coordinates — `spec.components.backupstrategy-controller.values` on the
`Package cozystack.cozystack-platform`
(`docs/operations/backup-classes.md`, "Admin overrides for
`cozy-default`") — had no effect: the `cozystack-platform` PackageSource
exposes exactly one component (`platform`), so the operator silently
ignored the block, and the platform chart rendered the child `Package
cozystack.backupstrategy-controller` via the `.default` helper with no
components argument, so nothing forwarded `backupStorage` either. Admins
running against an external or TLS S3 had no working supported path to
set `backupStorage.endpoint` / `region` / `systemSecretName`.
This PR wires the override through the platform chart, the same way
`bundles/iaas.yaml` forwards kubevirt and gpu-operator values:
- `packages/core/platform/values.yaml` gains a top-level `backupStorage:
{}` value.
- `templates/bundles/system.yaml` renders it into the emitted `Package
cozystack.backupstrategy-controller` as
`components.backupstrategy-controller.values.backupStorage`, from where
the cozystack operator merges it into the `backupstrategy-controller`
HelmRelease over the chart defaults. When `backupStorage` is unset the
components block is omitted and the emitted Package is byte-identical to
the historical render.
- `docs/operations/backup-classes.md` now documents the working path
(`spec.components.platform.values.backupStorage`) and explicitly calls
out the two plausible-looking paths that do not work (the old documented
one, and patching the child Package directly, which the platform
helm-reconcile reverts).
- A new helm-unittest suite
(`packages/core/platform/tests/bundles_backupstorage_wiring_test.yaml`)
is the behavioural regression test: it proves the unset case emits no
components block and the set case lands the values on the child Package,
on both `isp-full` and `isp-full-generic`.
Verified with `make test` in `packages/core/platform` (22 suites, 91
tests, all green). `make show` requires a live cluster (`cozyhr` +
`lookup` in `templates/repository.yaml`), so the unittest render of
`templates/bundles/system.yaml` is the offline verification path.
### Screenshots
Not a UI change.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#622
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
Walking the trigger map against the diff: the change to
`packages/core/platform/values.yaml` reaches **cozystack/website** twice
— `content/en/docs/next/operations/configuration/platform-package.md`
(hand-written table of `spec.components.platform.values.*` keys gains
`backupStorage`) and
`content/en/docs/next/operations/services/backup-classes.md` (mirrors
the broken override example this PR corrects). The follow-up is
cozystack/website#622, which also carries the superseded website#547
content and covers both next/ and v1.4/ docs, marked not to merge before
this PR lands; its v1.4 copy assumes this fix is backported to
release-1.4. No other downstream repository restates the touched files:
the templates change is chart-internal, no app `values.schema.json`,
CRD, variant list, `hack/` file, or namespace changed.
### Release note
```release-note
fix(platform): the documented admin override for the cozy-default backup S3 coordinates now works — set spec.components.platform.values.backupStorage on the cozystack.cozystack-platform Package and the platform chart forwards it to the backupstrategy-controller component (previously the documented spec.components.backupstrategy-controller path was silently ignored)
```
## What this PR does
`packages/apps/kubernetes/images/kubevirt-csi-driver/Dockerfile` pinned
its runtime stage to `quay.io/centos/centos:stream9` by digest. Quay
serves **only currently-tagged manifests** — when a rolling tag moves,
the superseded digest immediately returns `MANIFEST_UNKNOWN`, with no
grace period.
`stream9` rolls roughly every six days, so the pin has already died
twice:
| Date | Fix | Digest pinned | Status today |
|---|---|---|---|
| Jul 15 | #3303 | `0c9db821…` → `a93e3316…` | `0c9db821` gone |
| Jul 21 | #3386 | `a93e3316…` → `3714c89f…` | `a93e3316` gone |
Each death broke `make build` on `main` and on every PR triggering a
full rebuild. Because the root `build` target is serial, a failure in
`packages/apps/kubernetes` (4th of 29) aborts the remaining packages and
leaves the shared build cache unwritten.
Repinning cannot fix this: the freshly pinned digest is always the
current live tag, so it is next in line to be retired. This is not
specific to CentOS — the same probe against
`quay.io/fedora/fedora:latest` and `quay.io/centos/centos:stream10`
found every historical digest unfetchable.
This PR keeps the digest pin and changes the *source* to a registry that
provably retains superseded manifests, making the pin durable rather
than decorative. Renovate policy is unchanged — `pinDigests` stays on,
and `almalinux:9` is an ordinary rolling tag on a retaining registry
exactly like the `golang`, `alpine` and `debian` bases already in the
grouped digest PR.
### Verification
Both bases were run locally with the Dockerfile's exact install line
(`dnf install -y e2fsprogs xfsprogs nfs-utils`):
| Binary | CentOS Stream 9 | AlmaLinux 9 |
|---|---|---|
| `mkfs.ext4` | `/usr/sbin/mkfs.ext4` | `/usr/sbin/mkfs.ext4` |
| `mkfs.xfs` | `/usr/sbin/mkfs.xfs` | `/usr/sbin/mkfs.xfs` |
| `blkid` | `/usr/sbin/blkid` | `/usr/sbin/blkid` |
| `mount` | `/usr/bin/mount` | `/usr/bin/mount` |
| `umount` | `/usr/bin/umount` | `/usr/bin/umount` |
| `fsck.ext4` | `/usr/sbin/fsck.ext4` | `/usr/sbin/fsck.ext4` |
| `fsck.xfs` | `/usr/sbin/fsck.xfs` | `/usr/sbin/fsck.xfs` |
| `mount.nfs` | `/usr/sbin/mount.nfs` | `/usr/sbin/mount.nfs` |
Every path `k8s.io/mount-utils` shells out to is identical. Package
versions:
| Package | CentOS Stream 9 | AlmaLinux 9 |
|---|---|---|
| `e2fsprogs` | 1.46.5-8.el9 | 1.46.5-8.el9 |
| `xfsprogs` | 6.4.0-7.el9 | 6.4.0-7.el9 |
| `nfs-utils` | 2.5.4-44.el9 | 2.5.4-42.el9_8 |
| `util-linux` | 2.37.4-26.el9 | 2.37.4-25.el9 |
| `glibc` | 2.34-274.el9 | 2.34-270.el9_8 |
Same upstream versions throughout; the el9 release-number lag is
expected since AlmaLinux is downstream of RHEL 9 while CentOS Stream 9
is upstream. `e2fsprogs` and `xfsprogs` — the two that govern
`mkfs`/`fsck` behaviour — are identical NVRs. `glibc` is not linked at
all: the driver is built `CGO_ENABLED=0`.
The pinned digest is the OCI image index covering `amd64`, `arm64`,
`ppc64le` and `s390x` — the same architecture set as the previous base.
Retention was verified empirically rather than assumed: four superseded
`docker.io/library` digests taken from this repo's own git history
(`golang:1.26`, `golang:1.26-alpine`, `alpine:3.24`, `alpine:3.22`) all
still resolve after their tags moved.
### Scope
This addresses the immediate recurring breakage only. The broader
concern in #3305 — that any retired digest blocks the entire platform
build, with no advance warning and no fast path to repin — is not solved
here and remains open.
### Release note
```release-note
fix(kubernetes): move the kubevirt-csi-driver runtime base from quay.io/centos/centos:stream9 to docker.io/library/almalinux:9. Quay retires superseded manifests as soon as a rolling tag moves, which broke builds twice in six days; AlmaLinux 9 provides an equivalent RHEL 9 userspace on a registry that retains them, so the digest pin remains valid over time. No functional change to the driver.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the container runtime environment to use AlmaLinux 9 instead
of CentOS Stream 9.
* Kubernetes CSI driver behavior and startup configuration remain
unchanged.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
quay.io serves only currently-tagged manifests: when a rolling tag moves,
the superseded digest immediately returns MANIFEST_UNKNOWN. The pinned
`centos:stream9` digest has died twice in six days (#3303, #3386), each
time breaking `make build` on main and on every PR that triggers a full
rebuild.
Repinning is not a fix -- the newly pinned digest is always the live tag,
so it is next in line to be retired. Move the runtime stage to a registry
that provably retains superseded manifests instead, which makes the digest
pin durable rather than decorative. Pinning policy is unchanged.
Verified equivalent: after `dnf install -y e2fsprogs xfsprogs nfs-utils`,
both bases provide mkfs.ext4, mkfs.xfs, blkid, mount, umount, fsck.ext4,
fsck.xfs and mount.nfs at identical paths. e2fsprogs and xfsprogs are
identical NVRs. The driver is built CGO_ENABLED=0, so the base libc is not
linked. The pinned digest is the OCI index covering amd64, arm64, ppc64le
and s390x -- the same architecture set as the previous base.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Fixes a Helm-upgrade failure in the `opensearch-operator` chart that
blocks the new release-upgrade E2E lane (added in #3276).
Upgrading from a chart that shipped no explicit strategy (v1.5.x) to a
v1.6.0-rc build fails validation immediately, helm-controller retries
it, and the HelmRelease never becomes Ready:
```
Helm upgrade failed for release cozy-opensearch-operator/opensearch-operator:
Deployment.apps "opensearch-operator-controller-manager" is invalid:
spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is Recreate
```
Observed in the ["Upgrade E2E Test" job of run
29480724982](https://github.com/cozystack/cozystack/actions/runs/29480724982)
— `UpgradeFailed ... (x19 over 11m)`, with the HelmRelease never
reaching Ready.
### Root cause
#3040 (commit db797b1d) set the single-replica operator Deployment to
`strategy.type: Recreate` — a reasonable goal: with leader election
gated off on one replica, a default `RollingUpdate` briefly runs two
uncoordinated managers (`maxSurge`) during a rollout, and `Recreate`
closes that window.
The problem surfaces only on **upgrade from an older chart, under
server-side apply**:
- A v1.5.x release shipped no strategy at all, so the apiserver
**defaulted** `spec.strategy.rollingUpdate` to `25%/25%`, and the
`helm-controller` field manager never owned that block. Its
`managedFields` entry covers `f:replicas`, `f:selector` and
`f:template`, but not `f:strategy`.
- helm-controller applies **server-side** by default
(`Install.ServerSideApply` defaults to true, `Upgrade.ServerSideApply`
to `auto`; `internal/operator/package_reconciler.go` sets neither), and
SSA does not remove a field the applier never owned merely because the
new intent omits it.
- So the v1.6.0-rc manifest's `type: Recreate` merges onto a live object
that still carries the defaulted `rollingUpdate`, and the apiserver
rejects the result with `FieldValueForbidden`.
The **client-side** path is unaffected, which is why this surfaces
through helm-controller's SSA but not a local `helm upgrade`:
`DeploymentSpec.Strategy` carries `patchStrategy:"retainKeys"`
(`k8s.io/api@v0.34.1/apps/v1/types.go:395`), so a 3-way merge emits
`{"spec":{"strategy":{"$retainKeys":["type"],"type":"Recreate"}}}`,
which merges to a valid `{"strategy":{"type":"Recreate"}}` and does
clear the block.
This is a genuine product upgrade bug, not a test-harness issue — #3276
only adds the upgrade lane that exposes it; it does not touch
`opensearch-operator`.
> An earlier revision of this PR attributed the failure to Helm's
client-side 3-way merge. That was wrong, and the correction is
@lexfrei's — see [his
review](https://github.com/cozystack/cozystack/pull/3319#pullrequestreview-4713156867).
The mechanism above is now reproduced end to end (below).
### The fix
Keep `type: RollingUpdate` on the single-replica path and set
`rollingUpdate.maxSurge: 0 / maxUnavailable: 1` instead of switching to
`Recreate`:
- `maxSurge: 0` caps total pods at `replicas`, so the deployment
controller must scale the old manager down before it can scale the new
one up — unlike the default 25% surge, which starts a second manager
while the first is still fully live.
- This is **not** `Recreate`'s exclusivity, and the PR no longer claims
it is. `rolloutRecreate` gates scale-up on `oldPodsRunning()`, which
counts non-terminal `Status.Phase`, so `Recreate` waits for the old pod
to be gone. `rolloutRolling` has no such gate: `NewRSNewReplicas`
derives `currentPodCount` from `GetReplicaCountForReplicaSets`, which
sums `rs.Spec.Replicas` (desired, not live). The ordering that is
guaranteed is over ReplicaSet desired counts, not pod lifecycles, so a
draining old manager can still overlap the new pod within
`terminationGracePeriodSeconds` (10s here).
- Because `strategy.type` never becomes `Recreate`, the forbidden merge
cannot occur on any apply path. It also recovers anyone already on an rc
build with `Recreate`, since `Recreate -> RollingUpdate` is permitted.
- This matches the idiom already vendored in this repo:
`packages/system/kubeovn/charts/kube-ovn/templates/central-deploy.yaml:11-15`
uses `RollingUpdate` with `maxSurge: 0 / maxUnavailable: 1`.
Multi-replica installs (`manager.replicaCount > 1`) are unchanged: they
keep the default `RollingUpdate` with leader election enabled.
The change is carried in `patches/leaderElection.diff` so it survives a
chart re-vendor via `make update`; the patch applies to pristine
upstream 2.8.0 with no fuzz and reproduces the committed strategy block
exactly.
### Alternatives considered
| Option | Why not |
| --- | --- |
| Keep `Recreate`, but make the chart own `rollingUpdate` first (ship an
explicit block, then switch `type` in a later release) | This is the
only option that preserves real exclusivity, but it needs two releases
and only works once every install has reconciled the intermediate one.
Disproportionate for a single-replica operator whose residual overlap is
a draining manager. |
| Pre-upgrade hook / Job that patches or deletes the Deployment strategy
before the upgrade | Heavy: needs a kubectl image, a ServiceAccount and
RBAC to `patch`/`delete` Deployments, and hook ordering; runs on every
upgrade even when unneeded; adds standing attack surface —
disproportionate for a scalar field transition. |
| `helm.sh/resource-policy` / force-replace | `resource-policy: keep`
only governs deletion on uninstall (irrelevant). Helm has no
per-resource "force recreate" annotation; `--force` / HelmRelease
`spec.upgrade.force` is global and disruptive and not controllable from
within the chart. |
| Plain revert to default `RollingUpdate` (25%/25%) | Fixes the
rejection but starts a surge pod while the old manager is still fully
live — the window #3040 set out to close. `maxSurge: 0` narrows it to a
draining old pod without leaving `RollingUpdate`. |
### Verification
The mechanism and the fix were reproduced on kind v1.33.1 (a bare
apiserver is enough — the failure is defaulting + SSA + validation):
| Path | Result |
| --- | --- |
| SSA install of the v1.5.x manifest (no strategy) | apiserver defaults
`rollingUpdate: 25%/25%`; `managedFields` shows `helm-controller` does
not own `f:strategy` |
| SSA upgrade to `type: Recreate` | fails with the exact production
error above |
| Client-side apply of the same pair | succeeds; `rollingUpdate` is
cleared, live strategy becomes `{"type":"Recreate"}` |
| SSA upgrade to `maxSurge: 0` (this PR) | succeeds |
| SSA of `maxSurge: 0` onto a live `Recreate` Deployment (rc recovery) |
succeeds |
| Rollout under `maxSurge: 0` vs `Recreate` | `maxSurge: 0`: two pods
coexisted within 1s (old Terminating, new ContainerCreating).
`Recreate`: one pod, waited the full drain. This is the evidence for the
"not exclusivity" wording above. |
### Note on regression coverage
`tests/leader_election_test.yaml` was never executed:
`hack/helm-unit-tests.sh` runs a package's suite only when the package
Makefile defines a `test:` target, and this one did not — so the
coverage the previous revision of this PR claimed did not exist. This PR
adds the target (matching `packages/system/etcd-operator/Makefile`),
which revives the suite; the repo-wide runner now picks the package up
and a revert to `Recreate` fails it. Thanks to @lexfrei for catching
that.
The apiserver-side merge that produces the invalid object still needs a
live cluster, so the behavioural regression test remains the
release-upgrade E2E lane in #3276 (which this fix unblocks).
### Follow-ups filed
- #3324 — `packages/system/ouroboros` has the same latent shape
(`Recreate` gated on `controller.mode == "external-dns"`, while
`controller.mode` defaults to `coredns`). Not fixed here, and it likely
needs a different fix, since that mode genuinely depends on exclusivity.
- #3325 — `make update` does not reproduce this package's vendored
template: three hand-edits are not captured in `patches/`, and one of
them breaks rendering when `manager.extraEnv` is set. Pre-existing on
`main`; an earlier revision of this body claimed the regeneration was
exact, which was wrong.
### Downstream repositories
- [x] No downstream repository is affected by this change
<!-- Walked the trigger map in docs/agents/contributing.md against the
diff: the change touches only a vendored packages/system operator chart
template + its patch + its helm-unittest suite + the package Makefile
test target. No packages/apps or packages/extra add/rename/remove, no
values.schema.json / enum / default change, no node contract, no
annotations/labels, no hack/ or ApplicationDefinition change.
opensearch-k8s-operator is a third-party upstream (not a cozystack
downstream), and the fix lives in patches/leaderElection.diff so it
survives make update. -->
### Release note
```release-note
fix(opensearch-operator): fix a Helm upgrade failure on single-replica installs. Upgrading a release that predates an explicit Deployment strategy failed under server-side apply, because switching strategy.type to Recreate does not clear the apiserver-defaulted spec.strategy.rollingUpdate block that the chart never owned, and the merged object is rejected. The operator now uses RollingUpdate with maxSurge=0/maxUnavailable=1, which keeps the transition legal on every apply path and recovers releases already stuck on Recreate.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Updated OpenSearch Operator controller rollout behavior for
single-replica installs to use a safer rolling update configuration.
* Leader election is now applied only when multiple replicas are
configured, reducing the chance of conflicting controller activity.
* **Tests**
* Expanded Helm unittest assertions for the controller deployment
rollout strategy and leader-election behavior for single- and
multi-replica scenarios.
* **Chores**
* Added a `make test` target to run Helm unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
Adds typed `waitStrategy` + `healthCheckExprs` fields so a package/app
can declare the Flux helm-controller v1.5.0 kstatus health gate for the
custom resource it wraps — making `HelmRelease.Ready` mean "the CR is
actually healthy" instead of "helm applied the CR". This is the
operator/apiserver foundation of the kstatus epic (#2642); the API
surface was reviewed and approved by the API owner in #3263.
Commits (per the review's "isolated step-0" request):
1. `build(deps)` — the dependency bump (see below).
2. `feat(api)` — the fields, plumbing, tests, and regenerated code.
3. `fix(api)` — match the openapi code-generator pin to the 0.35 runtime
(fixes an aggregated-apiserver crash the bump surfaced).
4. `fix(api)` — declare `OpenAPIModelName` on the apps types so SSA
resolves catalog kinds.
5. `docs(api)` — note that `disableWait` silently defeats
`healthCheckExprs`.
## API surface
Typed fields on **both** HelmRelease-building paths:
- `ApplicationDefinition.spec.release` (apiserver / catalog apps →
`convertApplicationToHelmRelease`)
- `ComponentInstall` (operator / platform packages →
`buildHelmReleaseSpec`)
```go
// WaitStrategy maps to HelmReleaseSpec.WaitStrategy.Name — a deliberate scalar
// simplification of the upstream {name} object. One of poller|legacy.
WaitStrategy string
// HealthCheckExprs maps to HelmReleaseSpec.HealthCheckExprs (the upstream
// kustomize.CustomHealthCheck type verbatim).
HealthCheckExprs []kustomize.CustomHealthCheck
```
`config.ResolveWaitStrategy` is shared by both builders and **couples
the default**: `healthCheckExprs` are only evaluated under the `poller`
strategy, so when expressions are set and no strategy is given, the
generated HelmRelease defaults to `poller`. A package that sets only
`healthCheckExprs` is therefore self-contained, independent of the
controller's global default.
## The dependency bump (commit 1)
The `WaitStrategy` / `kustomize.CustomHealthCheck` /
`HelmReleaseSpec.HealthCheckExprs` types do not exist below
`helm-controller/api` v1.5.0 — the runtime is already v1.5.0 (shipped by
#2602), so this closes a Go-types-vs-runtime skew. It is **not** an
isolated bump: v1.5.x requires **k8s 0.35 + controller-runtime 0.23**,
cascading the k8s stack 0.34 → 0.35 (and the go directive to 1.26).
cozystack replaces `k8s.io/apimachinery` with a fork carrying the
still-open upstream `ConvertToVersion` patch (k/k#135537). That single
commit rebases cleanly onto apimachinery v0.35.0; published at
`cozystack/apimachinery@release-1.35-pr135537`, and `go.mod` pins its
pseudo-version. **The repo builds with zero source changes under the new
stack.**
> Maintainer note: please confirm the `release-1.35-pr135537` fork
branch naming, or fold it into the `cozystack` branch as you prefer —
the pin can be updated to match.
## Not included (conscious YAGNI)
`disableWait` is intentionally omitted. The only known need (the
kubernetes tenant-addon deadlock) was resolved child-side, and
`waitStrategy: legacy` is not equivalent (legacy still waits
Helm-v3-style; `disableWait` skips waiting entirely). Add it later when
a concrete leaf needs a true no-gate escape hatch.
## Tests
- `config.TestResolveWaitStrategy` — the couple-the-default logic
(unset+exprs→poller; explicit legacy honored; unset+no-exprs→nil).
- Operator + apiserver cases mirror each other
(`TestBuildHelmReleaseSpecHealthCheckExprs` /
`TestConvertApplicationToHelmRelease_HealthCheckExprs`).
- `go build ./...` clean, `go vet` clean, `gofmt` clean, codegen
regenerated and committed.
The 4 backupstrategy CRDs regenerate with benign k8s-0.35 PodSpec
description drift (DynamicResourceAllocation alpha→stable, toleration
Lt/Gt, resizePolicy note) — no functional change.
## Follow-up
Phase 3 (per-leaf adoption — postgres/kafka/mongodb/etc. declaring their
CEL via these fields) builds on this. The full CRD→CEL catalog is
maintained in the epic's readiness audit.
Refs #2642
```release-note
feat(api): add `waitStrategy` and `healthCheckExprs` to ApplicationDefinition and ComponentInstall, so a HelmRelease reports Ready only once the custom resource it wraps is actually healthy
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable CEL health-check expressions for application
releases and package components.
* Added `waitStrategy` support (`poller` and `legacy`) with automatic
`poller` selection when health checks are configured and no strategy is
set.
* Propagated these settings into generated Helm releases, affecting
readiness/wait behavior.
* **Documentation**
* Updated Application and PackageSource schemas to include
`healthCheckExprs` and `waitStrategy`.
* Extended related Kubernetes CRD documentation, including projected pod
certificate annotations and workload reference fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Branch-review on this PR: for provisionBucket: false the
backupstrategy-controller.bucketName helper returns
.Values.backupStorage.bucketName verbatim as the real S3 bucket name
and every strategy CR plus the Velero BSL bakes it in, so the
external-S3 section must require updating bucketName (the Secret's
bucketName key alone is not read by the strategies), and the knob
table must state the two-mode semantics instead of the COSI-only one.
Also route the external-S3 instruction through the platform component
override instead of naming raw chart values, and add the isp-hosted
variant case to the forwarding test (92 tests green).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The projector runs from both the BackupJob and RestoreJob controllers
(restorejob_controller.go), not only before BackupJobs — keep the
monorepo doc aligned with the website copy (cozystack/website#622).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Same corrections as the website copy (cozystack/website#622 round 4):
the re-render interval is 5m via the operator's helmrelease-interval
flag (not a 10-minute Flux default), the FoundationDB backup CR group
is apps.foundationdb.org, the BSL readiness check uses valid kubectl
wait --for=jsonpath syntax, and the projector's first synchronous round
is gated on leader election rather than pod readiness.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
IvanHunters