Commit Graph
7777 Commits
Author SHA1 Message Date
Andrey Kolkov aef9e201c7 fix(ci): make workflow cancellation effective (#3401)
## What this PR does

- allows superseded PR workflow runs to cancel `finalize` and `E2E
Tests` instead of continuing because of job-level `always()`
- makes `update-website-docs` cancellation-responsive while preserving
its behavior when changelog generation fails or is skipped
- retains step-level `always()` conditions for reports and sandbox
cleanup

### Screenshots

Not applicable.

### Release note

```release-note
fix(ci): ensure superseded and manually canceled workflows stop long-running jobs
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Prevented release and validation-related jobs from running when a
workflow is cancelled.
* Updated pull request check, end-to-end testing, and stable
documentation update conditions to avoid executing after cancellation.
* Kept existing success-gating so follow-up tasks still require
prerequisite steps to complete successfully.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-21 20:52:02 +04:00
Myasnikov Daniil 8ad25b970f test(ci): pin digests exactly and generalize the first-party markers
Round-3 review, both reviewers again converging on the same defects.

The file-level marker check was keyed to the exact ref spellings present
in the tree rather than to the join semantics _collect_yaml_shapes
implements, and three realistic first-party refs evaded it: `registry:
ghcr.io/cozystack/cozystack` + `repository: <name>` (kubeovn's layout
under the key name keycloak-operator and ingress-nginx use, which the old
marker missed because it required a trailing slash a complete-value host
does not have), a host+org split with precedent at fluxcd's values.yaml,
and a single-quoted `repository:` where only double quotes were admitted.
All three carried a version-free latest@sha256: tag -- exactly the
profile the check exists for -- and passed the whole suite. The markers
now follow the collector's join semantics; this matches 43 files and
needs no new allowlist entries. The residual limit (a host split at an
arbitrary point inside cozystack/cozystack) is stated rather than
implied, along with why widening further is not worth it: a bare
cozystack/cozystack marker matches 65 files and would need 23 allowlist
entries for Makefiles and test fixtures.

More seriously, making the completeness oracle share shape logic
introduced a regression the previous independent oracle would have
caught: corrupting the last hex digit of every emitted digest left the
rewrite, retag and mirror suites entirely green, because both sides of
the comparison move together and the per-shape test matched only a
repository substring. The digest is the only part deciding which bytes
get retagged, so the per-shape test now asserts the exact canonical
repo@sha256:digest for every shape.

Declared extras were also narrowed by the refactor: parsing alone means a
ref inside a block scalar yields the enclosing block (which the ownership
filter then discards) and an extra that stops parsing after `make update`
yields nothing at all. Both are silent skips, in the one storage shape
defined as a manifest vendored verbatim from upstream. Extras get the
textual scrape again alongside parsing, with the callers' dedup absorbing
the overlap.

Finally, enumerating a path only proves the path is visited: a declared
extra whose ref the parser cannot reach stayed enumerated but uncollected
while both completeness checks passed. Added a test asserting a declared
extra yields its ref, using an unparseable Helm template with a
single-quoted split-host ref -- the shape that defeated both guards.

Also corrected the claim that the per-shape test compensates for the
shared oracle: it catches a shape being DELETED, not narrowed. A shape
losing its registry rejoin is caught by promote-retag and nightly-mirror,
which assert named packages, and the comment now says so.

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 21:38:26 +05:00
Aleksei Sviridkin f3a0f2d323 test(e2e): avoid head -n 0 in prevlog_cap for BSD portability
A zero container cap reached head -n 0, which GNU head treats as an empty
result but BSD/macOS head rejects ("illegal line count -- 0"), aborting a
local run. Special-case the zero cap to consume stdin instead, so the
helper behaves the same on both coreutils flavours.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-21 19:32:42 +03:00
Myasnikov Daniil 648eb42cae test(ci): make the image-ref guards see split-host and unparseable files
Both round-2 reviewers independently landed on the same two test-side
holes, and both are the guard being blind in exactly the way the bug it
guards was.

The completeness oracle derived its expected set from a grep for a
contiguous ghcr.io/cozystack/cozystack/...@sha256: string, so it could
not see any split-host shape: cilium, keycloak-operator, kubeovn, the OCI
artifact and seven shape-3 refs, 12 of 43 first-party refs in all. That
includes keycloak-operator, which the test's own rationale names as the
motivating profile. The fixture-discovery grep in the same file had
already been widened for this exact reason; the fix was never carried to
the oracle.

The oracle now applies the same shape knowledge to an independently
discovered file list, which is the axis that actually drifts -- what each
consumer disagreed about historically is which FILES it visits, not how a
ref is spelled inside one. To keep that sharing safe, shape coverage is
now pinned by its own test, so deleting a shape still fails something
even though both sides of the comparison would move together.

Sharing required splitting collect_refs_from_file out of
collect_image_refs. Behaviour is unchanged: same 67 refs collected before
and after, multus and the packages artifact both still present.

A YAML oracle cannot cover everything, though. A reviewer planted a
split-host ref in a Helm template, which yq cannot parse at all, and it
survived every test. A file-level marker check now backstops that: any
file carrying a first-party marker must be enumerated or explicitly
allowlisted. Two files are allowlisted today, each with its reason -- the
talos-csr-signer Dockerfile placeholder, and the documented capi gzip
gap. This also closes the secondary weakness that comparing on repo@digest
masks an unenumerated occurrence when the same image sits at an
enumerated path, which is live for that capi file.

Second hole: the postcondition's key-position branch had no coverage.
Replacing the whole pattern with the naive "${RC}@sha256:" -- the
narrowing rejected because cilium keeps tag: and digest: under separate
keys -- left all 22 tests green, so the branch that makes the split shape
visible was asserted by nothing. Added a cilium-shaped fixture at an
unenumerated path.

Both previously-surviving mutations are now caught. The oracle was also
pre-filtered by registry marker before parsing, cutting it from 32s to 5s.

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 21:08:15 +05:00
Aleksei Sviridkin 76f7267ff2 test(e2e): describe --previous as the preceding instance, not the first
kubectl logs --previous returns the immediately preceding terminated
container instance, not the original first one — after multiple restarts
the first instance is already gone. Reword the helper, the BATS and
Chainsaw failure handlers, and the diagnostics doc to match the actual
--previous semantics.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-21 18:57:01 +03:00
Aleksei Sviridkin 1ff83f5c06 test(e2e): fix false claim that a job timeout preserves the report
The catch-timeout comment justified the raised ceiling by asserting that
if: always() upload survives a job timeout. It does not: a 180-minute cap
ends the job cancelled, and the post-cancellation grace window is consumed
by the earlier report-collection step, so both cozyreport.tgz and the job
log are lost. Correct the comment to describe the real failure mode.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-21 18:56:19 +03:00
Myasnikov Daniil 9bd8c17971 fix(ci): shape-filter the promote postcondition and close test gaps
Review of the previous commit found the postcondition itself could abort
a release. It grepped the bare "X.Y.Z-rc.N" substring across the tree, and
three live files carry such strings without being image refs:
kubevirt-csi-driver's go.sum pins github.com/golang/protobuf v1.4.0-rc.2
-- a version cozystack actually cut, so promoting v1.4.0 would have
failed on a Go checksum line -- metallb_test.yaml names v1.5.0-rc.2 in a
comment, and the console pnpm-lock.yaml carries rolldown@1.0.0-rc.15. The
remediation the error message gave would have made things worse: declaring
go.sum as a ref source would sed a checksum file.

Match the version only in image-reference position now: after an
image/repository/tag key with no intervening '#', or immediately before an
@sha256 digest. Narrowing it to "${RC}@sha256:" alone would have been
wrong -- cilium keeps `tag: v1.5.0` and `digest:` under separate keys, so
the version is not always adjacent to the digest. Verified against four
real rc versions that previously aborted.

The suite also only tested the fail-closed direction, which is why that
defect survived, and the collector had no completeness guard at all: a
first-party ref added at an unenumerated path on a `latest@` tag was
mirrored by nothing and passed all eighteen tests. Since the mirror's
failure mode is the worst of the three -- the published nightly points at
the private build registry -- that gap mattered most where coverage was
thinnest.

Added: a fail-open test using the three real rc versions above; a
collector completeness diff that discovers refs by content and fails on
any the enumeration misses; coverage for the OCI-artifact shape, whose
deletion previously left every suite green; and an unreadable-file test,
since a read error was being treated as "no match" and silently skipped.

The component-versioned test was vacuous -- its rc version shared no
substring with the kamaji tag it claimed to protect, so it passed by
construction. It now uses an rc version sharing kamaji's 0.19.0 prefix,
which a loosely-anchored implementation would corrupt.

Round-trip fixture discovery matched a combined 'ghcr.io/cozystack/
cozystack/' prefix, which skips keycloak-operator (host in a sibling
registry: key) and kubeovn (host in global.registry.address). It now
matches 'cozystack/cozystack' and stamps split `tag:` keys too.

Also documented two pre-existing gaps rather than half-fixing them: the
nightly host rewrite is a contiguous-substring replace and cannot reach a
split host, and a ref inside a gzip is invisible to every tool including
the postcondition, which skips binaries. Both are host-dimension defects;
the version dimension this change owns is unaffected, because the version
always lives in a tag key regardless of where the host sits.

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 20:42:28 +05:00
Aleksei Sviridkin 07c67a8913 test(e2e): capture previous container logs and stop killing the snapshot
Root-causing a crash-loop failure was blocked twice over by the
diagnostics collected on failure.

`kubectl logs` shows only the current container instance, so for a
crash-looping pod the suites' `podLogs` collector captured the Nth
restart -- a fresh replay of the same startup path -- while the
evidence that explained the failure sat in the first, interrupted
instance. Chainsaw's podLogs collector has no `previous` option, so
add e2e-capture-previous-logs.sh and drive it from the global catch
and from the BATS failure hook. It discovers containers with
restartCount > 0 at runtime (init containers included, where an
interrupted bootstrap lives), orders the app suites' namespace first,
and dumps each to the job log and the snapshot tree. Containers that
never restarted are skipped, so a healthy run emits one line rather
than a page of "previous terminated container not found".

Every way that capture can come up short is named rather than
inferred, because kubectl exits 1 for a garbage-collected container,
an RBAC denial and an unreachable node alike: the container cap, a
pod list that never returned (which is not the same as nothing having
restarted), a read that timed out versus one kubectl refused and the
reason it gave, and a previous instance that produced no output. A
partial capture is kept and marked truncated rather than deleted --
it is the only copy of a log already gone from the cluster.

The crust-gather snapshot was killed by the Chainsaw op containing it.
A script op is bounded by timeouts.exec (5m), and the catch's own
wall-clock wrappers summed to 960s, so the script was SIGKILLed
partway through -- taking out the process group mid-write and skipping
the data-plane capture entirely. Give each catch op an explicit
timeout above the budgets it contains, so an inner timeout fires
instead and leaves a partial, still-servable snapshot behind. The
data-plane leg drops from 600s to 300s at that call site so both legs
fit one envelope; the BATS hook keeps the wider budget, having no
containing op.

Set crust-gather --duration explicitly as well. It is the collection
phase's own budget, it defaults to 60s, and on elapse it abandons the
collection and skips its finish step, leaving a truncated tree with no
indication anything is missing. The outer wall-clock has to exceed it
because the API discovery that runs first is not covered by that
budget at all -- with an unhealthy aggregated APIService, the state a
failed run is usually in, discovery alone can consume the whole
timeout before a single object is written. Send each collector's
output to a log beside its snapshot instead of /dev/null, with a
complete/truncated/failed verdict, so that question is answerable
from the artifact.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-21 18:37:07 +03:00
Andrey Kolkov 64097dc455 fix(platform): forward backupStorage overrides to the backupstrategy-controller Package (#3333)
## What this PR does

Fixes #3245.

The documented admin override for the `cozy-default` backup S3
coordinates — `spec.components.backupstrategy-controller.values` on the
`Package cozystack.cozystack-platform`
(`docs/operations/backup-classes.md`, "Admin overrides for
`cozy-default`") — had no effect: the `cozystack-platform` PackageSource
exposes exactly one component (`platform`), so the operator silently
ignored the block, and the platform chart rendered the child `Package
cozystack.backupstrategy-controller` via the `.default` helper with no
components argument, so nothing forwarded `backupStorage` either. Admins
running against an external or TLS S3 had no working supported path to
set `backupStorage.endpoint` / `region` / `systemSecretName`.

This PR wires the override through the platform chart, the same way
`bundles/iaas.yaml` forwards kubevirt and gpu-operator values:

- `packages/core/platform/values.yaml` gains a top-level `backupStorage:
{}` value.
- `templates/bundles/system.yaml` renders it into the emitted `Package
cozystack.backupstrategy-controller` as
`components.backupstrategy-controller.values.backupStorage`, from where
the cozystack operator merges it into the `backupstrategy-controller`
HelmRelease over the chart defaults. When `backupStorage` is unset the
components block is omitted and the emitted Package is byte-identical to
the historical render.
- `docs/operations/backup-classes.md` now documents the working path
(`spec.components.platform.values.backupStorage`) and explicitly calls
out the two plausible-looking paths that do not work (the old documented
one, and patching the child Package directly, which the platform
helm-reconcile reverts).
- A new helm-unittest suite
(`packages/core/platform/tests/bundles_backupstorage_wiring_test.yaml`)
is the behavioural regression test: it proves the unset case emits no
components block and the set case lands the values on the child Package,
on both `isp-full` and `isp-full-generic`.

Verified with `make test` in `packages/core/platform` (22 suites, 91
tests, all green). `make show` requires a live cluster (`cozyhr` +
`lookup` in `templates/repository.yaml`), so the unittest render of
`templates/bundles/system.yaml` is the offline verification path.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#622
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Walking the trigger map against the diff: the change to
`packages/core/platform/values.yaml` reaches **cozystack/website** twice
— `content/en/docs/next/operations/configuration/platform-package.md`
(hand-written table of `spec.components.platform.values.*` keys gains
`backupStorage`) and
`content/en/docs/next/operations/services/backup-classes.md` (mirrors
the broken override example this PR corrects). The follow-up is
cozystack/website#622, which also carries the superseded website#547
content and covers both next/ and v1.4/ docs, marked not to merge before
this PR lands; its v1.4 copy assumes this fix is backported to
release-1.4. No other downstream repository restates the touched files:
the templates change is chart-internal, no app `values.schema.json`,
CRD, variant list, `hack/` file, or namespace changed.

### Release note

```release-note
fix(platform): the documented admin override for the cozy-default backup S3 coordinates now works — set spec.components.platform.values.backupStorage on the cozystack.cozystack-platform Package and the platform chart forwards it to the backupstrategy-controller component (previously the documented spec.components.backupstrategy-controller path was silently ignored)
```
2026-07-21 19:30:38 +04:00
Myasnikov Daniil 09323fff13 fix(ci): cover every image-ref storage shape in promote, retag and mirror
Promotion rewrote the rc version substring across the depth-2 package
values.yaml plus packages/apps/kubernetes/images/*.tag alone, on the
premise that the kubernetes app was the only one whose .tag files carry
the cozystack version. Nine other .tag files do, and system/multus seds
its ref straight into a vendored upstream manifest. v1.6.0 was therefore
staged with 33 of 55 first-party refs still reading v1.6.0-rc.4.

promote-retag.sh and nightly-mirror.sh had the same blind spot from the
other direction: both scanned only the depth-2 values.yaml, so those
images were never retagged to the stable version (30 refs selected
before, 42 after) and never mirrored to the public registry for a
nightly. A retag miss merely leaves an image short of a tag, since the
digest still resolves inside one registry; a mirror miss is worse,
because the host rewrite walks the same file list and a ref that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.

The three call sites each carried their own idea of where a ref can
live, which is how they drifted apart. Move that knowledge into
hack/lib/image-refs.sh as the single enumeration -- image_ref_files()
for the files, collect_image_refs() for the refs inside them -- and have
all three source it, so a new storage location is declared once and
reaches every consumer.

The rewrite itself moves out of promote-rc.yaml into
hack/promote-rewrite-tags.sh. Being workflow-inline is why this was only
observable by cutting a release: there was nothing to unit test. It now
also asserts its own postcondition, scanning wider than it rewrites and
failing the promotion when any ref survives in a location the
enumeration does not know about, rather than shipping it quietly.

Component-versioned images (kamaji's v0.19.0-cozystack.N) and
third-party pass-through images do not ride the cozystack version line
and are deliberately left alone; a test pins that so a future fix here
cannot over-reach. capi-providers-cpprovider is a documented known gap:
its ref is also embedded in a gzipped copy that the chart actually
ships, and rewriting only the readable copy would diverge the two.

docs/agents/image-refs.md records the contract -- three tag classes,
three storage shapes, the invariants, and which consumer does what.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 20:14:24 +05:00
myasnikovdaniil 3ad3c2e611 fix(kubernetes): move kubevirt-csi-driver base off quay.io/centos to almalinux (#3400)
## What this PR does

`packages/apps/kubernetes/images/kubevirt-csi-driver/Dockerfile` pinned
its runtime stage to `quay.io/centos/centos:stream9` by digest. Quay
serves **only currently-tagged manifests** — when a rolling tag moves,
the superseded digest immediately returns `MANIFEST_UNKNOWN`, with no
grace period.

`stream9` rolls roughly every six days, so the pin has already died
twice:

| Date | Fix | Digest pinned | Status today |
|---|---|---|---|
| Jul 15 | #3303 | `0c9db821…` → `a93e3316…` | `0c9db821` gone |
| Jul 21 | #3386 | `a93e3316…` → `3714c89f…` | `a93e3316` gone |

Each death broke `make build` on `main` and on every PR triggering a
full rebuild. Because the root `build` target is serial, a failure in
`packages/apps/kubernetes` (4th of 29) aborts the remaining packages and
leaves the shared build cache unwritten.

Repinning cannot fix this: the freshly pinned digest is always the
current live tag, so it is next in line to be retired. This is not
specific to CentOS — the same probe against
`quay.io/fedora/fedora:latest` and `quay.io/centos/centos:stream10`
found every historical digest unfetchable.

This PR keeps the digest pin and changes the *source* to a registry that
provably retains superseded manifests, making the pin durable rather
than decorative. Renovate policy is unchanged — `pinDigests` stays on,
and `almalinux:9` is an ordinary rolling tag on a retaining registry
exactly like the `golang`, `alpine` and `debian` bases already in the
grouped digest PR.

### Verification

Both bases were run locally with the Dockerfile's exact install line
(`dnf install -y e2fsprogs xfsprogs nfs-utils`):

| Binary | CentOS Stream 9 | AlmaLinux 9 |
|---|---|---|
| `mkfs.ext4` | `/usr/sbin/mkfs.ext4` | `/usr/sbin/mkfs.ext4` |
| `mkfs.xfs` | `/usr/sbin/mkfs.xfs` | `/usr/sbin/mkfs.xfs` |
| `blkid` | `/usr/sbin/blkid` | `/usr/sbin/blkid` |
| `mount` | `/usr/bin/mount` | `/usr/bin/mount` |
| `umount` | `/usr/bin/umount` | `/usr/bin/umount` |
| `fsck.ext4` | `/usr/sbin/fsck.ext4` | `/usr/sbin/fsck.ext4` |
| `fsck.xfs` | `/usr/sbin/fsck.xfs` | `/usr/sbin/fsck.xfs` |
| `mount.nfs` | `/usr/sbin/mount.nfs` | `/usr/sbin/mount.nfs` |

Every path `k8s.io/mount-utils` shells out to is identical. Package
versions:

| Package | CentOS Stream 9 | AlmaLinux 9 |
|---|---|---|
| `e2fsprogs` | 1.46.5-8.el9 | 1.46.5-8.el9 |
| `xfsprogs` | 6.4.0-7.el9 | 6.4.0-7.el9 |
| `nfs-utils` | 2.5.4-44.el9 | 2.5.4-42.el9_8 |
| `util-linux` | 2.37.4-26.el9 | 2.37.4-25.el9 |
| `glibc` | 2.34-274.el9 | 2.34-270.el9_8 |

Same upstream versions throughout; the el9 release-number lag is
expected since AlmaLinux is downstream of RHEL 9 while CentOS Stream 9
is upstream. `e2fsprogs` and `xfsprogs` — the two that govern
`mkfs`/`fsck` behaviour — are identical NVRs. `glibc` is not linked at
all: the driver is built `CGO_ENABLED=0`.

The pinned digest is the OCI image index covering `amd64`, `arm64`,
`ppc64le` and `s390x` — the same architecture set as the previous base.

Retention was verified empirically rather than assumed: four superseded
`docker.io/library` digests taken from this repo's own git history
(`golang:1.26`, `golang:1.26-alpine`, `alpine:3.24`, `alpine:3.22`) all
still resolve after their tags moved.

### Scope

This addresses the immediate recurring breakage only. The broader
concern in #3305 — that any retired digest blocks the entire platform
build, with no advance warning and no fast path to repin — is not solved
here and remains open.

### Release note

```release-note
fix(kubernetes): move the kubevirt-csi-driver runtime base from quay.io/centos/centos:stream9 to docker.io/library/almalinux:9. Quay retires superseded manifests as soon as a rolling tag moves, which broke builds twice in six days; AlmaLinux 9 provides an equivalent RHEL 9 userspace on a registry that retains them, so the digest pin remains valid over time. No functional change to the driver.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated the container runtime environment to use AlmaLinux 9 instead
of CentOS Stream 9.
* Kubernetes CSI driver behavior and startup configuration remain
unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-21 19:24:11 +05:00
Myasnikov Daniil a4b5b9d57b docs(release): update workflow cancellation condition
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 18:41:45 +05:00
Myasnikov Daniil 792951b9c6 fix(ci): make workflow cancellation effective
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 17:57:48 +05:00
Myasnikov Daniil 14af87296f fix(kubernetes): move kubevirt-csi-driver base off quay.io/centos to almalinux
quay.io serves only currently-tagged manifests: when a rolling tag moves,
the superseded digest immediately returns MANIFEST_UNKNOWN. The pinned
`centos:stream9` digest has died twice in six days (#3303, #3386), each
time breaking `make build` on main and on every PR that triggers a full
rebuild.

Repinning is not a fix -- the newly pinned digest is always the live tag,
so it is next in line to be retired. Move the runtime stage to a registry
that provably retains superseded manifests instead, which makes the digest
pin durable rather than decorative. Pinning policy is unchanged.

Verified equivalent: after `dnf install -y e2fsprogs xfsprogs nfs-utils`,
both bases provide mkfs.ext4, mkfs.xfs, blkid, mount, umount, fsck.ext4,
fsck.xfs and mount.nfs at identical paths. e2fsprogs and xfsprogs are
identical NVRs. The driver is built CGO_ENABLED=0, so the base libc is not
linked. The pinned digest is the OCI index covering amd64, arm64, ppc64le
and s390x -- the same architecture set as the previous base.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 17:56:11 +05:00
myasnikovdaniil 741a5acc4a fix(opensearch-operator): use RollingUpdate maxSurge=0 instead of Recreate (#3319)
## What this PR does

Fixes a Helm-upgrade failure in the `opensearch-operator` chart that
blocks the new release-upgrade E2E lane (added in #3276).

Upgrading from a chart that shipped no explicit strategy (v1.5.x) to a
v1.6.0-rc build fails validation immediately, helm-controller retries
it, and the HelmRelease never becomes Ready:

```
Helm upgrade failed for release cozy-opensearch-operator/opensearch-operator:
Deployment.apps "opensearch-operator-controller-manager" is invalid:
spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is Recreate
```

Observed in the ["Upgrade E2E Test" job of run
29480724982](https://github.com/cozystack/cozystack/actions/runs/29480724982)
— `UpgradeFailed ... (x19 over 11m)`, with the HelmRelease never
reaching Ready.

### Root cause

#3040 (commit db797b1d) set the single-replica operator Deployment to
`strategy.type: Recreate` — a reasonable goal: with leader election
gated off on one replica, a default `RollingUpdate` briefly runs two
uncoordinated managers (`maxSurge`) during a rollout, and `Recreate`
closes that window.

The problem surfaces only on **upgrade from an older chart, under
server-side apply**:

- A v1.5.x release shipped no strategy at all, so the apiserver
**defaulted** `spec.strategy.rollingUpdate` to `25%/25%`, and the
`helm-controller` field manager never owned that block. Its
`managedFields` entry covers `f:replicas`, `f:selector` and
`f:template`, but not `f:strategy`.
- helm-controller applies **server-side** by default
(`Install.ServerSideApply` defaults to true, `Upgrade.ServerSideApply`
to `auto`; `internal/operator/package_reconciler.go` sets neither), and
SSA does not remove a field the applier never owned merely because the
new intent omits it.
- So the v1.6.0-rc manifest's `type: Recreate` merges onto a live object
that still carries the defaulted `rollingUpdate`, and the apiserver
rejects the result with `FieldValueForbidden`.

The **client-side** path is unaffected, which is why this surfaces
through helm-controller's SSA but not a local `helm upgrade`:
`DeploymentSpec.Strategy` carries `patchStrategy:"retainKeys"`
(`k8s.io/api@v0.34.1/apps/v1/types.go:395`), so a 3-way merge emits
`{"spec":{"strategy":{"$retainKeys":["type"],"type":"Recreate"}}}`,
which merges to a valid `{"strategy":{"type":"Recreate"}}` and does
clear the block.

This is a genuine product upgrade bug, not a test-harness issue — #3276
only adds the upgrade lane that exposes it; it does not touch
`opensearch-operator`.

> An earlier revision of this PR attributed the failure to Helm's
client-side 3-way merge. That was wrong, and the correction is
@lexfrei's — see [his
review](https://github.com/cozystack/cozystack/pull/3319#pullrequestreview-4713156867).
The mechanism above is now reproduced end to end (below).

### The fix

Keep `type: RollingUpdate` on the single-replica path and set
`rollingUpdate.maxSurge: 0 / maxUnavailable: 1` instead of switching to
`Recreate`:

- `maxSurge: 0` caps total pods at `replicas`, so the deployment
controller must scale the old manager down before it can scale the new
one up — unlike the default 25% surge, which starts a second manager
while the first is still fully live.
- This is **not** `Recreate`'s exclusivity, and the PR no longer claims
it is. `rolloutRecreate` gates scale-up on `oldPodsRunning()`, which
counts non-terminal `Status.Phase`, so `Recreate` waits for the old pod
to be gone. `rolloutRolling` has no such gate: `NewRSNewReplicas`
derives `currentPodCount` from `GetReplicaCountForReplicaSets`, which
sums `rs.Spec.Replicas` (desired, not live). The ordering that is
guaranteed is over ReplicaSet desired counts, not pod lifecycles, so a
draining old manager can still overlap the new pod within
`terminationGracePeriodSeconds` (10s here).
- Because `strategy.type` never becomes `Recreate`, the forbidden merge
cannot occur on any apply path. It also recovers anyone already on an rc
build with `Recreate`, since `Recreate -> RollingUpdate` is permitted.
- This matches the idiom already vendored in this repo:
`packages/system/kubeovn/charts/kube-ovn/templates/central-deploy.yaml:11-15`
uses `RollingUpdate` with `maxSurge: 0 / maxUnavailable: 1`.

Multi-replica installs (`manager.replicaCount > 1`) are unchanged: they
keep the default `RollingUpdate` with leader election enabled.

The change is carried in `patches/leaderElection.diff` so it survives a
chart re-vendor via `make update`; the patch applies to pristine
upstream 2.8.0 with no fuzz and reproduces the committed strategy block
exactly.

### Alternatives considered

| Option | Why not |
| --- | --- |
| Keep `Recreate`, but make the chart own `rollingUpdate` first (ship an
explicit block, then switch `type` in a later release) | This is the
only option that preserves real exclusivity, but it needs two releases
and only works once every install has reconciled the intermediate one.
Disproportionate for a single-replica operator whose residual overlap is
a draining manager. |
| Pre-upgrade hook / Job that patches or deletes the Deployment strategy
before the upgrade | Heavy: needs a kubectl image, a ServiceAccount and
RBAC to `patch`/`delete` Deployments, and hook ordering; runs on every
upgrade even when unneeded; adds standing attack surface —
disproportionate for a scalar field transition. |
| `helm.sh/resource-policy` / force-replace | `resource-policy: keep`
only governs deletion on uninstall (irrelevant). Helm has no
per-resource "force recreate" annotation; `--force` / HelmRelease
`spec.upgrade.force` is global and disruptive and not controllable from
within the chart. |
| Plain revert to default `RollingUpdate` (25%/25%) | Fixes the
rejection but starts a surge pod while the old manager is still fully
live — the window #3040 set out to close. `maxSurge: 0` narrows it to a
draining old pod without leaving `RollingUpdate`. |

### Verification

The mechanism and the fix were reproduced on kind v1.33.1 (a bare
apiserver is enough — the failure is defaulting + SSA + validation):

| Path | Result |
| --- | --- |
| SSA install of the v1.5.x manifest (no strategy) | apiserver defaults
`rollingUpdate: 25%/25%`; `managedFields` shows `helm-controller` does
not own `f:strategy` |
| SSA upgrade to `type: Recreate` | fails with the exact production
error above |
| Client-side apply of the same pair | succeeds; `rollingUpdate` is
cleared, live strategy becomes `{"type":"Recreate"}` |
| SSA upgrade to `maxSurge: 0` (this PR) | succeeds |
| SSA of `maxSurge: 0` onto a live `Recreate` Deployment (rc recovery) |
succeeds |
| Rollout under `maxSurge: 0` vs `Recreate` | `maxSurge: 0`: two pods
coexisted within 1s (old Terminating, new ContainerCreating).
`Recreate`: one pod, waited the full drain. This is the evidence for the
"not exclusivity" wording above. |

### Note on regression coverage

`tests/leader_election_test.yaml` was never executed:
`hack/helm-unit-tests.sh` runs a package's suite only when the package
Makefile defines a `test:` target, and this one did not — so the
coverage the previous revision of this PR claimed did not exist. This PR
adds the target (matching `packages/system/etcd-operator/Makefile`),
which revives the suite; the repo-wide runner now picks the package up
and a revert to `Recreate` fails it. Thanks to @lexfrei for catching
that.

The apiserver-side merge that produces the invalid object still needs a
live cluster, so the behavioural regression test remains the
release-upgrade E2E lane in #3276 (which this fix unblocks).

### Follow-ups filed

- #3324 — `packages/system/ouroboros` has the same latent shape
(`Recreate` gated on `controller.mode == "external-dns"`, while
`controller.mode` defaults to `coredns`). Not fixed here, and it likely
needs a different fix, since that mode genuinely depends on exclusivity.
- #3325 — `make update` does not reproduce this package's vendored
template: three hand-edits are not captured in `patches/`, and one of
them breaks rendering when `manager.extraEnv` is set. Pre-existing on
`main`; an earlier revision of this body claimed the regeneration was
exact, which was wrong.

### Downstream repositories

- [x] No downstream repository is affected by this change

<!-- Walked the trigger map in docs/agents/contributing.md against the
diff: the change touches only a vendored packages/system operator chart
template + its patch + its helm-unittest suite + the package Makefile
test target. No packages/apps or packages/extra add/rename/remove, no
values.schema.json / enum / default change, no node contract, no
annotations/labels, no hack/ or ApplicationDefinition change.
opensearch-k8s-operator is a third-party upstream (not a cozystack
downstream), and the fix lives in patches/leaderElection.diff so it
survives make update. -->

### Release note

```release-note
fix(opensearch-operator): fix a Helm upgrade failure on single-replica installs. Upgrading a release that predates an explicit Deployment strategy failed under server-side apply, because switching strategy.type to Recreate does not clear the apiserver-defaulted spec.strategy.rollingUpdate block that the chart never owned, and the merged object is rejected. The operator now uses RollingUpdate with maxSurge=0/maxUnavailable=1, which keeps the transition legal on every apply path and recovers releases already stuck on Recreate.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Updated OpenSearch Operator controller rollout behavior for
single-replica installs to use a safer rolling update configuration.
* Leader election is now applied only when multiple replicas are
configured, reducing the chance of conflicting controller activity.
* **Tests**
* Expanded Helm unittest assertions for the controller deployment
rollout strategy and leader-election behavior for single- and
multi-replica scenarios.
* **Chores**
  * Added a `make test` target to run Helm unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-21 17:17:03 +05:00
myasnikovdaniil 960d853706 feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) (#3273)
## What

Adds typed `waitStrategy` + `healthCheckExprs` fields so a package/app
can declare the Flux helm-controller v1.5.0 kstatus health gate for the
custom resource it wraps — making `HelmRelease.Ready` mean "the CR is
actually healthy" instead of "helm applied the CR". This is the
operator/apiserver foundation of the kstatus epic (#2642); the API
surface was reviewed and approved by the API owner in #3263.

Commits (per the review's "isolated step-0" request):
1. `build(deps)` — the dependency bump (see below).
2. `feat(api)` — the fields, plumbing, tests, and regenerated code.
3. `fix(api)` — match the openapi code-generator pin to the 0.35 runtime
(fixes an aggregated-apiserver crash the bump surfaced).
4. `fix(api)` — declare `OpenAPIModelName` on the apps types so SSA
resolves catalog kinds.
5. `docs(api)` — note that `disableWait` silently defeats
`healthCheckExprs`.

## API surface

Typed fields on **both** HelmRelease-building paths:
- `ApplicationDefinition.spec.release` (apiserver / catalog apps →
`convertApplicationToHelmRelease`)
- `ComponentInstall` (operator / platform packages →
`buildHelmReleaseSpec`)

```go
// WaitStrategy maps to HelmReleaseSpec.WaitStrategy.Name — a deliberate scalar
// simplification of the upstream {name} object. One of poller|legacy.
WaitStrategy string
// HealthCheckExprs maps to HelmReleaseSpec.HealthCheckExprs (the upstream
// kustomize.CustomHealthCheck type verbatim).
HealthCheckExprs []kustomize.CustomHealthCheck
```

`config.ResolveWaitStrategy` is shared by both builders and **couples
the default**: `healthCheckExprs` are only evaluated under the `poller`
strategy, so when expressions are set and no strategy is given, the
generated HelmRelease defaults to `poller`. A package that sets only
`healthCheckExprs` is therefore self-contained, independent of the
controller's global default.

## The dependency bump (commit 1)

The `WaitStrategy` / `kustomize.CustomHealthCheck` /
`HelmReleaseSpec.HealthCheckExprs` types do not exist below
`helm-controller/api` v1.5.0 — the runtime is already v1.5.0 (shipped by
#2602), so this closes a Go-types-vs-runtime skew. It is **not** an
isolated bump: v1.5.x requires **k8s 0.35 + controller-runtime 0.23**,
cascading the k8s stack 0.34 → 0.35 (and the go directive to 1.26).

cozystack replaces `k8s.io/apimachinery` with a fork carrying the
still-open upstream `ConvertToVersion` patch (k/k#135537). That single
commit rebases cleanly onto apimachinery v0.35.0; published at
`cozystack/apimachinery@release-1.35-pr135537`, and `go.mod` pins its
pseudo-version. **The repo builds with zero source changes under the new
stack.**

> Maintainer note: please confirm the `release-1.35-pr135537` fork
branch naming, or fold it into the `cozystack` branch as you prefer —
the pin can be updated to match.

## Not included (conscious YAGNI)

`disableWait` is intentionally omitted. The only known need (the
kubernetes tenant-addon deadlock) was resolved child-side, and
`waitStrategy: legacy` is not equivalent (legacy still waits
Helm-v3-style; `disableWait` skips waiting entirely). Add it later when
a concrete leaf needs a true no-gate escape hatch.

## Tests

- `config.TestResolveWaitStrategy` — the couple-the-default logic
(unset+exprs→poller; explicit legacy honored; unset+no-exprs→nil).
- Operator + apiserver cases mirror each other
(`TestBuildHelmReleaseSpecHealthCheckExprs` /
`TestConvertApplicationToHelmRelease_HealthCheckExprs`).
- `go build ./...` clean, `go vet` clean, `gofmt` clean, codegen
regenerated and committed.

The 4 backupstrategy CRDs regenerate with benign k8s-0.35 PodSpec
description drift (DynamicResourceAllocation alpha→stable, toleration
Lt/Gt, resizePolicy note) — no functional change.

## Follow-up

Phase 3 (per-leaf adoption — postgres/kafka/mongodb/etc. declaring their
CEL via these fields) builds on this. The full CRD→CEL catalog is
maintained in the epic's readiness audit.

Refs #2642

```release-note
feat(api): add `waitStrategy` and `healthCheckExprs` to ApplicationDefinition and ComponentInstall, so a HelmRelease reports Ready only once the custom resource it wraps is actually healthy
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added configurable CEL health-check expressions for application
releases and package components.
* Added `waitStrategy` support (`poller` and `legacy`) with automatic
`poller` selection when health checks are configured and no strategy is
set.
* Propagated these settings into generated Helm releases, affecting
readiness/wait behavior.

* **Documentation**
* Updated Application and PackageSource schemas to include
`healthCheckExprs` and `waitStrategy`.
* Extended related Kubernetes CRD documentation, including projected pod
certificate annotations and workload reference fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-21 17:02:37 +05:00
Timofei Larkin ebe111c439 feat(gateway): advertise HTTP/2 via ALPN on Gateway API listeners
Browsers negotiate HTTP/2 exclusively through TLS ALPN "h2". Cilium's
Gateway API implementation ships with enable-gateway-api-alpn=false, so
every Gateway listener advertised no ALPN protocols and every browser
silently fell back to HTTP/1.1 — a regression against the ingress-nginx
path, which advertised h2 out of the box.

Enable ALPN (h2, then http/1.1) by default on the client-facing hop:

- packages/system/cilium: default gatewayAPI.enableAlpn=true.
- platform chart: new gateway.http2 value (default true), threaded as
  inline component values into the cozystack.networking Package for the
  isp-full and isp-full-generic bundles, so an explicit false overrides
  the system/cilium valuesFiles default either way.
- apps/kubernetes: tenant-cluster cilium gets enableAlpn=true whenever
  the gatewayAPI addon is enabled; opt out per cluster via
  addons.cilium.valuesOverride.

The backend hop is unchanged: gateway-to-upstream stays HTTP/1.1 unless
a Service opts in per GEP-1911 via appProtocol kubernetes.io/h2c (that
support is switched on together with ALPN; no cozystack-shipped Service
declares such a value today).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-07-21 14:40:30 +03:00
Andrey KolkovandClaude Fable 5 7d117679d4 docs(backups): document bucketName's two modes, cover isp-hosted in the wiring test
Branch-review on this PR: for provisionBucket: false the
backupstrategy-controller.bucketName helper returns
.Values.backupStorage.bucketName verbatim as the real S3 bucket name
and every strategy CR plus the Velero BSL bakes it in, so the
external-S3 section must require updating bucketName (the Secret's
bucketName key alone is not read by the strategies), and the knob
table must state the two-mode semantics instead of the COSI-only one.
Also route the external-S3 instruction through the platform component
override instead of naming raw chart values, and add the isp-hosted
variant case to the forwarding test (92 tests green).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 0cd3fe20b8 docs(backups): credentials are projected for RestoreJob reconciles too
The projector runs from both the BackupJob and RestoreJob controllers
(restorejob_controller.go), not only before BackupJobs — keep the
monorepo doc aligned with the website copy (cozystack/website#622).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 93de6adc04 docs(backups): fix HelmRelease interval, FDB API group, BSL wait syntax, leader-election phrasing
Same corrections as the website copy (cozystack/website#622 round 4):
the re-render interval is 5m via the operator's helmrelease-interval
flag (not a 10-minute Flux default), the FoundationDB backup CR group
is apps.foundationdb.org, the BSL readiness check uses valid kubectl
wait --for=jsonpath syntax, and the projector's first synchronous round
is gated on leader election rather than pod readiness.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 a5dff22c51 docs(backups): fix PromQL examples and bucket-name phrasing, document namespace knob and bslEnabled gap
Keep the monorepo doc aligned with the website copy
(cozystack/website#622 branch-review round 3): full
cozystack_backup_credentials_projection_* metric names with range
vectors in the alerting examples, the intro no longer conflates the
Bucket CR name with the COSI-assigned S3 bucket name, the knob table
gains the backupStorage.namespace row, and the external-S3 section
states that the Velero BSL derives from the same backupStorage block
while velero.bslEnabled is not carried by the override path.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 b4c1e44fd5 docs(backups): namespace the flux reconcile example
The backupstrategy-controller HelmRelease lives in
cozy-backup-controller (packages/core/platform/sources/
backupstrategy-controller.yaml), so the un-namespaced command fails
out of the box.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 e74c8cb459 docs(backups): pin the placeholder-credentials upgrade note to v1.5
Same note as the website copy (cozystack/website#622 review round 2):
name the explicit version boundary instead of 'pre-PR / After this PR',
so the two documents cannot diverge — the empty-string defaults and
useSystemBucket shipped in release-1.5.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 1422f8b78a fix(platform): forward backupStorage overrides to the backupstrategy-controller Package (#3245)
The documented admin override for the cozy-default backup S3
coordinates (spec.components.backupstrategy-controller.values on the
cozystack.cozystack-platform Package) had no effect: that PackageSource
exposes only the platform component, so the operator silently ignored
the block, and nothing in the platform chart forwarded backupStorage to
the child Package.

Wire it the same way bundles/iaas.yaml forwards kubevirt / gpu-operator
values: a new top-level backupStorage value on the platform chart is
rendered into the emitted cozystack.backupstrategy-controller Package
CR as components.backupstrategy-controller.values.backupStorage, from
where the cozystack operator merges it into the
backupstrategy-controller HelmRelease over the chart defaults. When
backupStorage is unset the emitted Package is unchanged.

Update docs/operations/backup-classes.md to the working override path
(spec.components.platform.values.backupStorage) and call out the two
plausible-looking paths that do not work. Add a helm-unittest suite
covering the no-op default and the forwarding on isp-full and
isp-full-generic.

Fixes #3245

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-21 14:47:08 +04:00
Andrey Kolkov e73ad68a85 fix(backups): support cnpg-operator with barman plugin (#3313)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->


## What this PR does

Fixes #3300. Migrates CNPG Postgres backups off the deprecated native
`spec.backup.barmanObjectStore` (removed in CNPG 1.29; the
`barman-cloud-*` binaries it needs are absent from the `standard` image
variant keycloak-db pins, and the bare system-flavor tags apps/postgres
pins are deprecated upstream) onto the [CloudNativePG Barman Cloud
plugin](https://github.com/cloudnative-pg/plugin-barman-cloud), across
every Postgres backup surface: `apps/postgres`, `system/keycloak`, and
the `cozy-default-cnpg` BackupClass strategy driver. The chart renders
`spec.plugins` + a `barmancloud.cnpg.io/ObjectStore` CR; the
backup-controller driver SSA-applies an `ObjectStore` and patches
`spec.plugins`; `Backup`/`ScheduledBackup` use `method: plugin`. The
`postgres-operator` chart vendors and co-locates the
`plugin-barman-cloud` operator (with its cert-manager-issued mTLS) so
CNPG can discover it.

One platform-level change is required for the plugin to run and ships
with this PR (see the upgrade notes in
`docs/operations/backup-classes.md` and the release note below): the
CloudNativePG operator is bumped to 1.28.1 (also fixing an operator/CRD
version skew that broke all backups). The other prerequisite — the
LINSTOR scheduler admission webhook stripping
`initContainers[].restartPolicy` cluster-wide, which broke the plugin's
restartable sidecar — is already fixed on `main` (f4aef7305 re-vendors
`linstor-scheduler` to upstream chart 0.3.1 / extender v0.3.6, which
keeps the webhook and stops stripping unknown pod fields), so after the
rebase this PR no longer touches `linstor-scheduler`; just do not hold
that package back on an older version when rolling out the plugin.

Supersedes #3306 (the keycloak `system`-image workaround): keycloak-db
now backs up via the plugin on the `standard` image, so that stopgap is
no longer needed.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: docs/operations/backup-classes.md changed; a website mirror
follow-up is warranted (not yet opened).
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): migrate CNPG Postgres backups to the CloudNativePG Barman Cloud plugin

Postgres backups (apps/postgres, system/keycloak, and the cozy-default-cnpg BackupClass strategy) now use the CloudNativePG Barman Cloud plugin (spec.plugins + a barmancloud.cnpg.io/ObjectStore, method: plugin) instead of the deprecated native spec.backup.barmanObjectStore (removed in CNPG 1.29; the barman-cloud-* binaries it needs are absent from the standard image variant keycloak-db pins, and the bare system-flavor tags apps/postgres pins are deprecated upstream). The barman-cloud plugin operator is installed alongside the CNPG operator.

One platform-level change rides along and requires operator attention on upgrade:
- The CloudNativePG operator is bumped to 1.28.1 (fleet-wide minor upgrade); this also fixes an operator-newer-than-its-CRDs skew that made the operator report "instance manager was restarted during backup" and fail every backup. The vendored chart ships the CNPG CRDs as ordinary release manifests (crds.create), so helm upgrade updates them with the operator; the skew came from the image.tag pin outrunning the chart's CRDs, which this PR removes.
- The plugin's restartable sidecar requires the linstor-scheduler admission webhook that no longer strips initContainers[].restartPolicy (upstream chart 0.3.1 / extender v0.3.6, shipped separately on main); do not hold the linstor-scheduler package back on an older version when rolling out the plugin.
```

Close #3300, #3246

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Switched PostgreSQL CNPG backups/restores to the plugin-based Barman
Cloud workflow, using dedicated ObjectStore resources for backup and
recovery.
* Added the Barman Cloud CNPG-I plugin chart (deployment, TLS, RBAC,
leader election).
* **Bug Fixes**
* Improved backup/restore attachment messaging when the CNPG Cluster
isn’t ready yet.
* Preserves server-name behavior to avoid WAL-archive prefix issues;
clarified endpoint CA handling.
* **Changes**
* Helm integration no longer uses the legacy
`spec.backup.barmanObjectStore` path; updated gating, credential
projection, and ObjectStore-related RBAC/cleanup.
* Removed the linstor-scheduler admission webhook; updated
CloudNativePG/operator versions and CRDs.
* **Tests**
* Updated controller and Helm-unittest checks for the plugin flow;
added/extended end-to-end backup/restore coverage.
* **Documentation**
* Expanded upgrade/migration and backup documentation; updated backup
example materials and comments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-21 14:45:52 +04:00
cozystack-ci[bot] d95685c427 Prepare release v1.6.0-rc.4
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-07-21 10:31:23 +00:00
myasnikovdaniil cc66e22952 fix(ci): mirror and retag images whose digest is embedded in the values tag (#3375)
## What's broken

`collect_refs` in both `hack/nightly-mirror.sh` and
`hack/promote-retag.sh` recognises three values shapes. The one most
package Makefiles actually write is not among them:

```yaml
image:
  repository: <registry>/linstor-csi
  tag: main@sha256:<digest>      # digest embedded in the tag
```

Package Makefiles set `.image.tag` to `"$(IMAGE_TAG)@$(digest)"` in a
single `yq` call rather than maintaining a separate `digest` key. That
matches neither the `{repository, tag, digest}` rule (no `digest` key)
nor, usefully, the single-string rule — that one matches the bare tag
value, which carries no repository, so `ref_repo()` reduces
`main@sha256:...` to the repository `main` and the ownership filter
drops it.

Being dropped is silent. Both scripts then report success having simply
never seen those refs.

The same emit-then-drop hits a second class, found in review: refs whose
registry host does not live inside `repository` at all.
`keycloak-operator` splits it into a sibling `registry` key, and
`kubeovn` keeps it in the document-level `global.registry.address` with
a bare `repository: kubeovn`. Both are built and pushed to `$REGISTRY`
by cozystack — `kubeovn` by the `cozystack/kubeovn-chart` wrapper repo,
whose own `make image` writes all three keys — and both were emitted
host-less and discarded as third-party.

## Impact

**Nightly (`nightly-mirror.sh`) — this is the one that broke.** The
image is never copied to GHCR, *and* the unconditional host rewrite at
the end of the script still repoints its repository at the destination
registry. The result is a published reference to an image that was never
pushed there.

Confirmed against the `:nightly` tag on GHCR — the two packages using a
recognised shape are present, and eight images across six packages using
this shape are absent:

| shape | packages | `:nightly` |
|---|---|---|
| `{repository,tag,digest}` | cilium | present |
| single-string full ref | cozystack-operator | present |
| `{repository, tag:<t>@sha256}` | kamaji, kilo, linstor-csi,
piraeus-server, linstor-gui, metallb-controller, metallb-speaker,
redis-operator | **absent** |

This surfaced as the first full nightly run failing at install: the
`e2e-image-prepuller` DaemonSet never became ready (0/3, 10-minute
timeout) because `linstor-csi` and `piraeus-server` 404 on GHCR. The
prepuller only templates kubeovn/linstor/cert-manager, so it exposed two
of the eight; the other six would have failed later at HelmRelease
install.

**Promotion (`promote-retag.sh`) — same blind spot, milder
consequence.** This retags within a single registry, so the digests
already exist and digest-pinned installs still resolve — there is no
dangling reference and no 404. What is missing is the `:<version>` tag
(and `:latest` under `MOVE_LATEST=1`) for those images, so a promoted
release does not carry the tags it claims to.

The registry shows how long that has been true for the two host-split
images. `keycloak-operator` carries exactly one tag, `latest`, though
the package has shipped since 2024-11. `kubeovn`'s newest
cozystack-versioned tag is `v1.14.11-v0.38.8`, from before the 1.0
versioning switch — while `cilium`, which uses an already-recognised
shape, carries `v1.5.0` through `v1.6.0-rc.2`. Neither has received a
tag from any 1.x release.

Measured against the committed tree, the dry-run selects **19 unique
repositories before this change and 29 after** — the eight digest-in-tag
images plus `keycloak-operator` and `kubeovn`, nothing lost.

### What the nightly does *not* recover

Four of the ten are not in the root `Makefile`'s `build` target: `kilo`,
`redis-operator`, `keycloak-operator` and `kubeovn`. Their baked refs
keep the committed GHCR host, so the `SRC_REGISTRY` filter still drops
them and their `:nightly` tags remain absent after this PR. Installs are
unaffected — those digests already resolve on GHCR. The nightly recovers
the six OCIR-built images; promotion recovers all ten.

## Why this was invisible

Latent since `nightly-mirror.sh` was added — the nightly had never
completed a full run, so its mirror step was never exercised end to end.

Both test suites were green throughout. `nightly-mirror_test.bats`'s
fixture covered the single-string, `{repository,tag,digest}` and
platformSource shapes — a layout only cilium actually uses — and omitted
the dominant one. `promote-retag_test.bats` asserted "at least one
cozystack-owned image is selected", which is satisfied by the shapes
that already worked and cannot detect an omission.

Two comments actively pointed the wrong way, and are corrected here. The
header presented its list as the shapes the build writes, which reads as
exhaustive; it now says the list is empirical, names what an audit found
outside it, and says to re-run that audit when a package is added. The
ownership-filter comment named `kilo`, `kube-ovn` and
`keycloak-operator` as third parties "not pushed to `$REGISTRY`" — all
three are pushed there, and that wrong comment is part of why the
missing tags went unnoticed.

Both fixtures are extended here. The promotion test names concrete
packages rather than asserting a count or a generic pattern, since
either would drift back to proving nothing.

## Type guards

Both digest-in-tag rules check `.tag` is a string before `test()`.
Without it a non-string tag aborts the whole expression (`cannot match
with !!int, can only match strings`), and since these invocations
swallow stderr and status, that abort would silently discard **every**
ref of this shape in the same file — reintroducing this very bug in any
file that happens to carry an unquoted `tag: 1.24` next to a cozystack
image. Rule 1 is immune because its `select(tag == "!!str")` already
precedes its `test()`.

`registry`, `repository` and `digest` are guarded too, against a
narrower failure. Those keys are concatenated rather than `test()`ed,
and yq coerces int, bool, null and seq on `+` — each merely yields a ref
the ownership filter discards. Only a `!!map` aborts, raising `!!str ()
cannot be added to a !!map`, which takes the file's whole expression
with it. So that guard is insurance against a nested-map typo in a
vendored chart, not against unquoted scalars. It is a `select()`, so an
offending map drops only itself and its neighbours still resolve.

No non-string value of either kind exists in the tree today, so both are
prevention rather than an active fix.

## Testing

- Both suites pass: `nightly-mirror_test.bats` 2/2,
`promote-retag_test.bats` 4/4.
- Every new assertion was mutation-tested — removing the rule under test
turns it red. Verified individually for the shape-3 registry join, the
shape-2 registry join, the `global.registry.address` rule, and the
`repository` type guard. Also verified that a rule matching the bare tag
instead of the repository (the subtle wrong fix) fails, which is why the
assertions check the full source ref rather than only the destination.
- `promote-retag.sh --dry-run` against the real committed tree: 19 → 29
unique repositories, no losses.
- An audit of every `@sha256:` digest under `packages/*/*/values.yaml`
against the selector output accounts for all 35: 30 are selected, and
the remaining five are third-party
(`ghcr.io/lexfrei/{kuberture,ouroboros}`, `docker.io/clastix/kubectl`,
`ghcr.io/kvaps/...`) or sit outside `$REGISTRY`
(`ghcr.io/cozystack/ingress-nginx-with-protobuf-exporter/*`).
- `shellcheck` clean on both scripts at default severity and `-S style`.

## Ordering note for reviewers

Worth landing before the next release candidate is cut, rather than
treating this as routine CI cleanup.

`promote-rc.yaml` builds the release branch from the rc's staging
branch, and `pull-requests-release.yaml` runs `./hack/promote-retag.sh`
from that merged tree — so the copy of the script that executes at
promotion is the one frozen into the rc tag's tree, not whatever is on
the default branch at promotion time. An rc cut before this merges will
carry the old script and silently skip those retags when promoted.

The nightly half is also what unblocks the full-suite e2e lane, which
cannot currently publish a working install closure to GHCR.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff: it touches only `hack/nightly-mirror.sh`, `hack/promote-retag.sh`
and their two `.bats` suites. No `packages/`, no chart, no
`values.schema.json`, no `ApplicationDefinition`, no CRD or RBAC. The
`cozystack/ccp` trigger covers moving or renaming files under `hack/`
and changing what a make target does; neither applies — the anchor files
it gates on (`hack/package.mk`, `hack/common-envs.mk`) are untouched and
no make target changes behaviour.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(ci): mirror and retag every cozystack-owned image, including those whose digest is embedded in the values `tag` and those whose registry host is stored outside `repository`. Ten images across eight packages were silently skipped: the nightly published references that 404 at install, and promoted releases did not carry the `:<version>` tags they claim.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved nightly mirroring and retag promotion to recognize
digest-pinned image references across more YAML value shapes, including
digests embedded in the `tag` field and optional registry wrappers.
* Expanded support for chart-global image definitions and safer parsing
to avoid silently skipped references.
* Strengthened selection logic to exclude third-party images from copy
plans.
* **Tests**
* Expanded dry-run coverage with additional fixture cases for split
maps, `tag`-embedded digests, numeric-tag variants, and more “owned”
image assertions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
v1.6.0-rc.4
2026-07-21 14:59:48 +05:00
myasnikovdaniil 1b777d8077 fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver (#3386)
## What this PR does

`quay.io/centos/centos:stream9` is a **rolling tag** — upstream deletes
the previously-pinned digest whenever it publishes a new `stream9`
build. The digest re-pinned in ce4b263ca (`sha256:a93e3316…`) has now
itself been deleted upstream, so every `kubevirt-csi-driver` image build
fails:

```
#3 ERROR: quay.io/centos/centos:stream9@sha256:a93e3316…: not found
ERROR: failed to solve: … failed to resolve source metadata … not found
make: *** [Makefile:43: image-kubevirt-csi-driver] Error 1
```

This breaks the **Build packages/apps/kubernetes** job on `main` and on
every open PR (e.g. observed on #3383).

Re-pins to the current `stream9` manifest-list digest
`sha256:3714c89f1903dac5a5e1eb6c02d84189ff6d962a0c18df01feba0a5406856926`,
resolved via `docker buildx imagetools inspect
quay.io/centos/centos:stream9`.

> Note: this is the second recurrence of the same class of breakage
(ce4b263ca fixed the first). Because `stream9` is a rolling tag, a
pinned digest will keep rotting; a follow-up could track it via renovate
or pin a date-stamped/immutable tag instead. Out of scope here — this
just unblocks CI.

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated the container build to use a refreshed CentOS Stream 9 base
image.
  * No end-user-facing functionality or behavior changed.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-21 13:47:53 +05:00
Myasnikov Daniil 3ca62266f8 fix(platform): qualify package names in the bundle guard messages
Address review feedback on the guards added alongside the isp-hosted
dependency-closure fix.

Spell every package the guards name in its fully-qualified form
(cozystack.cozystack-engine rather than cozystack-engine, and likewise for
cert-manager, prometheus-operator-crds, postgres-operator,
objectstorage-controller and cozystack-basics). The messages exist to be
acted on, and an operator who greps the emitted Package CRs for the name
they were given should find it.

Also correct the direction of the cross-reference in the bucket-application
comment: the velero comment it points at sits above the block, not below.

No behavioural change — the guards fire on the same conditions and the
tests match on the unchanged leading clause.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 13:36:50 +05:00
Myasnikov Daniil 0bfbff26ff fix(platform): restore the isp-hosted dependency closure
The isp-hosted preset shipped bundles.system.enabled: false with paas and
naas enabled. templates/bundles/system.yaml is gated end to end on that
flag, but it carries two separable things: a variant-selected data-plane
head (kubeovn/cilium/linstor on isp-full, a noop networking Package on
isp-hosted) and a variant-agnostic control-plane body — cozystack-engine,
the sole source of the application-definition-crd component, plus
cert-manager, prometheus-operator-crds, postgres-operator and
objectstorage-controller.

"Hosted" means the host already provides CNI, storage and virtualisation,
so only the data-plane head should be dropped. Disabling the whole bundle
also deleted the control-plane body, leaving an app catalog whose entire
dependency closure was absent: all 24 emitted Packages sat in
DependenciesNotReady, the *-rd HelmReleases failed with `no matches for
kind "ApplicationDefinition" in version "cozystack.io/v1alpha1"`, and the
cozystack-platform HelmRelease never went Ready. That the variant field
was still set to isp-hosted — and that system.yaml has an isp-hosted
branch which could never render — shows the disable was the mistake, not
the intent; iaas.yaml already refuses to render on this variant, encoding
the same "system bundle minus iaas" contract. The preset has never worked
since it was introduced in 2d022e38e.

Enable the system bundle for the preset. isp-hosted now emits 53 Packages
with every dependsOn satisfied, and installs 59 components against the 60
of the pre-refactor bundles/paas-hosted.yaml, with no LINSTOR, KubeVirt,
CAPI or Kamaji leakage.

Also move cozystack.bucket-application from the iaas bundle into the
system bundle. Its own dependsOn closure is networking,
objectstorage-controller and cozystack-engine, none of which are iaas, and
cozystack.backupstrategy-controller — a default package of the system
bundle — hard-depends on it. Any system-on/iaas-off configuration
therefore wedged backupstrategy-controller; both isp-full presets masked
this by enabling iaas.

Guard the class of bug rather than only these two instances. The existing
paas/naas/iaas guards check bundles.system.variant, which keeps its value
when the bundle is disabled, so they passed on exactly this
configuration. Make all three require bundles.system.enabled too and point
the operator at the supported topology, turning a silent multi-hour
DependenciesNotReady hang into a render-time failure.

Covered by two helm-unittest suites: one pinning the dependency closure
the catalog needs — including a case that loads values-isp-hosted.yaml
unmodified, so a regression in the preset itself is caught — and one
pinning the guards. The fix changes and the guards were mutation-tested by
reverting each in turn and confirming the suite fails.

This restores the dependency closure; it is not by itself sufficient for a
fully converged hosted install. Review surfaced further hosted-unaware
defaults that the dead bundle had been masking — notably apps/tenant
rendering cilium.io/v2 policies with no .Capabilities guard, and
extra/monitoring defaulting logsStorages to the LINSTOR-only "replicated"
StorageClass. Those are tracked separately. bundles.disabledPackages can
also still remove a dependency while its dependents stay emitted; the
guards do not cover that path.

Refs #3376

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 11:51:52 +05:00
Andrey KolkovandClaude Opus 4.8 9c3bae012a fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver
quay.io/centos/centos:stream9 is a rolling tag; upstream deletes the pinned
digest whenever it publishes a new stream9 build. The digest re-pinned in
ce4b263ca (sha256:a93e3316...) has now itself been deleted, so every
kubevirt-csi-driver image build fails with:

  quay.io/centos/centos:stream9@sha256:a93e3316...: not found
  failed to resolve source metadata ...
  make: *** [image-kubevirt-csi-driver] Error 1

breaking the "Build packages/apps/kubernetes" job on main and every PR.

Re-pin to the current stream9 manifest-list digest
(sha256:3714c89f1903dac5a5e1eb6c02d84189ff6d962a0c18df01feba0a5406856926),
resolved via `docker buildx imagetools inspect quay.io/centos/centos:stream9`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-21 10:40:17 +04:00
Myasnikov Daniil 292f8ec65a fix(ci): type-guard the concatenated keys in the ref selectors
Address non-blocking review feedback from lexfrei on
hack/nightly-mirror.sh:81 and hack/promote-retag.sh:84.

`registry`, `repository` and `digest` are joined into a ref by string
concatenation with no type guard, so a non-string value could abort the
whole per-file expression. Because collect_refs swallows both stderr and
status, that abort silently discards every co-located owned ref — the
same per-file blast radius the `tag` guard already prevents.

The mechanism differs from `tag`'s, though, and the narrower reading is
worth recording. These keys are never fed to test(), so the "cannot
match with !!int" abort does not apply to them: yq coerces int, bool,
null and seq on `+`, and each of those merely yields a ref the
ownership filter discards. Only a !!map actually aborts, raising
"!!str () cannot be added to a !!map". The guard is therefore narrow
insurance against a nested-map typo in a vendored chart, not protection
against unquoted scalars.

Implemented as select() rather than a test(), so an offending map drops
only itself and its neighbours in the same file still resolve.

The nightly fixture gains a file pairing a map-typed `repository` with
an owned ref, asserting the latter is still mirrored; removing the
guard turns it red. Selection over the committed tree is unchanged at
30 repositories.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 11:06:47 +05:00
Myasnikov Daniil acd88bdd00 fix(ci): collect image refs whose host is outside repository
Address review feedback from lexfrei and IvanHunters on
hack/promote-retag.sh:84 and hack/nightly-mirror.sh:81.

The shape-3 rule assumed the registry host lives inside `repository`.
Two cozystack-built images do not satisfy that and were emitted
host-less, so the ownership filter dropped them — the same silent
emit-then-drop this PR fixes for the other shapes:

  - keycloak-operator splits the host into a sibling `registry` key
    ({registry: ghcr.io, repository: cozystack/cozystack/keycloak-
    operator, tag: latest@sha256:...}), and its own Makefile writes
    exactly that layout.
  - kubeovn keeps the host in the document-level
    global.registry.address with a bare `repository: kubeovn`. It is
    built and pushed to $REGISTRY by the cozystack/kubeovn-chart
    wrapper, whose `make image` writes all three keys.

Neither has received a 1.x release tag: on GHCR keycloak-operator
carries only `latest`, and kubeovn's newest cozystack-versioned tag is
v1.14.11-v0.38.8, from before the 1.0 versioning switch — while cilium,
which uses an already-covered shape, has v1.5.0 through v1.6.0-rc.2.

Shapes 2 and 3 now rejoin an optional `registry` sibling, and a new
shape-4 rule handles the chart-global layout. An audit of every
@sha256: digest under packages/*/*/values.yaml against the selector
output confirms the remaining five misses are third-party or sit
outside $REGISTRY. Dry-run over the committed tree: 28 -> 30
repositories, exactly these two gained, nothing lost.

Also correct two comments that overstated coverage. The header no
longer presents its list as the shapes the build is limited to — the
list is empirical and charts are vendored — and the ownership-filter
comment no longer names kilo, kube-ovn and keycloak-operator as
third parties not pushed to $REGISTRY. All three are pushed there, and
that wrong comment is part of why the missing tags went unnoticed.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 11:04:21 +05:00
Andrey KolkovandClaude Fable 5 b897b53107 test(postgres): fix S3 CA secret name in backup roundtrip e2e
The backup roundtrip e2e copied the seaweedfs S3 CA from a Secret named
seaweedfs-system-ca-cert in tenant-root, a name captured off a dev
cluster. On current main the seaweedfs chart pins
fullnameOverride: seaweedfs, so the cert-manager CA Certificate's secret
is seaweedfs-ca-cert; the old name is absent and the first CI run of the
e2e failed at step 05a with 'secrets "seaweedfs-system-ca-cert" not
found'.

Default S3_CA_SECRET to seaweedfs-ca-cert and, when that Secret is
absent, auto-discover the seaweedfs CA Certificate (the seaweedfs-labelled
one with spec.isCA=true) and read its secretName, so a future
fullnameOverride change does not silently break the copy again. Fail with
a clear message when neither resolves. README and chainsaw comment
updated to match.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-20 22:31:27 +04:00
Myasnikov Daniil 770314b3cb fix(ci): retag images whose digest is embedded in the values tag
promote-retag.sh carries the same collect_refs blind spot the previous commit
fixes in nightly-mirror.sh, and nightly-mirror.sh's own comment asserted the
two were "identical to" each other -- which is how the gap stayed invisible
once one of them was fixed.

Measured against the committed tree, the dry-run selects 19 repositories
before this change and 27 after. The eight it was missing are exactly the set
that broke the nightly: kamaji, kilo, linstor-csi, piraeus-server,
linstor-gui, metallb-controller, metallb-speaker, redis-operator. Nothing is
lost from the previous selection.

The consequence differs from the nightly case and is milder. This retag is
within a single registry, so the digests already exist and every digest-pinned
install still resolves -- there is no dangling reference and no 404. What is
missing is the :<version> tag (and :latest under MOVE_LATEST=1) for those
eight images, so a promoted release does not carry the tags it claims to, and
anything resolving a cozystack component by version tag rather than by digest
does not find it. Nothing in the promotion detects this: the script reports
success having simply never seen those refs.

The existing test could not catch it. It asserts "at least one cozystack-owned
image is selected" and that nothing third-party leaks in -- both satisfied by
the shapes that already worked, which is why eight omissions sat under a green
suite. It now names concrete shape-3 packages, since a count or a generic
pattern would drift back to proving nothing. Verified the assertion fails
against the unfixed script.

Same type guard on `.tag` as the previous commit, for the same reason: without
it an unquoted numeric tag anywhere in a file silently discards every shape-3
ref in that file, and the swallowed stderr leaves no trace.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 19:07:04 +05:00
Myasnikov Daniil 06278ac18b fix(ci): mirror images whose digest is embedded in the values tag
The nightly's full-suite e2e failed at install: the e2e-image-prepuller
DaemonSet never became ready, because
ghcr.io/cozystack/cozystack/linstor-csi@sha256:2f314cb9... and
.../piraeus-server@sha256:39fdacec... both 404 on GHCR. They are not there
because nightly-mirror.sh never copied them.

collect_refs handled three values shapes, and the one most package Makefiles
actually write is not among them: a split map whose digest is embedded in the
tag, {repository, tag: <tag>@sha256:<digest>}, produced by a single yq call
setting .image.tag to "$(IMAGE_TAG)@$(digest)". It matches neither the
{repository, tag, digest} rule (no `digest` key) nor, usefully, the
single-string rule -- that one matches the bare tag value, which carries no
repository, so ref_repo() reduces "main@sha256:..." to the repository "main"
and the SRC_REGISTRY case drops it.

Being dropped there is silent, and the host rewrite at the end of the script
runs over every depth-2 values.yaml regardless of what was mirrored. So the
image was never copied AND its repository was still repointed at
DST_REGISTRY, leaving a reference to an image that was never pushed there --
exactly the dangling-ref failure the comment above that rewrite warns about,
reached by a path it did not anticipate. The 404 then surfaces at pull time,
one job later, as a DaemonSet rollout timeout that names no image.

This is not limited to linstor. Checking the :nightly tag on GHCR, the two
packages using a recognised shape (cilium, cozystack-operator) are present
and eight images across six packages using this shape are absent: kamaji,
kilo, linstor-csi, piraeus-server, linstor-gui, metallb-controller,
metallb-speaker, redis-operator. The prepuller only templates
kubeovn/linstor/cert-manager, so it surfaced two of the eight; the rest would
have failed later at HelmRelease install.

Latent since the script was added in 4777273db (2026-06-26) -- the nightly
never completed a full run before today, so the mirror step had never been
exercised end to end.

The new rule type-guards `.tag` before test(). Without it a non-string tag
aborts the whole expression ("cannot match with !!int"), and since the
invocation swallows stderr and status that abort would silently discard every
shape-3 ref in the SAME file -- reintroducing this very bug in any file that
happens to carry an unquoted `tag: 1.24` next to a cozystack image. Rule 1 is
immune because its select(tag == "!!str") already precedes its test().

The test fixture is why this passed green: _make_tree() covered the
single-string, {repository,tag,digest} and platformSource shapes, a layout
only cilium actually uses, and omitted the dominant one. It now builds that
shape too, co-locates a numeric third-party tag with a cozystack shape-3 ref
to pin the type guard, and asserts the planned copy carries the REPOSITORY
rather than the bare tag -- a rule matching the tag alone would plan a copy
from "main@sha256:..." and be dropped as non-SRC_REGISTRY, so asserting only
the destination would not catch the regression.

Verified each assertion fails against the unfixed script and passes with it,
and that a realistic baked linstor tree now plans both copies.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 19:06:40 +05:00
myasnikovdaniil f5227ee60f fix(seaweedfs): close the 4.31 rename fallout on the 1.5.x→1.6 upgrade path (#3339)
## What this PR does

Cozystack v1.5.0 bumped the vendored SeaweedFS chart 4.0.405 → 4.31.0,
which renamed every workload after the Helm release (`<name>-system-*`).
StatefulSet names are immutable, so upgrades through 1.5.x stood up a
second, empty set beside the running one instead of renaming. #3282
pinned `fullnameOverride: seaweedfs` so 1.6 adopts running workloads in
place — this PR closes the remaining holes on that adoption path, found
by driving a disposable 3-node cluster through a real v1.4.5 → v1.5.3 →
main upgrade with SeaweedFS tenants planted in every reachable state.

- **The naming guard moves into `packages/system/seaweedfs`** — the
render a platform upgrade actually re-renders: the `<name>-system`
HelmRelease pulls this chart from a platform-managed ExternalArtifact,
so the previous guard in `extra/seaweedfs` was never in the path.
Upgrading a fresh-1.5.x tenant therefore created an empty chart-named
set beside its live data and flipped the guard's own legacy-data signal.
`extra/` keeps a sibling copy for operator visibility; a bats parity
suite pins the two detection blocks byte-identical.
- **The guard refuses instead of guessing when both naming generations
exist.** Nothing durable distinguishes a duplicate that never served
from one that served and crashed: claim timestamps invert during the
recovery runbook's own re-bind, `readyReplicas: 0` is a snapshot, and
Helm birth order answers "which generation is original", not "is the
other one empty". Exactly one generation present is decidable, and the
render proceeds (or refuses as class S) on its own.
- **Cluster-scoped RBAC is named per namespace again.** 4.31 named four
cluster-scoped objects after the release — identical for every tenant —
so all tenants collided on one ClusterRole/ClusterRoleBinding and only
the last-reconciled tenant's COSI provisioner kept its RBAC. Names
return to `global.seaweedfs.serviceAccountName`; three of four are
byte-identical to pre-4.31 and adopt in place.
- **The `seaweedfs-db` hand-over runs for every instance name.**
Migration 43 compared the owning release against the literal
`seaweedfs-system`, so an instance named `foo` was skipped and its CNPG
Cluster — the filer metadata for every object in that tenant's S3 — was
pruned on the next reconcile, PVC included. The comparison now matches
the `-system` suffix (shared `lib/seaweedfs-db-adopt.sh`), and new
migration 53 re-runs the hand-over for clusters already past 43, before
anything re-renders.
- **`hack/seaweedfs-naming-audit.sh` +
`docs/operations/seaweedfs-431-rename-recovery.md`** — what the guard's
refusal points operators at: read-only classification (`L` / `S` /
`MIXED`, naming the candidate duplicate from relative PV vintage, never
a clock window) and the recovery procedures.

**Scope.** The supported SeaweedFS deployment is the tenant module,
which hardcodes the instance name
(`packages/apps/tenant/templates/seaweedfs.yaml`); a tenant only enables
or disables it. The runbook and its selectors are scoped to that name;
instances created directly against the API under other names are
classified by the audit but routed to escalation. Zone/pool keys of ~40
or more characters fall outside the guard's reconstruction — an accepted
limit, recorded in `_naming.tpl` and the runbook.

**Upgrade impact.** 1.4.x → 1.6: no manual steps — one generation,
adopted in place. 1.5.x → 1.6 with SeaweedFS: migration 53 protects
every `seaweedfs-db` first; then the upgrade **refuses** for any tenant
holding both naming generations until the operator resolves the
duplicate. This is deliberate — which generation holds the data is not
decidable from inside a render, and guessing wrong destroys it. Release
notes should present the refusal as expected behavior.

**Testing.** 55 chart unit tests across both packages (including new
MultiZone zone-component guard cases — the suite previously had no zone
shapes), 11 audit bats, 8 guard-parity bats, migration bats. Validated
end-to-end on a disposable 3-node cluster driven v1.4.5 → v1.5.3 → main
with five tenants covering: never-saw-4.31 (renders untouched),
fresh-1.5.x (refused as class S), wedged duplicate, split duplicate, and
a non-default-named instance (its database survives only with this fix).
The audit classifies all five correctly, and its two independent signals
— revision-1 birth scheme and relative PV vintage — agree on every MIXED
tenant.

Related: #3282 (fullnameOverride pin), #3335 (etcd adoption backup gate
— separate, also required for the 1.5.x→1.6 path).

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change

Walked the trigger map against the diff: no package added/renamed under
`packages/{apps,extra}/`, the `packages/core/platform/values.yaml`
change is only the migrations `targetVersion` bump (no
`spec.components.platform.values.*` key changes), no
variant/bundle/component changes, no asset renames, no
ApplicationDefinition semantic changes.

### Release note

```release-note
fix(seaweedfs): the 1.6 upgrade no longer renames a SeaweedFS instance away from its data. The naming guard now runs in the chart a platform upgrade actually re-renders and refuses when both pre- and post-4.31 naming generations exist; hack/seaweedfs-naming-audit.sh and docs/operations/seaweedfs-431-rename-recovery.md guide recovery, and the refusal is expected for tenants that passed through 1.5.x. Cluster-scoped COSI RBAC is named per namespace again (the 4.31 release-based names collided across tenants), and the seaweedfs-db hand-over runs for every instance name — previously an instance not named `seaweedfs` had its filer metadata database pruned on upgrade; new migration 53 repairs clusters that already ran the old hand-over.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added fail-closed upgrade protections for SeaweedFS naming migrations
(mixed/damaged classification), including safer behavior when cluster
visibility is limited.
* Added cluster-scoped RBAC uniqueness safeguards to prevent
cross-tenant name collisions during rendering.
* Improved SeaweedFS database adoption/repair migrations and
strengthened post-delete cleanup ownership checks.
* **Bug Fixes**
* Hardened SeaweedFS 4.31 rename recovery and PV rebind flow, including
reclaim policy preservation and long/non-default instance-name edge
cases.
* **Documentation**
* Expanded the SeaweedFS 4.31 rename-recovery runbook with clarified
auditing, verification, and escalation.
* **Tests**
* Added/expanded integration and Helm rendering tests for the above
scenarios and refusal/fail-closed behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
v1.6.0-rc.3
2026-07-20 17:34:50 +05:00
myasnikovdaniil 8020656725 fix(ci): resolve nightly packages digest without image-specific skopeo ops (#3332)
## What this PR does

The Nightly workflow has **never completed a run**. Every scheduled run
since `NIGHTLY_ENABLED` was set has failed in `mirror` at "Resolve
source commit and version", ~80s in, with:

```
level=fatal msg="unsupported image-specific operation on artifact with type \"application/vnd.cncf.flux.config.v1+json\""
```

Nothing has ever been published — `cozy-installer:nightly` and
`cozystack-packages:nightly` both 404 on GHCR, and `cozystack-nocloud`
does not exist.

This fixes that failure and the next two behind it.

### 1. The digest lookup could never have worked

`cozystack-packages` is a **Flux OCI artifact** (config mediaType
`application/vnd.cncf.flux.config.v1+json`), not a container image. A
plain `skopeo inspect` parses the config blob as an image config and
fatals on it, so `skopeo inspect --format '{{.Digest}}'` was never
viable against this ref — the `--raw` call on the very next line already
acknowledged this.

Replaced with `skopeo inspect --raw` + `skopeo manifest-digest`, which
hashes the manifest bytes — exactly what the registry addresses the
artifact by, with no image-specific parsing.

This also collapses two registry fetches into one: the digest and the
`org.opencontainers.image.revision` annotation now come from the **same
bytes**, which strengthens the race the original comment describes —
there is no longer any window between pinning the tag and reading it.

### 2. `flux pull artifact` rejects a non-existent output path (×2)

`flux pull artifact --output _baked` fails with `invalid output path
"_baked": no such file or directory`. `_baked` is neither tracked nor
created by anything. `pull-requests.yaml` does `rm -rf` + `mkdir -p`
first; `nightly.yaml` omitted it. Since the workflow always died at step
1, this was never reached.

Same bug for `_rw` in the e2e staging step. Both now create the
directory first.

## Verification

- Reproduced the original fatal against the live artifact with skopeo
1.22.2.
- Ran the fixed resolve step end-to-end against the real registry:
returns the digest the registry reports, and a `sha` that matches the
current `main` HEAD.
- Reproduced the `_baked` failure and confirmed the fix extracts the
tree.
- `actionlint` clean; `zizmor --offline` clean; `shellcheck` clean on
changed lines.

Also checked and **not** changed, each disproven rather than assumed:

- `skopeo copy --multi-arch all` on a Flux artifact — works; `copy` is
mediaType-agnostic.
- The `awk -F@ '/artifact successfully pushed/'` digest capture —
matches current flux CLI output, and is the same idiom
`packages/core/installer/Makefile` already uses in production.
- `nightly-mirror.sh`'s `inspect --format Digest` — all mirrored refs
there are genuine image manifests, so it is correct as written.
- `make -C packages/core/talos talos-nocloud` — exists and is CI-proven
via `tags.yaml`.

## Honest scope

This fixes the `mirror` job's blockers. It does **not** promise a green
nightly on the first run: every step after `mirror` is still unproven,
having never executed. The most likely next wall is `build-disk` —
`cozystack-nocloud` is a brand-new package, so the first `oras push`
creates it, and the follow-up `oras tag` can hit `403 permission_denied`
until the package is linked (the risk the workflow already documents
in-line). The first-ever full e2e run may well surface its own issues.

Reviewing that follow-on separately seems better than growing this PR.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Improved nightly artifact mirroring and staging reliability.
* Prevented artifact pull failures by ensuring required output
directories are created before use.
  * Enhanced retrieval of artifact version and source revision metadata.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-20 16:49:38 +05:00
myasnikovdaniil a15206502f ci(labeler): map cert-manager, external-secrets and metrics-server scopes (#3362)
## What this PR does

PRs titled with the `cert-manager`, `external-secrets` or
`metrics-server` scopes landed on `area/uncategorized` because the
labeler's scope map had no entry for them — all three surfaced at once
in the current port-collision series (#3359, #3360, #3361).
metrics-server feeds the monitoring pipeline, so it maps to
`area/monitoring`; cert-manager and external-secrets are platform
services, so they map to `area/platform`.

### Release note

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Chores**
  - Improved pull request labeling automation.
- Pull requests involving metrics, certificate management, and external
secrets are now categorized more accurately with the appropriate area
labels.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-20 16:38:51 +05:00
Myasnikov Daniil bd770b2288 fix(hack): make the seaweedfs naming audit POSIX sh
hack/seaweedfs-naming-audit.bats sources the audit script at top level,
and cozytest.sh sources the converted test file into its own /bin/sh --
dash 0.5.12 on the Ubuntu CI runner. dash has no `pipefail`, so
`set -uo pipefail` aborted the runner before the first @test and took
the whole file down with it:

    hack/cozytest.sh: 48: set: Illegal option -o pipefail

The bash shebang could not prevent this: `.` executes the file's
contents in the caller's shell, so the shebang is only a comment there.

The script was already POSIX apart from those two lines -- no arrays,
no [[, no local, no bashisms at all -- so the script becomes POSIX sh
rather than the runner becoming bash. That keeps the shell under test
and the shell at run time the same one, instead of testing under dash
while shipping under bash.

Dropping `pipefail` changes no behaviour: the script never sets -e and
never reads a pipeline's exit status, reporting failure as empty output
checked with [ -n ... ].

Verified under dash (ubuntu:24.04): 11/11 tests pass, and the script
runs clean against a live cluster, classifying five seaweedfs tenants
with the revision-1 scheme and the PV ages agreeing on every MIXED
case. shellcheck -s sh reports nothing.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:49 +05:00
Myasnikov Daniil 9b82d1bf36 test(seaweedfs): run the db-adopt migrations in the shell they ship with
Address review feedback from IvanHunters on
packages/core/platform/images/migrations/migrations/43:23:

The suite invoked the migrations as `sh "$MIG_DIR/<n>"`. On the CI runner
/bin/sh is dash, which has no `set -o pipefail`, so both scripts aborted
on their `set` line with "set: Illegal option -o pipefail" (exit 2)
before doing any work, and the job went red. Production was unaffected —
the migrations image is FROM alpine, so /bin/sh there is busybox ash —
which is exactly why checking the runtime image missed it.

Rather than change a shebang or drop pipefail, run the migrations the way
run-migrations.sh does: by path, inside the image's own pinned alpine
base, read out of the migrations Dockerfile so the interpreter under test
cannot drift from the one they ship on. pipefail is load-bearing here —
lib/cozystack-version.sh pipes the rendered manifest into `kubectl
apply`, so without it a failed render stamps the version from empty
input — and a shell that lacks it can prove nothing about that path.

run_migration() returns the container's status explicitly: cozytest.sh's
awk generator rewrites every bare `}` in column 0 into `return 0` + `}`,
so a helper that falls off its own end reports success no matter what it
ran, and every fail-closed assertion would have passed vacuously.

15/15 pass. Verified the interpreter under test is BusyBox v1.37.0 ash
with pipefail supported, and that a fail-open mutation in
lib/seaweedfs-db-adopt.sh turns the suite red.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:49 +05:00
Myasnikov Daniil 8355f3e36f docs(seaweedfs): correct the stale "D-wedged is adopted" comments
The naming guard now refuses every two-generation state, but two
comments in extra/seaweedfs still described a D-wedged duplicate (zero
ready replicas) as safe to adopt and rendering through. readyReplicas is
a snapshot, not proof the duplicate never served, so it cannot be told
apart from D-split and the guard refuses it too.

Align the state summary in the seaweedfs template comment and the test
header with the presence-only refusal behaviour. No classifier change --
comments only.

Address review feedback from coderabbitai on
packages/extra/seaweedfs/tests/fullname_override_test.yaml.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:49 +05:00
Myasnikov Daniil a099b7e086 fix(seaweedfs): fail the cleanup Job on a database-owner lookup error
The post-delete cleanup hook read Cluster/seaweedfs-db's Helm owner with
`|| true`, so a Forbidden, a timeout or any API error collapsed to an
empty owner, took the "nothing to reclaim" branch, and let the Job
complete -- permanently orphaning the keep-protected Cluster the hook
exists to reclaim, with nothing else that will ever collect it.

Only a genuine NotFound now counts as absence; every other error prints
the failure and exits non-zero so the Job retries. This matches the
fail-closed handling the delete path already had. Add a regression
assertion that the ownership lookup no longer swallows failures.

Address review feedback from coderabbitai on
packages/extra/seaweedfs/templates/hooks/cleanup.yaml.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:49 +05:00
Myasnikov Daniil b4f98af940 docs(seaweedfs): fail closed in the rename runbook's reclaim-policy handling
The 1.5.x->1.6 rename recovery runbook deletes each volume PVC after
switching its PV to Retain, but the switch was never gated on success.
With no `set -e`, a failed `kubectl patch ... Retain` left the PV at
Delete and the following `delete pvc` took the data with the claim.

Gate the delete on both the annotation record and the Retain patch
succeeding, aborting the re-bind loop otherwise, so no PVC is deleted
while its PV is unprotected. In the Step 5 restore loop, chain the
annotation removal after the restore patch with `&&` so a failed patch
keeps the saved policy for a retry.

Address review feedback from coderabbitai on
docs/operations/seaweedfs-431-rename-recovery.md.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:49 +05:00
Myasnikov Daniil b16a51e60e test(seaweedfs): pin the naming guard's MultiZone zone-component classification
MultiZone (and Simple-with-pools) tenants render no plain `volume` component —
extra/seaweedfs sets volume.enabled=false and the vendored chart emits one
component per zone key with suffix `volume-<key>`, truncated at
(62 - len(suffix)), which is SHORTER than the 56-char cut the guard's
renamedVolumePrefix replays. The final review pass flagged this divergence as a
guard bypass. For the supported instance the prefixes cannot diverge: the
tenant module hardcodes the name `seaweedfs`, the fullname is 16 chars, and
nothing truncates until a zone/pool key of ~40+ characters. Three new cases pin
that supported path — legacy-only zones adopt, release-named-only zones refuse
as class S, both generations refuse — since the suite previously had no zone
shapes at all and the zone objects are the ONLY evidence on a MultiZone tenant.

The divergence beyond that boundary (long instance names or absurd keys) is an
accepted, documented limit, not a fix: decided in the 1.6 triage, recorded in
_naming.tpl (both copies, kept in lockstep) and the runbook's Scope section.
Instance names are fixed by the tenant module, so absurd keys are the only
reachable trigger; revisit if instance naming is ever opened up.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:49 +05:00
Myasnikov Daniil 95fcc76902 docs(seaweedfs): scope the rename runbook to the supported instance name
The tenant module is the supported way to run SeaweedFS and it hardcodes the
instance name: packages/apps/tenant/templates/seaweedfs.yaml creates the
HelmRelease as `seaweedfs`, and a tenant can only enable or disable it. Every
shell selector in this runbook assumes that name, so say so up front, and route
instances created directly against the API under other names to escalation
instead of through loops whose name filters provably cannot see truncated
claims. The ~40-char zone/pool key limit of the guard's reconstruction is
called out as an accepted limit for default-named instances too.

Two content fixes ride along, both for findings the final review pass raised:

- The emptiness check told the operator to kubectl exec the candidate's own
  volume pod — which for a D-wedged duplicate can never start; that is the
  definition of the class, so the check was unexecutable exactly where it
  matters most and every D-wedged tenant funneled to escalation. Mount each
  bound candidate claim read-only in a scratch pod instead (a Pending claim
  has no PV and is empty by construction). The weed shell cross-check stays
  on the authoritative master, which is running.

- Step 1's description of the audit now matches the audit itself: direction
  is decided by the relative strictly-newer PV rule (the same precondition
  Step 2a enforces), and an interrupted re-bind reads as overlapping vintages
  with no candidate — not by a clock window anchored on first_deployed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:49 +05:00
Myasnikov Daniil 24bcb0bb69 fix(seaweedfs): decide MIXED direction by relative PV vintage, not a clock window
The audit named the candidate duplicate by measuring each generation's oldest
PV against the release's first_deployed, with an absolute 120-second window
for the mid-rebind case. The shipped StorageClasses are WaitForFirstConsumer,
so PVs are provisioned at pod-SCHEDULE time — on a cold cluster minutes after
first_deployed — and a tenant interrupted mid-way through Step 2's re-bind
(both generations sitting on original-vintage PVs) fell out of the window and
into a branch decided by which volume index happened to provision first. The
advice printed there routes the operator to a step that deletes the claims
Step 2 had not yet re-bound.

Replace the window with the relative rule runbook Step 2a already enforces: a
generation is the candidate duplicate only when EVERY one of its bound PVs is
strictly newer than every bound PV of the other generation. Overlapping or
tying vintages name no candidate — finish Step 2 or escalate. first_deployed
is still printed, as context only.

The direction decision is now a pure function (classify_mixed_direction) so
the bats suite drives it directly: clean duplicate both ways, interrupted
re-bind, tie, and a no-clock case pinning the WaitForFirstConsumer regression.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:49 +05:00
Myasnikov Daniil 601f164c5f fix(seaweedfs): stop the naming guard guessing which volume generation holds the data
The guard told a class-S tenant from a D tenant by comparing PVC creationTimestamps,
on the premise that "PVCs are never recreated in place, so the older generation is
where the data was born". The premise is false, and this runbook is the counter-
example: Step 2's re-bind DELETES each release-named claim and recreates it under
the chart name against the same PV.

So a tenant interrupted part-way through Step 2 — after volume-0, before volume-1 —
has data1-seaweedfs-volume-0 created SECONDS ago holding real data, beside
data1-seaweedfs-system-volume-1 with the original timestamp also holding real data.
That is the exact inversion of the rule. Reproduced against the old template:

  keeps its data on the release-named volume PVCs (created 2026-06-20T07:01:59Z,
  OLDER than the chart-named data1-seaweedfs-volume-* claims created
  2026-07-17T09:30:00Z) ... Remove the empty chart-named set ... (Step 2a, then Step 2)

Step 2a then runs `grep -E '^data1-seaweedfs-volume' | xargs kubectl delete pvc` —
deleting the claim Step 2 had just re-bound — and Step 2 had already restored the
PV's original reclaim policy, so on any Delete-policy StorageClass the volume and
the bytes go with it. The guard's own recovery path was a data-loss instruction.

Two more holes in the same classifier:

- The S-damaged branch was gated on $systemPVC, which requires the PVC name to
  contain "seaweedfs". seaweedfs.componentName truncates the fullname to 56 chars
  before appending -volume, so an instance name >= ~40 chars drops the chart name
  off the PVC name — the very case fullname_override_named_test.yaml exists for.
  Probed with `archive-of-quarterly-financial-statements-x1`: damaged + renamed set
  Ready fell through to the D-split branch ("is running two live SeaweedFS sets ...
  Stop the split"), sending the operator to quiesce the set holding ALL the data —
  the exact misclassification the parent commit exists to remove. Damaged + not
  Ready fired no branch at all and rendered through unguarded.
- readyReplicas == 0 was read as proof the duplicate never served. It is a
  snapshot: a duplicate that crashed, was scaled down, or lost readiness AFTER
  serving writes reads identically, and adoption strands whatever it wrote.

There is no sound signal here. Claim timestamps are mutable, StatefulSets are
recreated by the adoption hook, and liveness is a snapshot — nothing durable in the
object graph distinguishes "empty set a bad upgrade created" from "the data,
mid-re-bind". Rather than add a fourth heuristic, the guard now decides only what
is decidable:

  legacy generation only (or nothing) -> render; adoption is correct
  release-named generation only       -> refuse, class S (Step 2)
  BOTH generations                    -> refuse; an operator classifies

Both generation flags now OR the PVC and StatefulSet evidence, so the truncated
long-name tenant is seen via its label-matched StatefulSet.

The cost is bounded and lands only where it should. A duplicate exists only on a
tenant that passed through the broken 4.31 rename in 1.5.x — already damaged,
already needing an operator. A 1.4.x tenant upgrading straight to 1.6 never
renames, so it has one generation and is untouched. Confirmed on the stand:
tenant-l (1.4.5-era, never upgraded) is legacy-only and dry-runs clean; the four
both-generation tenants all already need an operator. What is lost is D-wedged
auto-adoption, which was resting on the readyReplicas assumption above.

Classification moves to the runbook, where a human has signals a template does not:
Step 1 now reads the ages of the PVs (durable — they survive the re-bind) via
.spec.volumeName instead of the claims, reports MIXED instead of picking, and walks
the operator through interrupted-Step-2 first, then strict age separation, then
verifying emptiness directly, then escalating rather than guessing. Step 2 stashes
the PV's original reclaim policy ON THE PV and leaves it Retain until Step 5
verifies the tenant, so a mis-step during recovery is survivable. Step 2a grows a
precondition that refuses unless every chart-named claim is bound to a strictly
newer PV.

Tests: 7 of 14 in the enforcing suite fail against the old template, including the
interrupted-Step-2 tenant and both truncated-long-name cases. hack/seaweedfs-guard-
parity.bats pins the two copies (system/ enforcing, extra/ sibling) against silent
drift — they cannot share a library, neither chart depends on cozy-lib, and "keep
these in sync" as a comment is the failure class this whole branch is about; it
also pins that creationTimestamp and readyReplicas never come back.

`make test` in both charts now runs helm template --is-upgrade and greps for the
refusal. The fail-closed canary — the headline safety property — had ZERO automated
coverage: guard_fail_closed_test.yaml passes with AND without the fix, because
helm-unittest 1.0.3 ignores release.isUpgrade. Verified the new target fails when
the canary is removed.

Review round 3 — the refusal was right; the justification was false and the
procedure that replaced the chart's judgement was broken on both axes.

THE COMMENT WAS WRONG. It said "nothing durable in the object graph distinguishes
the empty generation from the data mid-re-bind". A durable signal does exist, and
the reviewers were right to call that out: the Helm release history. Verified on
the stand — sh.helm.release.v1.<name>-system.v1 records the scheme the tenant was
BORN under (tenant-root/l/dsplit/named rev 1 render seaweedfs-master => pre-4.31;
tenant-fresh renders seaweedfs-system-master => 1.5.x), and info.first_deployed is
on every retained revision, so anchoring PV creationTimestamps against it survives
history pruning and the Step 2 re-bind (tenant-root: chart-named +10s vs
release-named +2084s; tenant-fresh: +5225s vs +6s).

BUT IT DOES NOT UNLOCK AUTO-ADOPTION, and that is the substantive disagreement
with the round-3 direction. Birth order answers "which generation is ORIGINAL".
Adoption needs "is the OTHER one EMPTY". Those come apart exactly where it
matters: D-wedged (duplicate never scheduled, safe to adopt) and D-split
(duplicate served writes, adopting strands them) are BOTH born pre-4.31 and are
therefore identical on every durable signal. Verified: tenant-root and
tenant-dsplit have the same rev-1 scheme and the same first_deployed deltas, yet
one duplicate served and the other did not. The only thing that ever separated
them was readyReplicas, which is a snapshot and not evidence. So the render keeps
refusing on presence, and the comment now says why — naming the signal, so nobody
re-derives the broken one.

The signal IS used where it belongs, in the audit.

- naming-guard.yaml / seaweedfs.yaml: match the release-named generation by
  RECONSTRUCTING the name 4.31 gives this release's volume component
  (templates/_naming.tpl, replaying seaweedfs.fullname + componentName), tested
  FIRST because the prefixes nest. Fixes a blocker: an instance legitimately named
  `seaweedfs-volume` renders seaweedfs-volume-system-volume and
  data1-seaweedfs-volume-system-volume-0, which BOTH satisfy the chart-named
  prefixes — a prefix-only guard read its live storage as legacy and would render
  onto the empty chart-named claims, stranding the real generation. The parity
  bats passed straight through it, because parity pins sameness, not correctness.
  Reconstruction also removes the name-match blind spot at source: a truncated long
  instance name now matches by PVC, not only by StatefulSet.

- hack/seaweedfs-naming-audit.sh + .bats: the classifier is a tested script now.
  Two revisions of it shipped as an untested snippet inside the runbook and both
  were wrong in ways that routed a live tenant into the step that strands its data:
  a selector `^data1-(.*seaweedfs.*)-volume` that also matched the chart-named
  claims, so the release-named range spanned BOTH generations, every tenant read as
  "overlapping / mid-rebind", and a genuine S-damaged tenant was routed AWAY from
  Step 2a into Step 2 — which deletes its data claim, fails to recreate it, then
  deletes its StatefulSets; and a `grep seaweedfs` name match that reported
  "L — nothing to do" for a long-named tenant the chart refuses. Both are covered.
  The audit reports L/S/MIXED matching the guard, names the original generation
  from both durable signals, detects MID-REBIND (both generations at
  first_deployed), and states plainly that "original" is not "empty" and that
  emptiness must be established from the duplicate's volume files before deleting.
  A bats case renders _naming.tpl through helm and compares it to the script's
  reconstruction, so the two implementations cannot drift.

- Runbook: Step 1 calls the script. Steps 3/4 were circular — Step 3 ended in an
  upgrade the presence-gated guard refuses, and Step 4 told the operator to clear
  duplicates AFTER an upgrade that cannot run until they are gone. Step 3 now
  removes the duplicate BEFORE the upgrade and re-runs the audit to prove it; Step 4
  starts from "every tenant reads L or S". D-wedged no longer claims to need
  nothing. Step 3's "enumerate fids that resolve only on the duplicate" was not an
  executable procedure and is now an explicit escalation: two masters allocate
  volume IDs from independent sequences into one shared seaweedfs-db, so there is
  no supported tool that reconciles two volume-ID spaces against one metadata
  store. Step 2's re-bind restores the cleanup hook's labels on the recreated
  claims, without which a recovered tenant leaks its volumes on a later delete.

Fixture corrected: data1-...-x1-system-volume-0 is not a name the chart renders;
componentName gives ...-x1-system-seaw-volume-0.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:48 +05:00
Myasnikov Daniil 47e24f01d1 fix(platform): hand seaweedfs-db over for every instance name, not just the default
Migration 43 performs the v1.5.0 db-split hand-over: it re-owns Cluster/seaweedfs-db
from <name>-system to the new <name>-db release and stamps
helm.sh/resource-policy: keep, so the <name>-system upgrade — whose post-split
chart no longer renders the Cluster — does not prune it as a removed resource.

It compared the owning release name against the literal "seaweedfs-system".
SeaweedFS is a user-creatable kind, so an instance named `foo` is owned by
`foo-system` and was silently skipped: no re-own, no keep. The <name>-system
upgrade then deleted the Cluster, CNPG took its PVC with it, and the filer
metadata — the index for every object in that tenant's S3 — was destroyed. Data
loss, not an outage: the volume PVCs still hold the bytes, nothing can find them.

Verified live on the upgrade stand, which is a natural control group. All five
tenants sit in the SAME failed-upgrade loop (<name>-system last-deployed is
rev 1, a pre-split revision whose manifest still contains the Cluster, so every
retry recomputes the same deletion):

  tenant-root/dsplit/l   instance `seaweedfs`  owner=seaweedfs-db  keep   Cluster alive
  tenant-named           instance `foo`        no Cluster at all          Cluster GONE

The only difference is that migration 43 fired for the default-named instances
and skipped `foo`. That also answers whether `keep` is sufficient: three tenants
survive the identical prune loop with it.

Correction to the reported mechanism: this is the 1.4->1.5 db split (PR #2601,
v1.5.0), not a 1.5->1.6 change — v1.5.3 and main are identical on every
db-split file. And the split does protect the object; it just only protected the
instances someone happened to name `seaweedfs`.

- lib/seaweedfs-db-adopt.sh: shared hand-over matching the `-system` SUFFIX, so
  `foo-system` -> `foo-db` exactly as `seaweedfs-system` -> `seaweedfs-db`.
  Idempotent, and refuses a release named literally `-system` rather than
  annotating an owner of `-db` that no release would claim.
- Migration 43 sources it: covers clusters upgrading from before 43 (1.4.x).
- Migration 53 (new) re-runs it: migrations never re-run, so any cluster already
  at >= 44 ran the hardcoded version and is still exposed. As a pre-upgrade hook
  it lands before <name>-system re-renders, closing the window on THIS upgrade.
  This is the one that matters for 1.5.x -> 1.6.
- targetVersion 53 -> 54.

Tests fail on unfixed code: the non-default-name case produces no ANNOTATE at
all against the hardcoded 43, while the default-name case passes — which is
precisely the bug's shape. They drive the real migration scripts against a fake
kubectl, following hack/migration-50-etcd-adopt.bats.

LIMIT, stated plainly: a Cluster that is already deleted cannot be recovered by
either migration — there is nothing left to re-annotate and the PVC went with
it. Such a tenant needs seaweedfs-db restored from a backup. Runbook Step 0 adds
the ownership audit that distinguishes at-risk from already-lost, and records
that <name>-db can report Ready while its Cluster is gone (it rendered fine; a
later <name>-system prune removed it and Flux has not re-checked) — so the
audit trusts `kubectl get cluster`, not the HelmRelease status.

Review round 2 — two ways this still lost the database:

- OWNERSHIP IS NOT SAFETY. The helper skipped any Cluster already owned by
  <name>-db, treating ownership as proof the hand-over was done. It is not: where
  the hand-over was skipped, <name>-system prunes the Cluster and <name>-db
  RECREATES it under its own ownership with NO keep, while <name>-system's prune
  baseline still lists it — so the next reconcile deletes it again. That is the
  observed delete/recreate loop, and the skip walked straight past it. Verified on
  the stand: tenant-l and tenant-root are seaweedfs-db-owned AND their
  <name>-system deployed revision (rev 1) still contains the Cluster, so only keep
  saves them; tenant-fresh is seaweedfs-db-owned without keep but was installed
  after the split, so its baseline never had one. Telling those apart needs the
  release's deployed manifest, which this script cannot read cheaply or reliably.
  The costs are asymmetric — a needless keep leaves an orphan on delete (now
  reclaimed by the cleanup hook), a missing one loses the database — so keep is
  stamped on every Cluster owned by either side of the split.

- FAIL OPEN, THEN STAMP. `set -euo pipefail` does not abort on a failing command
  substitution in a `for` word-list: the loop ran zero times, the script continued,
  and stamp_cozystack_version ran anyway. Migrations never re-run, so one throttle,
  RBAC hiccup, or not-yet-established CNPG apiservice behind the pre-upgrade hook
  permanently left every at-risk tenant exposed with no later migration to catch
  them. Reproduced: against the old helper a failing fleet scan still logged
  STAMP 54. Now every kubectl failure is fatal EXCEPT the two that genuinely mean
  "nothing to do" — the CNPG resource type not being served (the fail-open that IS
  load-bearing: a cluster without CNPG must still upgrade) and a Cluster vanishing
  between scan and read. An unreadable annotation is no longer indistinguishable
  from an absent one, and the unowned case now warns instead of passing silently.

- ORPHAN ON DELETE. `keep` is permanent (removal deferred to the 1.7 batch
  migrations) and also survives the <name>-db release's own uninstall, so
  `kubectl delete seaweedfs <name>` left the CNPG Cluster, its Postgres pods and
  its PVCs behind — the cleanup hook selects
  app.kubernetes.io/instance=<name>-system, which the db PVCs
  (cnpg.io/cluster=seaweedfs-db) do not carry. Migration 53 widened that from
  pre-1.5 default-named tenants to every instance name, so the hook now deletes
  the Cluster explicitly; its PVCs carry ownerReferences to it and follow.

- Tests assert the stamped VERSION (44 / 54), not a bare "STAMP": a wrong number
  would loop run-migrations.sh forever and the old assertion could not tell.

Review round 3 — the reclaim introduced a data-loss path of its own:

- DELETING ONE APP DELETED ANOTHER APP'S DATABASE. The db chart hardcodes the
  Cluster name `seaweedfs-db` for every instance, and the reclaim was scoped by
  that name in the namespace rather than by what the release owns.
  templates/seaweedfs-db.yaml gates the <name>-db HelmRelease on
  `topology != "Client"`; the cleanup hook had no such gate. Verified by render:
  `helm template s3-remote . --set topology=Client` emits ZERO s3-remote-db
  HelmReleases yet still emitted the Role granting delete on clusters/seaweedfs-db
  AND `kubectl delete cluster.postgresql.cnpg.io -n tenant-root seaweedfs-db`. So
  in a namespace running a server instance plus a Client, deleting the CLIENT
  destroyed the SERVER's filer metadata — CNPG takes the PVC via ownerReferences —
  for an app the operator never touched. Reachable the same way via a second Simple
  instance, whose <name>-db loses Helm's ownership check but which still installs
  and still carries the hook. main's hook was instance-scoped; this was new, and
  mine.
  Now gated exactly as seaweedfs-db.yaml is, and the Job re-checks
  meta.helm.sh/release-name == <release>-db at run time before deleting. The gate
  stops the permission being granted where it can never be legitimate; the runtime
  check stops the delete where it can.

- `|| echo` turned any API/authz/admission failure into exit zero, so backoffLimit
  never retried and hook-delete-policy removed the only cleanup Job while the
  keep-protected Cluster stayed orphaned. A failed reclaim now fails the Job.

- TEST SHELL. The migrations run under /bin/sh = busybox ash (the image is FROM
  alpine, run-migrations.sh is #!/bin/sh), but the tests drove them with bash — so
  the fail-closed guarantees were asserted in a shell that never runs them, which
  matters more than usual given the bug they cover was a fail-open. The scripts are
  now invoked via `sh` and the fake kubectl is POSIX sh. Verified directly against
  the production base image (busybox 1.37.0 / alpine:3.24): `set -o pipefail` is
  supported, every script passes `ash -n`, and the failing-fleet-scan path exits
  non-zero without stamping under real ash. The recipe is in the bats header.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:48 +05:00
Myasnikov Daniil ae93356ed1 fix(seaweedfs): name cluster-scoped RBAC per namespace, not after the pinned fullname
ClusterRoles and ClusterRoleBindings are cluster-scoped: one object per name,
cluster-wide. This chart installs once per tenant, so every cluster-scoped name
it renders must be unique per install. Four of them were not.

Upstream 4.31 switched them from global.serviceAccountName to seaweedfs.fullname.
That is fatal in combination with the #3282 pin: fullnameOverride: seaweedfs is
what adopts the running NAMESPACED workloads across the rename, so fullname is by
construction the SAME string in every tenant. Every SeaweedFS instance in the
fleet therefore renders seaweedfs-objectstorage-provisioner, seaweedfs-rw-cr and
their bindings — one object, five claimants.

NOT a #3282 regression. Rendering v1.4.5, v1.5.3 and main for the same tenants:

  v1.4.5  tenant-root-seaweedfs-*      tenant-named-seaweedfs-*     unique
  v1.5.3  seaweedfs-system-*           foo-system-seaweedfs-*       COLLIDES
          (identical for every default-named instance; a non-default
           instance name was the only thing that escaped)
  main    seaweedfs-*                  seaweedfs-*                  COLLIDES
          (nothing escapes: the pin removes the last distinguishing input)

So the collision shipped in 1.5.0 with the 4.31 bump; #3282 widens it from
"every default-named instance" to "every instance". Observed live on the upgrade
stand: one ClusterRole/seaweedfs-objectstorage-provisioner annotated to a single
owning tenant, four other tenants' COSI provisioners left with no RBAC, and
tenant-dsplit's helm history recording
  Upgrade failed: original object ClusterRole "seaweedfs-objectstorage-provisioner" not found

The binding is the sharp end: its subject names a ServiceAccount in the release
namespace, so the single live ClusterRoleBinding grants cluster-wide secrets RBAC
to whichever tenant reconciled last, and silently revokes it from the rest.

Fix: put the four names back on global.seaweedfs.serviceAccountName, which
extra/seaweedfs already sets to <namespace>-seaweedfs and which
cosi-provisioner-sa-name.patch already uses for the ServiceAccount and the
binding's subject — the cluster-scoped names were simply left behind. Three of
the four come out byte-identical to pre-4.31, so a 1.4.x tenant adopts them in
place; the fourth (rw-crb) replaces upstream's username-shaped
system:serviceaccount:<sa>:default and is pruned from the release's own manifest.
The namespaced pin is untouched: workloads still render seaweedfs-master/-filer/
-volume, so #3282's adoption is intact.

Uniqueness now rests on a value, so assert it rather than trust it:
templates/cluster-scoped-rbac-guard.yaml refuses to render when the name would
not carry the release namespace. It sits behind the same namespace canary as the
naming guard, because system/seaweedfs's own values.yaml ships a placeholder that
only extra/seaweedfs overrides — a client-side render must not trip on it.

Tests fail on unfixed code: cluster_scoped_names_test asserts two tenants get two
names, and unpatched both render seaweedfs-objectstorage-provisioner (5 failed,
0 passed). They test the vendored chart templates directly, which is where the
rename happens.

The guard includes seaweedfs.compat before reading. extra/seaweedfs sets the OLD
flat key global.serviceAccountName, and the shim is what folds it into the
canonical global.seaweedfs.serviceAccountName; without it the guard compares
against this chart's placeholder default and refuses every real upgrade. Caught
by helm upgrade --dry-run=server against the stand, not by unit tests — the
mocks never carry the flat key.

Verified live with --dry-run=server on the stand: tenant-l (installed pre-4.31,
never upgraded) already owns tenant-l-seaweedfs-objectstorage-provisioner and
tenant-l-seaweedfs-rw-cr, and the fixed chart renders exactly those — adopted in
place, no ownership error. tenant-root renders tenant-root-seaweedfs-* and also
dry-runs clean. Leftovers on the release-based names that a 1.5.x tenant passed
through are inert once every tenant is upgraded (no release renders them any
more) but are not always pruned, since more than one release claimed them; the
runbook adds the sweep.

Review round 2:
- The uniqueness guard used `contains .Release.Namespace $sa`, a substring test
  that accepts namespace `tenant-a` for service account `tenant-ab-seaweedfs` —
  a different namespace's. Anchored to hasPrefix "<namespace>-". It still asserts
  the VALUE the names are built from, not the rendered names, so it cannot catch
  the chart being reverted to name them after the pinned fullname;
  cluster_scoped_names_test.yaml covers that axis and is what actually bites.
- Added the case that would have caught the compat bug: every test set the
  CANONICAL global.seaweedfs.serviceAccountName, while production sets the OLD
  FLAT global.serviceAccountName that only seaweedfs.compat folds in. Deleting
  the include left the suite passing 5/5 while every real upgrade refused. The
  new case sets the flat key; with the include removed it now fails with the
  production error.
- Runbook sweep missed the fourth pre-4.31 name: 4.31 renamed the master-rw
  binding from upstream's username-shaped system:serviceaccount:<sa>:default to
  <sa>-rw-crb, and the old one carries neither suffix nor a tenant prefix, so the
  grep could not see it (live on the stand as
  system:serviceaccount:tenant-l-seaweedfs:default). Also documents the rolling-
  upgrade window: Helm prunes by name without checking ownership, so the first
  tenant onto the fixed chart deletes shared objects a not-yet-upgraded tenant is
  still bound through — COSI provisioning 403s until they reconcile, S3 traffic
  unaffected.

Review round 3:
- hasPrefix "<namespace>-" was still too loose. It accepts `tenant-a-b-seaweedfs`
  for namespace `tenant-a` — which is ALSO the natural value for namespace
  `tenant-a-b`, so two namespaces would accept one string and collide on the very
  object this guard exists to keep unique. extra/seaweedfs sets exactly
  `<namespace>-seaweedfs`, so assert equality with it and nothing looser.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:11 +05:00