mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 12:43:32 +09:00
f83db514b0eb88cefc2923d17aad59daccb4e358
7777
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
aef9e201c7 |
fix(ci): make workflow cancellation effective (#3401)
## What this PR does - allows superseded PR workflow runs to cancel `finalize` and `E2E Tests` instead of continuing because of job-level `always()` - makes `update-website-docs` cancellation-responsive while preserving its behavior when changelog generation fails or is skipped - retains step-level `always()` conditions for reports and sandbox cleanup ### Screenshots Not applicable. ### Release note ```release-note fix(ci): ensure superseded and manually canceled workflows stop long-running jobs ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Prevented release and validation-related jobs from running when a workflow is cancelled. * Updated pull request check, end-to-end testing, and stable documentation update conditions to avoid executing after cancellation. * Kept existing success-gating so follow-up tasks still require prerequisite steps to complete successfully. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
8ad25b970f |
test(ci): pin digests exactly and generalize the first-party markers
Round-3 review, both reviewers again converging on the same defects. The file-level marker check was keyed to the exact ref spellings present in the tree rather than to the join semantics _collect_yaml_shapes implements, and three realistic first-party refs evaded it: `registry: ghcr.io/cozystack/cozystack` + `repository: <name>` (kubeovn's layout under the key name keycloak-operator and ingress-nginx use, which the old marker missed because it required a trailing slash a complete-value host does not have), a host+org split with precedent at fluxcd's values.yaml, and a single-quoted `repository:` where only double quotes were admitted. All three carried a version-free latest@sha256: tag -- exactly the profile the check exists for -- and passed the whole suite. The markers now follow the collector's join semantics; this matches 43 files and needs no new allowlist entries. The residual limit (a host split at an arbitrary point inside cozystack/cozystack) is stated rather than implied, along with why widening further is not worth it: a bare cozystack/cozystack marker matches 65 files and would need 23 allowlist entries for Makefiles and test fixtures. More seriously, making the completeness oracle share shape logic introduced a regression the previous independent oracle would have caught: corrupting the last hex digit of every emitted digest left the rewrite, retag and mirror suites entirely green, because both sides of the comparison move together and the per-shape test matched only a repository substring. The digest is the only part deciding which bytes get retagged, so the per-shape test now asserts the exact canonical repo@sha256:digest for every shape. Declared extras were also narrowed by the refactor: parsing alone means a ref inside a block scalar yields the enclosing block (which the ownership filter then discards) and an extra that stops parsing after `make update` yields nothing at all. Both are silent skips, in the one storage shape defined as a manifest vendored verbatim from upstream. Extras get the textual scrape again alongside parsing, with the callers' dedup absorbing the overlap. Finally, enumerating a path only proves the path is visited: a declared extra whose ref the parser cannot reach stayed enumerated but uncollected while both completeness checks passed. Added a test asserting a declared extra yields its ref, using an unparseable Helm template with a single-quoted split-host ref -- the shape that defeated both guards. Also corrected the claim that the per-shape test compensates for the shared oracle: it catches a shape being DELETED, not narrowed. A shape losing its registry rejoin is caught by promote-retag and nightly-mirror, which assert named packages, and the comment now says so. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
f3a0f2d323 |
test(e2e): avoid head -n 0 in prevlog_cap for BSD portability
A zero container cap reached head -n 0, which GNU head treats as an empty
result but BSD/macOS head rejects ("illegal line count -- 0"), aborting a
local run. Special-case the zero cap to consume stdin instead, so the
helper behaves the same on both coreutils flavours.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
|
648eb42cae |
test(ci): make the image-ref guards see split-host and unparseable files
Both round-2 reviewers independently landed on the same two test-side
holes, and both are the guard being blind in exactly the way the bug it
guards was.
The completeness oracle derived its expected set from a grep for a
contiguous ghcr.io/cozystack/cozystack/...@sha256: string, so it could
not see any split-host shape: cilium, keycloak-operator, kubeovn, the OCI
artifact and seven shape-3 refs, 12 of 43 first-party refs in all. That
includes keycloak-operator, which the test's own rationale names as the
motivating profile. The fixture-discovery grep in the same file had
already been widened for this exact reason; the fix was never carried to
the oracle.
The oracle now applies the same shape knowledge to an independently
discovered file list, which is the axis that actually drifts -- what each
consumer disagreed about historically is which FILES it visits, not how a
ref is spelled inside one. To keep that sharing safe, shape coverage is
now pinned by its own test, so deleting a shape still fails something
even though both sides of the comparison would move together.
Sharing required splitting collect_refs_from_file out of
collect_image_refs. Behaviour is unchanged: same 67 refs collected before
and after, multus and the packages artifact both still present.
A YAML oracle cannot cover everything, though. A reviewer planted a
split-host ref in a Helm template, which yq cannot parse at all, and it
survived every test. A file-level marker check now backstops that: any
file carrying a first-party marker must be enumerated or explicitly
allowlisted. Two files are allowlisted today, each with its reason -- the
talos-csr-signer Dockerfile placeholder, and the documented capi gzip
gap. This also closes the secondary weakness that comparing on repo@digest
masks an unenumerated occurrence when the same image sits at an
enumerated path, which is live for that capi file.
Second hole: the postcondition's key-position branch had no coverage.
Replacing the whole pattern with the naive "${RC}@sha256:" -- the
narrowing rejected because cilium keeps tag: and digest: under separate
keys -- left all 22 tests green, so the branch that makes the split shape
visible was asserted by nothing. Added a cilium-shaped fixture at an
unenumerated path.
Both previously-surviving mutations are now caught. The oracle was also
pre-filtered by registry marker before parsing, cutting it from 32s to 5s.
Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
76f7267ff2 |
test(e2e): describe --previous as the preceding instance, not the first
kubectl logs --previous returns the immediately preceding terminated container instance, not the original first one — after multiple restarts the first instance is already gone. Reword the helper, the BATS and Chainsaw failure handlers, and the diagnostics doc to match the actual --previous semantics. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
1ff83f5c06 |
test(e2e): fix false claim that a job timeout preserves the report
The catch-timeout comment justified the raised ceiling by asserting that if: always() upload survives a job timeout. It does not: a 180-minute cap ends the job cancelled, and the post-cancellation grace window is consumed by the earlier report-collection step, so both cozyreport.tgz and the job log are lost. Correct the comment to describe the real failure mode. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
9bd8c17971 |
fix(ci): shape-filter the promote postcondition and close test gaps
Review of the previous commit found the postcondition itself could abort
a release. It grepped the bare "X.Y.Z-rc.N" substring across the tree, and
three live files carry such strings without being image refs:
kubevirt-csi-driver's go.sum pins github.com/golang/protobuf v1.4.0-rc.2
-- a version cozystack actually cut, so promoting v1.4.0 would have
failed on a Go checksum line -- metallb_test.yaml names v1.5.0-rc.2 in a
comment, and the console pnpm-lock.yaml carries rolldown@1.0.0-rc.15. The
remediation the error message gave would have made things worse: declaring
go.sum as a ref source would sed a checksum file.
Match the version only in image-reference position now: after an
image/repository/tag key with no intervening '#', or immediately before an
@sha256 digest. Narrowing it to "${RC}@sha256:" alone would have been
wrong -- cilium keeps `tag: v1.5.0` and `digest:` under separate keys, so
the version is not always adjacent to the digest. Verified against four
real rc versions that previously aborted.
The suite also only tested the fail-closed direction, which is why that
defect survived, and the collector had no completeness guard at all: a
first-party ref added at an unenumerated path on a `latest@` tag was
mirrored by nothing and passed all eighteen tests. Since the mirror's
failure mode is the worst of the three -- the published nightly points at
the private build registry -- that gap mattered most where coverage was
thinnest.
Added: a fail-open test using the three real rc versions above; a
collector completeness diff that discovers refs by content and fails on
any the enumeration misses; coverage for the OCI-artifact shape, whose
deletion previously left every suite green; and an unreadable-file test,
since a read error was being treated as "no match" and silently skipped.
The component-versioned test was vacuous -- its rc version shared no
substring with the kamaji tag it claimed to protect, so it passed by
construction. It now uses an rc version sharing kamaji's 0.19.0 prefix,
which a loosely-anchored implementation would corrupt.
Round-trip fixture discovery matched a combined 'ghcr.io/cozystack/
cozystack/' prefix, which skips keycloak-operator (host in a sibling
registry: key) and kubeovn (host in global.registry.address). It now
matches 'cozystack/cozystack' and stamps split `tag:` keys too.
Also documented two pre-existing gaps rather than half-fixing them: the
nightly host rewrite is a contiguous-substring replace and cannot reach a
split host, and a ref inside a gzip is invisible to every tool including
the postcondition, which skips binaries. Both are host-dimension defects;
the version dimension this change owns is unaffected, because the version
always lives in a tag key regardless of where the host sits.
Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
07c67a8913 |
test(e2e): capture previous container logs and stop killing the snapshot
Root-causing a crash-loop failure was blocked twice over by the diagnostics collected on failure. `kubectl logs` shows only the current container instance, so for a crash-looping pod the suites' `podLogs` collector captured the Nth restart -- a fresh replay of the same startup path -- while the evidence that explained the failure sat in the first, interrupted instance. Chainsaw's podLogs collector has no `previous` option, so add e2e-capture-previous-logs.sh and drive it from the global catch and from the BATS failure hook. It discovers containers with restartCount > 0 at runtime (init containers included, where an interrupted bootstrap lives), orders the app suites' namespace first, and dumps each to the job log and the snapshot tree. Containers that never restarted are skipped, so a healthy run emits one line rather than a page of "previous terminated container not found". Every way that capture can come up short is named rather than inferred, because kubectl exits 1 for a garbage-collected container, an RBAC denial and an unreachable node alike: the container cap, a pod list that never returned (which is not the same as nothing having restarted), a read that timed out versus one kubectl refused and the reason it gave, and a previous instance that produced no output. A partial capture is kept and marked truncated rather than deleted -- it is the only copy of a log already gone from the cluster. The crust-gather snapshot was killed by the Chainsaw op containing it. A script op is bounded by timeouts.exec (5m), and the catch's own wall-clock wrappers summed to 960s, so the script was SIGKILLed partway through -- taking out the process group mid-write and skipping the data-plane capture entirely. Give each catch op an explicit timeout above the budgets it contains, so an inner timeout fires instead and leaves a partial, still-servable snapshot behind. The data-plane leg drops from 600s to 300s at that call site so both legs fit one envelope; the BATS hook keeps the wider budget, having no containing op. Set crust-gather --duration explicitly as well. It is the collection phase's own budget, it defaults to 60s, and on elapse it abandons the collection and skips its finish step, leaving a truncated tree with no indication anything is missing. The outer wall-clock has to exceed it because the API discovery that runs first is not covered by that budget at all -- with an unhealthy aggregated APIService, the state a failed run is usually in, discovery alone can consume the whole timeout before a single object is written. Send each collector's output to a log beside its snapshot instead of /dev/null, with a complete/truncated/failed verdict, so that question is answerable from the artifact. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
64097dc455 |
fix(platform): forward backupStorage overrides to the backupstrategy-controller Package (#3333)
## What this PR does Fixes #3245. The documented admin override for the `cozy-default` backup S3 coordinates — `spec.components.backupstrategy-controller.values` on the `Package cozystack.cozystack-platform` (`docs/operations/backup-classes.md`, "Admin overrides for `cozy-default`") — had no effect: the `cozystack-platform` PackageSource exposes exactly one component (`platform`), so the operator silently ignored the block, and the platform chart rendered the child `Package cozystack.backupstrategy-controller` via the `.default` helper with no components argument, so nothing forwarded `backupStorage` either. Admins running against an external or TLS S3 had no working supported path to set `backupStorage.endpoint` / `region` / `systemSecretName`. This PR wires the override through the platform chart, the same way `bundles/iaas.yaml` forwards kubevirt and gpu-operator values: - `packages/core/platform/values.yaml` gains a top-level `backupStorage: {}` value. - `templates/bundles/system.yaml` renders it into the emitted `Package cozystack.backupstrategy-controller` as `components.backupstrategy-controller.values.backupStorage`, from where the cozystack operator merges it into the `backupstrategy-controller` HelmRelease over the chart defaults. When `backupStorage` is unset the components block is omitted and the emitted Package is byte-identical to the historical render. - `docs/operations/backup-classes.md` now documents the working path (`spec.components.platform.values.backupStorage`) and explicitly calls out the two plausible-looking paths that do not work (the old documented one, and patching the child Package directly, which the platform helm-reconcile reverts). - A new helm-unittest suite (`packages/core/platform/tests/bundles_backupstorage_wiring_test.yaml`) is the behavioural regression test: it proves the unset case emits no components block and the set case lands the values on the child Package, on both `isp-full` and `isp-full-generic`. Verified with `make test` in `packages/core/platform` (22 suites, 91 tests, all green). `make show` requires a live cluster (`cozyhr` + `lookup` in `templates/repository.yaml`), so the unittest render of `templates/bundles/system.yaml` is the offline verification path. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#622 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Walking the trigger map against the diff: the change to `packages/core/platform/values.yaml` reaches **cozystack/website** twice — `content/en/docs/next/operations/configuration/platform-package.md` (hand-written table of `spec.components.platform.values.*` keys gains `backupStorage`) and `content/en/docs/next/operations/services/backup-classes.md` (mirrors the broken override example this PR corrects). The follow-up is cozystack/website#622, which also carries the superseded website#547 content and covers both next/ and v1.4/ docs, marked not to merge before this PR lands; its v1.4 copy assumes this fix is backported to release-1.4. No other downstream repository restates the touched files: the templates change is chart-internal, no app `values.schema.json`, CRD, variant list, `hack/` file, or namespace changed. ### Release note ```release-note fix(platform): the documented admin override for the cozy-default backup S3 coordinates now works — set spec.components.platform.values.backupStorage on the cozystack.cozystack-platform Package and the platform chart forwards it to the backupstrategy-controller component (previously the documented spec.components.backupstrategy-controller path was silently ignored) ``` |
||
|
|
09323fff13 |
fix(ci): cover every image-ref storage shape in promote, retag and mirror
Promotion rewrote the rc version substring across the depth-2 package values.yaml plus packages/apps/kubernetes/images/*.tag alone, on the premise that the kubernetes app was the only one whose .tag files carry the cozystack version. Nine other .tag files do, and system/multus seds its ref straight into a vendored upstream manifest. v1.6.0 was therefore staged with 33 of 55 first-party refs still reading v1.6.0-rc.4. promote-retag.sh and nightly-mirror.sh had the same blind spot from the other direction: both scanned only the depth-2 values.yaml, so those images were never retagged to the stable version (30 refs selected before, 42 after) and never mirrored to the public registry for a nightly. A retag miss merely leaves an image short of a tag, since the digest still resolves inside one registry; a mirror miss is worse, because the host rewrite walks the same file list and a ref that is neither mirrored nor rewritten leaves the published tree pointing at the private build registry. The three call sites each carried their own idea of where a ref can live, which is how they drifted apart. Move that knowledge into hack/lib/image-refs.sh as the single enumeration -- image_ref_files() for the files, collect_image_refs() for the refs inside them -- and have all three source it, so a new storage location is declared once and reaches every consumer. The rewrite itself moves out of promote-rc.yaml into hack/promote-rewrite-tags.sh. Being workflow-inline is why this was only observable by cutting a release: there was nothing to unit test. It now also asserts its own postcondition, scanning wider than it rewrites and failing the promotion when any ref survives in a location the enumeration does not know about, rather than shipping it quietly. Component-versioned images (kamaji's v0.19.0-cozystack.N) and third-party pass-through images do not ride the cozystack version line and are deliberately left alone; a test pins that so a future fix here cannot over-reach. capi-providers-cpprovider is a documented known gap: its ref is also embedded in a gzipped copy that the chart actually ships, and rewriting only the readable copy would diverge the two. docs/agents/image-refs.md records the contract -- three tag classes, three storage shapes, the invariants, and which consumer does what. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
3ad3c2e611 |
fix(kubernetes): move kubevirt-csi-driver base off quay.io/centos to almalinux (#3400)
## What this PR does `packages/apps/kubernetes/images/kubevirt-csi-driver/Dockerfile` pinned its runtime stage to `quay.io/centos/centos:stream9` by digest. Quay serves **only currently-tagged manifests** — when a rolling tag moves, the superseded digest immediately returns `MANIFEST_UNKNOWN`, with no grace period. `stream9` rolls roughly every six days, so the pin has already died twice: | Date | Fix | Digest pinned | Status today | |---|---|---|---| | Jul 15 | #3303 | `0c9db821…` → `a93e3316…` | `0c9db821` gone | | Jul 21 | #3386 | `a93e3316…` → `3714c89f…` | `a93e3316` gone | Each death broke `make build` on `main` and on every PR triggering a full rebuild. Because the root `build` target is serial, a failure in `packages/apps/kubernetes` (4th of 29) aborts the remaining packages and leaves the shared build cache unwritten. Repinning cannot fix this: the freshly pinned digest is always the current live tag, so it is next in line to be retired. This is not specific to CentOS — the same probe against `quay.io/fedora/fedora:latest` and `quay.io/centos/centos:stream10` found every historical digest unfetchable. This PR keeps the digest pin and changes the *source* to a registry that provably retains superseded manifests, making the pin durable rather than decorative. Renovate policy is unchanged — `pinDigests` stays on, and `almalinux:9` is an ordinary rolling tag on a retaining registry exactly like the `golang`, `alpine` and `debian` bases already in the grouped digest PR. ### Verification Both bases were run locally with the Dockerfile's exact install line (`dnf install -y e2fsprogs xfsprogs nfs-utils`): | Binary | CentOS Stream 9 | AlmaLinux 9 | |---|---|---| | `mkfs.ext4` | `/usr/sbin/mkfs.ext4` | `/usr/sbin/mkfs.ext4` | | `mkfs.xfs` | `/usr/sbin/mkfs.xfs` | `/usr/sbin/mkfs.xfs` | | `blkid` | `/usr/sbin/blkid` | `/usr/sbin/blkid` | | `mount` | `/usr/bin/mount` | `/usr/bin/mount` | | `umount` | `/usr/bin/umount` | `/usr/bin/umount` | | `fsck.ext4` | `/usr/sbin/fsck.ext4` | `/usr/sbin/fsck.ext4` | | `fsck.xfs` | `/usr/sbin/fsck.xfs` | `/usr/sbin/fsck.xfs` | | `mount.nfs` | `/usr/sbin/mount.nfs` | `/usr/sbin/mount.nfs` | Every path `k8s.io/mount-utils` shells out to is identical. Package versions: | Package | CentOS Stream 9 | AlmaLinux 9 | |---|---|---| | `e2fsprogs` | 1.46.5-8.el9 | 1.46.5-8.el9 | | `xfsprogs` | 6.4.0-7.el9 | 6.4.0-7.el9 | | `nfs-utils` | 2.5.4-44.el9 | 2.5.4-42.el9_8 | | `util-linux` | 2.37.4-26.el9 | 2.37.4-25.el9 | | `glibc` | 2.34-274.el9 | 2.34-270.el9_8 | Same upstream versions throughout; the el9 release-number lag is expected since AlmaLinux is downstream of RHEL 9 while CentOS Stream 9 is upstream. `e2fsprogs` and `xfsprogs` — the two that govern `mkfs`/`fsck` behaviour — are identical NVRs. `glibc` is not linked at all: the driver is built `CGO_ENABLED=0`. The pinned digest is the OCI image index covering `amd64`, `arm64`, `ppc64le` and `s390x` — the same architecture set as the previous base. Retention was verified empirically rather than assumed: four superseded `docker.io/library` digests taken from this repo's own git history (`golang:1.26`, `golang:1.26-alpine`, `alpine:3.24`, `alpine:3.22`) all still resolve after their tags moved. ### Scope This addresses the immediate recurring breakage only. The broader concern in #3305 — that any retired digest blocks the entire platform build, with no advance warning and no fast path to repin — is not solved here and remains open. ### Release note ```release-note fix(kubernetes): move the kubevirt-csi-driver runtime base from quay.io/centos/centos:stream9 to docker.io/library/almalinux:9. Quay retires superseded manifests as soon as a rolling tag moves, which broke builds twice in six days; AlmaLinux 9 provides an equivalent RHEL 9 userspace on a registry that retains them, so the digest pin remains valid over time. No functional change to the driver. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated the container runtime environment to use AlmaLinux 9 instead of CentOS Stream 9. * Kubernetes CSI driver behavior and startup configuration remain unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a4b5b9d57b |
docs(release): update workflow cancellation condition
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
792951b9c6 |
fix(ci): make workflow cancellation effective
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
14af87296f |
fix(kubernetes): move kubevirt-csi-driver base off quay.io/centos to almalinux
quay.io serves only currently-tagged manifests: when a rolling tag moves, the superseded digest immediately returns MANIFEST_UNKNOWN. The pinned `centos:stream9` digest has died twice in six days (#3303, #3386), each time breaking `make build` on main and on every PR that triggers a full rebuild. Repinning is not a fix -- the newly pinned digest is always the live tag, so it is next in line to be retired. Move the runtime stage to a registry that provably retains superseded manifests instead, which makes the digest pin durable rather than decorative. Pinning policy is unchanged. Verified equivalent: after `dnf install -y e2fsprogs xfsprogs nfs-utils`, both bases provide mkfs.ext4, mkfs.xfs, blkid, mount, umount, fsck.ext4, fsck.xfs and mount.nfs at identical paths. e2fsprogs and xfsprogs are identical NVRs. The driver is built CGO_ENABLED=0, so the base libc is not linked. The pinned digest is the OCI index covering amd64, arm64, ppc64le and s390x -- the same architecture set as the previous base. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
741a5acc4a |
fix(opensearch-operator): use RollingUpdate maxSurge=0 instead of Recreate (#3319)
## What this PR does
Fixes a Helm-upgrade failure in the `opensearch-operator` chart that
blocks the new release-upgrade E2E lane (added in #3276).
Upgrading from a chart that shipped no explicit strategy (v1.5.x) to a
v1.6.0-rc build fails validation immediately, helm-controller retries
it, and the HelmRelease never becomes Ready:
```
Helm upgrade failed for release cozy-opensearch-operator/opensearch-operator:
Deployment.apps "opensearch-operator-controller-manager" is invalid:
spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is Recreate
```
Observed in the ["Upgrade E2E Test" job of run
29480724982](https://github.com/cozystack/cozystack/actions/runs/29480724982)
— `UpgradeFailed ... (x19 over 11m)`, with the HelmRelease never
reaching Ready.
### Root cause
#3040 (commit
|
||
|
|
960d853706 |
feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) (#3273)
## What Adds typed `waitStrategy` + `healthCheckExprs` fields so a package/app can declare the Flux helm-controller v1.5.0 kstatus health gate for the custom resource it wraps — making `HelmRelease.Ready` mean "the CR is actually healthy" instead of "helm applied the CR". This is the operator/apiserver foundation of the kstatus epic (#2642); the API surface was reviewed and approved by the API owner in #3263. Commits (per the review's "isolated step-0" request): 1. `build(deps)` — the dependency bump (see below). 2. `feat(api)` — the fields, plumbing, tests, and regenerated code. 3. `fix(api)` — match the openapi code-generator pin to the 0.35 runtime (fixes an aggregated-apiserver crash the bump surfaced). 4. `fix(api)` — declare `OpenAPIModelName` on the apps types so SSA resolves catalog kinds. 5. `docs(api)` — note that `disableWait` silently defeats `healthCheckExprs`. ## API surface Typed fields on **both** HelmRelease-building paths: - `ApplicationDefinition.spec.release` (apiserver / catalog apps → `convertApplicationToHelmRelease`) - `ComponentInstall` (operator / platform packages → `buildHelmReleaseSpec`) ```go // WaitStrategy maps to HelmReleaseSpec.WaitStrategy.Name — a deliberate scalar // simplification of the upstream {name} object. One of poller|legacy. WaitStrategy string // HealthCheckExprs maps to HelmReleaseSpec.HealthCheckExprs (the upstream // kustomize.CustomHealthCheck type verbatim). HealthCheckExprs []kustomize.CustomHealthCheck ``` `config.ResolveWaitStrategy` is shared by both builders and **couples the default**: `healthCheckExprs` are only evaluated under the `poller` strategy, so when expressions are set and no strategy is given, the generated HelmRelease defaults to `poller`. A package that sets only `healthCheckExprs` is therefore self-contained, independent of the controller's global default. ## The dependency bump (commit 1) The `WaitStrategy` / `kustomize.CustomHealthCheck` / `HelmReleaseSpec.HealthCheckExprs` types do not exist below `helm-controller/api` v1.5.0 — the runtime is already v1.5.0 (shipped by #2602), so this closes a Go-types-vs-runtime skew. It is **not** an isolated bump: v1.5.x requires **k8s 0.35 + controller-runtime 0.23**, cascading the k8s stack 0.34 → 0.35 (and the go directive to 1.26). cozystack replaces `k8s.io/apimachinery` with a fork carrying the still-open upstream `ConvertToVersion` patch (k/k#135537). That single commit rebases cleanly onto apimachinery v0.35.0; published at `cozystack/apimachinery@release-1.35-pr135537`, and `go.mod` pins its pseudo-version. **The repo builds with zero source changes under the new stack.** > Maintainer note: please confirm the `release-1.35-pr135537` fork branch naming, or fold it into the `cozystack` branch as you prefer — the pin can be updated to match. ## Not included (conscious YAGNI) `disableWait` is intentionally omitted. The only known need (the kubernetes tenant-addon deadlock) was resolved child-side, and `waitStrategy: legacy` is not equivalent (legacy still waits Helm-v3-style; `disableWait` skips waiting entirely). Add it later when a concrete leaf needs a true no-gate escape hatch. ## Tests - `config.TestResolveWaitStrategy` — the couple-the-default logic (unset+exprs→poller; explicit legacy honored; unset+no-exprs→nil). - Operator + apiserver cases mirror each other (`TestBuildHelmReleaseSpecHealthCheckExprs` / `TestConvertApplicationToHelmRelease_HealthCheckExprs`). - `go build ./...` clean, `go vet` clean, `gofmt` clean, codegen regenerated and committed. The 4 backupstrategy CRDs regenerate with benign k8s-0.35 PodSpec description drift (DynamicResourceAllocation alpha→stable, toleration Lt/Gt, resizePolicy note) — no functional change. ## Follow-up Phase 3 (per-leaf adoption — postgres/kafka/mongodb/etc. declaring their CEL via these fields) builds on this. The full CRD→CEL catalog is maintained in the epic's readiness audit. Refs #2642 ```release-note feat(api): add `waitStrategy` and `healthCheckExprs` to ApplicationDefinition and ComponentInstall, so a HelmRelease reports Ready only once the custom resource it wraps is actually healthy ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable CEL health-check expressions for application releases and package components. * Added `waitStrategy` support (`poller` and `legacy`) with automatic `poller` selection when health checks are configured and no strategy is set. * Propagated these settings into generated Helm releases, affecting readiness/wait behavior. * **Documentation** * Updated Application and PackageSource schemas to include `healthCheckExprs` and `waitStrategy`. * Extended related Kubernetes CRD documentation, including projected pod certificate annotations and workload reference fields. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ebe111c439 |
feat(gateway): advertise HTTP/2 via ALPN on Gateway API listeners
Browsers negotiate HTTP/2 exclusively through TLS ALPN "h2". Cilium's Gateway API implementation ships with enable-gateway-api-alpn=false, so every Gateway listener advertised no ALPN protocols and every browser silently fell back to HTTP/1.1 — a regression against the ingress-nginx path, which advertised h2 out of the box. Enable ALPN (h2, then http/1.1) by default on the client-facing hop: - packages/system/cilium: default gatewayAPI.enableAlpn=true. - platform chart: new gateway.http2 value (default true), threaded as inline component values into the cozystack.networking Package for the isp-full and isp-full-generic bundles, so an explicit false overrides the system/cilium valuesFiles default either way. - apps/kubernetes: tenant-cluster cilium gets enableAlpn=true whenever the gatewayAPI addon is enabled; opt out per cluster via addons.cilium.valuesOverride. The backend hop is unchanged: gateway-to-upstream stays HTTP/1.1 unless a Service opts in per GEP-1911 via appProtocol kubernetes.io/h2c (that support is switched on together with ALPN; no cozystack-shipped Service declares such a value today). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
|
7d117679d4 |
docs(backups): document bucketName's two modes, cover isp-hosted in the wiring test
Branch-review on this PR: for provisionBucket: false the backupstrategy-controller.bucketName helper returns .Values.backupStorage.bucketName verbatim as the real S3 bucket name and every strategy CR plus the Velero BSL bakes it in, so the external-S3 section must require updating bucketName (the Secret's bucketName key alone is not read by the strategies), and the knob table must state the two-mode semantics instead of the COSI-only one. Also route the external-S3 instruction through the platform component override instead of naming raw chart values, and add the isp-hosted variant case to the forwarding test (92 tests green). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
0cd3fe20b8 |
docs(backups): credentials are projected for RestoreJob reconciles too
The projector runs from both the BackupJob and RestoreJob controllers (restorejob_controller.go), not only before BackupJobs — keep the monorepo doc aligned with the website copy (cozystack/website#622). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
93de6adc04 |
docs(backups): fix HelmRelease interval, FDB API group, BSL wait syntax, leader-election phrasing
Same corrections as the website copy (cozystack/website#622 round 4): the re-render interval is 5m via the operator's helmrelease-interval flag (not a 10-minute Flux default), the FoundationDB backup CR group is apps.foundationdb.org, the BSL readiness check uses valid kubectl wait --for=jsonpath syntax, and the projector's first synchronous round is gated on leader election rather than pod readiness. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
a5dff22c51 |
docs(backups): fix PromQL examples and bucket-name phrasing, document namespace knob and bslEnabled gap
Keep the monorepo doc aligned with the website copy (cozystack/website#622 branch-review round 3): full cozystack_backup_credentials_projection_* metric names with range vectors in the alerting examples, the intro no longer conflates the Bucket CR name with the COSI-assigned S3 bucket name, the knob table gains the backupStorage.namespace row, and the external-S3 section states that the Velero BSL derives from the same backupStorage block while velero.bslEnabled is not carried by the override path. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
b4c1e44fd5 |
docs(backups): namespace the flux reconcile example
The backupstrategy-controller HelmRelease lives in cozy-backup-controller (packages/core/platform/sources/ backupstrategy-controller.yaml), so the un-namespaced command fails out of the box. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
e74c8cb459 |
docs(backups): pin the placeholder-credentials upgrade note to v1.5
Same note as the website copy (cozystack/website#622 review round 2): name the explicit version boundary instead of 'pre-PR / After this PR', so the two documents cannot diverge — the empty-string defaults and useSystemBucket shipped in release-1.5. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
1422f8b78a |
fix(platform): forward backupStorage overrides to the backupstrategy-controller Package (#3245)
The documented admin override for the cozy-default backup S3 coordinates (spec.components.backupstrategy-controller.values on the cozystack.cozystack-platform Package) had no effect: that PackageSource exposes only the platform component, so the operator silently ignored the block, and nothing in the platform chart forwarded backupStorage to the child Package. Wire it the same way bundles/iaas.yaml forwards kubevirt / gpu-operator values: a new top-level backupStorage value on the platform chart is rendered into the emitted cozystack.backupstrategy-controller Package CR as components.backupstrategy-controller.values.backupStorage, from where the cozystack operator merges it into the backupstrategy-controller HelmRelease over the chart defaults. When backupStorage is unset the emitted Package is unchanged. Update docs/operations/backup-classes.md to the working override path (spec.components.platform.values.backupStorage) and call out the two plausible-looking paths that do not work. Add a helm-unittest suite covering the no-op default and the forwarding on isp-full and isp-full-generic. Fixes #3245 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
e73ad68a85 |
fix(backups): support cnpg-operator with barman plugin (#3313)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Fixes #3300. Migrates CNPG Postgres backups off the deprecated native
`spec.backup.barmanObjectStore` (removed in CNPG 1.29; the
`barman-cloud-*` binaries it needs are absent from the `standard` image
variant keycloak-db pins, and the bare system-flavor tags apps/postgres
pins are deprecated upstream) onto the [CloudNativePG Barman Cloud
plugin](https://github.com/cloudnative-pg/plugin-barman-cloud), across
every Postgres backup surface: `apps/postgres`, `system/keycloak`, and
the `cozy-default-cnpg` BackupClass strategy driver. The chart renders
`spec.plugins` + a `barmancloud.cnpg.io/ObjectStore` CR; the
backup-controller driver SSA-applies an `ObjectStore` and patches
`spec.plugins`; `Backup`/`ScheduledBackup` use `method: plugin`. The
`postgres-operator` chart vendors and co-locates the
`plugin-barman-cloud` operator (with its cert-manager-issued mTLS) so
CNPG can discover it.
One platform-level change is required for the plugin to run and ships
with this PR (see the upgrade notes in
`docs/operations/backup-classes.md` and the release note below): the
CloudNativePG operator is bumped to 1.28.1 (also fixing an operator/CRD
version skew that broke all backups). The other prerequisite — the
LINSTOR scheduler admission webhook stripping
`initContainers[].restartPolicy` cluster-wide, which broke the plugin's
restartable sidecar — is already fixed on `main` (
|
||
|
|
d95685c427 |
Prepare release v1.6.0-rc.4
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
|
cc66e22952 |
fix(ci): mirror and retag images whose digest is embedded in the values tag (#3375)
## What's broken
`collect_refs` in both `hack/nightly-mirror.sh` and
`hack/promote-retag.sh` recognises three values shapes. The one most
package Makefiles actually write is not among them:
```yaml
image:
repository: <registry>/linstor-csi
tag: main@sha256:<digest> # digest embedded in the tag
```
Package Makefiles set `.image.tag` to `"$(IMAGE_TAG)@$(digest)"` in a
single `yq` call rather than maintaining a separate `digest` key. That
matches neither the `{repository, tag, digest}` rule (no `digest` key)
nor, usefully, the single-string rule — that one matches the bare tag
value, which carries no repository, so `ref_repo()` reduces
`main@sha256:...` to the repository `main` and the ownership filter
drops it.
Being dropped is silent. Both scripts then report success having simply
never seen those refs.
The same emit-then-drop hits a second class, found in review: refs whose
registry host does not live inside `repository` at all.
`keycloak-operator` splits it into a sibling `registry` key, and
`kubeovn` keeps it in the document-level `global.registry.address` with
a bare `repository: kubeovn`. Both are built and pushed to `$REGISTRY`
by cozystack — `kubeovn` by the `cozystack/kubeovn-chart` wrapper repo,
whose own `make image` writes all three keys — and both were emitted
host-less and discarded as third-party.
## Impact
**Nightly (`nightly-mirror.sh`) — this is the one that broke.** The
image is never copied to GHCR, *and* the unconditional host rewrite at
the end of the script still repoints its repository at the destination
registry. The result is a published reference to an image that was never
pushed there.
Confirmed against the `:nightly` tag on GHCR — the two packages using a
recognised shape are present, and eight images across six packages using
this shape are absent:
| shape | packages | `:nightly` |
|---|---|---|
| `{repository,tag,digest}` | cilium | present |
| single-string full ref | cozystack-operator | present |
| `{repository, tag:<t>@sha256}` | kamaji, kilo, linstor-csi,
piraeus-server, linstor-gui, metallb-controller, metallb-speaker,
redis-operator | **absent** |
This surfaced as the first full nightly run failing at install: the
`e2e-image-prepuller` DaemonSet never became ready (0/3, 10-minute
timeout) because `linstor-csi` and `piraeus-server` 404 on GHCR. The
prepuller only templates kubeovn/linstor/cert-manager, so it exposed two
of the eight; the other six would have failed later at HelmRelease
install.
**Promotion (`promote-retag.sh`) — same blind spot, milder
consequence.** This retags within a single registry, so the digests
already exist and digest-pinned installs still resolve — there is no
dangling reference and no 404. What is missing is the `:<version>` tag
(and `:latest` under `MOVE_LATEST=1`) for those images, so a promoted
release does not carry the tags it claims to.
The registry shows how long that has been true for the two host-split
images. `keycloak-operator` carries exactly one tag, `latest`, though
the package has shipped since 2024-11. `kubeovn`'s newest
cozystack-versioned tag is `v1.14.11-v0.38.8`, from before the 1.0
versioning switch — while `cilium`, which uses an already-recognised
shape, carries `v1.5.0` through `v1.6.0-rc.2`. Neither has received a
tag from any 1.x release.
Measured against the committed tree, the dry-run selects **19 unique
repositories before this change and 29 after** — the eight digest-in-tag
images plus `keycloak-operator` and `kubeovn`, nothing lost.
### What the nightly does *not* recover
Four of the ten are not in the root `Makefile`'s `build` target: `kilo`,
`redis-operator`, `keycloak-operator` and `kubeovn`. Their baked refs
keep the committed GHCR host, so the `SRC_REGISTRY` filter still drops
them and their `:nightly` tags remain absent after this PR. Installs are
unaffected — those digests already resolve on GHCR. The nightly recovers
the six OCIR-built images; promotion recovers all ten.
## Why this was invisible
Latent since `nightly-mirror.sh` was added — the nightly had never
completed a full run, so its mirror step was never exercised end to end.
Both test suites were green throughout. `nightly-mirror_test.bats`'s
fixture covered the single-string, `{repository,tag,digest}` and
platformSource shapes — a layout only cilium actually uses — and omitted
the dominant one. `promote-retag_test.bats` asserted "at least one
cozystack-owned image is selected", which is satisfied by the shapes
that already worked and cannot detect an omission.
Two comments actively pointed the wrong way, and are corrected here. The
header presented its list as the shapes the build writes, which reads as
exhaustive; it now says the list is empirical, names what an audit found
outside it, and says to re-run that audit when a package is added. The
ownership-filter comment named `kilo`, `kube-ovn` and
`keycloak-operator` as third parties "not pushed to `$REGISTRY`" — all
three are pushed there, and that wrong comment is part of why the
missing tags went unnoticed.
Both fixtures are extended here. The promotion test names concrete
packages rather than asserting a count or a generic pattern, since
either would drift back to proving nothing.
## Type guards
Both digest-in-tag rules check `.tag` is a string before `test()`.
Without it a non-string tag aborts the whole expression (`cannot match
with !!int, can only match strings`), and since these invocations
swallow stderr and status, that abort would silently discard **every**
ref of this shape in the same file — reintroducing this very bug in any
file that happens to carry an unquoted `tag: 1.24` next to a cozystack
image. Rule 1 is immune because its `select(tag == "!!str")` already
precedes its `test()`.
`registry`, `repository` and `digest` are guarded too, against a
narrower failure. Those keys are concatenated rather than `test()`ed,
and yq coerces int, bool, null and seq on `+` — each merely yields a ref
the ownership filter discards. Only a `!!map` aborts, raising `!!str ()
cannot be added to a !!map`, which takes the file's whole expression
with it. So that guard is insurance against a nested-map typo in a
vendored chart, not against unquoted scalars. It is a `select()`, so an
offending map drops only itself and its neighbours still resolve.
No non-string value of either kind exists in the tree today, so both are
prevention rather than an active fix.
## Testing
- Both suites pass: `nightly-mirror_test.bats` 2/2,
`promote-retag_test.bats` 4/4.
- Every new assertion was mutation-tested — removing the rule under test
turns it red. Verified individually for the shape-3 registry join, the
shape-2 registry join, the `global.registry.address` rule, and the
`repository` type guard. Also verified that a rule matching the bare tag
instead of the repository (the subtle wrong fix) fails, which is why the
assertions check the full source ref rather than only the destination.
- `promote-retag.sh --dry-run` against the real committed tree: 19 → 29
unique repositories, no losses.
- An audit of every `@sha256:` digest under `packages/*/*/values.yaml`
against the selector output accounts for all 35: 30 are selected, and
the remaining five are third-party
(`ghcr.io/lexfrei/{kuberture,ouroboros}`, `docker.io/clastix/kubectl`,
`ghcr.io/kvaps/...`) or sit outside `$REGISTRY`
(`ghcr.io/cozystack/ingress-nginx-with-protobuf-exporter/*`).
- `shellcheck` clean on both scripts at default severity and `-S style`.
## Ordering note for reviewers
Worth landing before the next release candidate is cut, rather than
treating this as routine CI cleanup.
`promote-rc.yaml` builds the release branch from the rc's staging
branch, and `pull-requests-release.yaml` runs `./hack/promote-retag.sh`
from that merged tree — so the copy of the script that executes at
promotion is the one frozen into the rc tag's tree, not whatever is on
the default branch at promotion time. An rc cut before this merges will
carry the old script and silently skip those retags when promoted.
The nightly half is also what unblocks the full-suite e2e lane, which
cannot currently publish a working install closure to GHCR.
### Downstream repositories
Walked the trigger map in `docs/agents/contributing.md` against the
diff: it touches only `hack/nightly-mirror.sh`, `hack/promote-retag.sh`
and their two `.bats` suites. No `packages/`, no chart, no
`values.schema.json`, no `ApplicationDefinition`, no CRD or RBAC. The
`cozystack/ccp` trigger covers moving or renaming files under `hack/`
and changing what a make target does; neither applies — the anchor files
it gates on (`hack/package.mk`, `hack/common-envs.mk`) are untouched and
no make target changes behaviour.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
fix(ci): mirror and retag every cozystack-owned image, including those whose digest is embedded in the values `tag` and those whose registry host is stored outside `repository`. Ten images across eight packages were silently skipped: the nightly published references that 404 at install, and promoted releases did not carry the `:<version>` tags they claim.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved nightly mirroring and retag promotion to recognize
digest-pinned image references across more YAML value shapes, including
digests embedded in the `tag` field and optional registry wrappers.
* Expanded support for chart-global image definitions and safer parsing
to avoid silently skipped references.
* Strengthened selection logic to exclude third-party images from copy
plans.
* **Tests**
* Expanded dry-run coverage with additional fixture cases for split
maps, `tag`-embedded digests, numeric-tag variants, and more “owned”
image assertions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
v1.6.0-rc.4
|
||
|
|
1b777d8077 |
fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver (#3386)
## What this PR does `quay.io/centos/centos:stream9` is a **rolling tag** — upstream deletes the previously-pinned digest whenever it publishes a new `stream9` build. The digest re-pinned in |
||
|
|
3ca62266f8 |
fix(platform): qualify package names in the bundle guard messages
Address review feedback on the guards added alongside the isp-hosted dependency-closure fix. Spell every package the guards name in its fully-qualified form (cozystack.cozystack-engine rather than cozystack-engine, and likewise for cert-manager, prometheus-operator-crds, postgres-operator, objectstorage-controller and cozystack-basics). The messages exist to be acted on, and an operator who greps the emitted Package CRs for the name they were given should find it. Also correct the direction of the cross-reference in the bucket-application comment: the velero comment it points at sits above the block, not below. No behavioural change — the guards fire on the same conditions and the tests match on the unchanged leading clause. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
0bfbff26ff |
fix(platform): restore the isp-hosted dependency closure
The isp-hosted preset shipped bundles.system.enabled: false with paas and
naas enabled. templates/bundles/system.yaml is gated end to end on that
flag, but it carries two separable things: a variant-selected data-plane
head (kubeovn/cilium/linstor on isp-full, a noop networking Package on
isp-hosted) and a variant-agnostic control-plane body — cozystack-engine,
the sole source of the application-definition-crd component, plus
cert-manager, prometheus-operator-crds, postgres-operator and
objectstorage-controller.
"Hosted" means the host already provides CNI, storage and virtualisation,
so only the data-plane head should be dropped. Disabling the whole bundle
also deleted the control-plane body, leaving an app catalog whose entire
dependency closure was absent: all 24 emitted Packages sat in
DependenciesNotReady, the *-rd HelmReleases failed with `no matches for
kind "ApplicationDefinition" in version "cozystack.io/v1alpha1"`, and the
cozystack-platform HelmRelease never went Ready. That the variant field
was still set to isp-hosted — and that system.yaml has an isp-hosted
branch which could never render — shows the disable was the mistake, not
the intent; iaas.yaml already refuses to render on this variant, encoding
the same "system bundle minus iaas" contract. The preset has never worked
since it was introduced in
|
||
|
|
9c3bae012a |
fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver
quay.io/centos/centos:stream9 is a rolling tag; upstream deletes the pinned
digest whenever it publishes a new stream9 build. The digest re-pinned in
|
||
|
|
292f8ec65a |
fix(ci): type-guard the concatenated keys in the ref selectors
Address non-blocking review feedback from lexfrei on hack/nightly-mirror.sh:81 and hack/promote-retag.sh:84. `registry`, `repository` and `digest` are joined into a ref by string concatenation with no type guard, so a non-string value could abort the whole per-file expression. Because collect_refs swallows both stderr and status, that abort silently discards every co-located owned ref — the same per-file blast radius the `tag` guard already prevents. The mechanism differs from `tag`'s, though, and the narrower reading is worth recording. These keys are never fed to test(), so the "cannot match with !!int" abort does not apply to them: yq coerces int, bool, null and seq on `+`, and each of those merely yields a ref the ownership filter discards. Only a !!map actually aborts, raising "!!str () cannot be added to a !!map". The guard is therefore narrow insurance against a nested-map typo in a vendored chart, not protection against unquoted scalars. Implemented as select() rather than a test(), so an offending map drops only itself and its neighbours in the same file still resolve. The nightly fixture gains a file pairing a map-typed `repository` with an owned ref, asserting the latter is still mirrored; removing the guard turns it red. Selection over the committed tree is unchanged at 30 repositories. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
acd88bdd00 |
fix(ci): collect image refs whose host is outside repository
Address review feedback from lexfrei and IvanHunters on
hack/promote-retag.sh:84 and hack/nightly-mirror.sh:81.
The shape-3 rule assumed the registry host lives inside `repository`.
Two cozystack-built images do not satisfy that and were emitted
host-less, so the ownership filter dropped them — the same silent
emit-then-drop this PR fixes for the other shapes:
- keycloak-operator splits the host into a sibling `registry` key
({registry: ghcr.io, repository: cozystack/cozystack/keycloak-
operator, tag: latest@sha256:...}), and its own Makefile writes
exactly that layout.
- kubeovn keeps the host in the document-level
global.registry.address with a bare `repository: kubeovn`. It is
built and pushed to $REGISTRY by the cozystack/kubeovn-chart
wrapper, whose `make image` writes all three keys.
Neither has received a 1.x release tag: on GHCR keycloak-operator
carries only `latest`, and kubeovn's newest cozystack-versioned tag is
v1.14.11-v0.38.8, from before the 1.0 versioning switch — while cilium,
which uses an already-covered shape, has v1.5.0 through v1.6.0-rc.2.
Shapes 2 and 3 now rejoin an optional `registry` sibling, and a new
shape-4 rule handles the chart-global layout. An audit of every
@sha256: digest under packages/*/*/values.yaml against the selector
output confirms the remaining five misses are third-party or sit
outside $REGISTRY. Dry-run over the committed tree: 28 -> 30
repositories, exactly these two gained, nothing lost.
Also correct two comments that overstated coverage. The header no
longer presents its list as the shapes the build is limited to — the
list is empirical and charts are vendored — and the ownership-filter
comment no longer names kilo, kube-ovn and keycloak-operator as
third parties not pushed to $REGISTRY. All three are pushed there, and
that wrong comment is part of why the missing tags went unnoticed.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
b897b53107 |
test(postgres): fix S3 CA secret name in backup roundtrip e2e
The backup roundtrip e2e copied the seaweedfs S3 CA from a Secret named seaweedfs-system-ca-cert in tenant-root, a name captured off a dev cluster. On current main the seaweedfs chart pins fullnameOverride: seaweedfs, so the cert-manager CA Certificate's secret is seaweedfs-ca-cert; the old name is absent and the first CI run of the e2e failed at step 05a with 'secrets "seaweedfs-system-ca-cert" not found'. Default S3_CA_SECRET to seaweedfs-ca-cert and, when that Secret is absent, auto-discover the seaweedfs CA Certificate (the seaweedfs-labelled one with spec.isCA=true) and read its secretName, so a future fullnameOverride change does not silently break the copy again. Fail with a clear message when neither resolves. README and chainsaw comment updated to match. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
770314b3cb |
fix(ci): retag images whose digest is embedded in the values tag
promote-retag.sh carries the same collect_refs blind spot the previous commit fixes in nightly-mirror.sh, and nightly-mirror.sh's own comment asserted the two were "identical to" each other -- which is how the gap stayed invisible once one of them was fixed. Measured against the committed tree, the dry-run selects 19 repositories before this change and 27 after. The eight it was missing are exactly the set that broke the nightly: kamaji, kilo, linstor-csi, piraeus-server, linstor-gui, metallb-controller, metallb-speaker, redis-operator. Nothing is lost from the previous selection. The consequence differs from the nightly case and is milder. This retag is within a single registry, so the digests already exist and every digest-pinned install still resolves -- there is no dangling reference and no 404. What is missing is the :<version> tag (and :latest under MOVE_LATEST=1) for those eight images, so a promoted release does not carry the tags it claims to, and anything resolving a cozystack component by version tag rather than by digest does not find it. Nothing in the promotion detects this: the script reports success having simply never seen those refs. The existing test could not catch it. It asserts "at least one cozystack-owned image is selected" and that nothing third-party leaks in -- both satisfied by the shapes that already worked, which is why eight omissions sat under a green suite. It now names concrete shape-3 packages, since a count or a generic pattern would drift back to proving nothing. Verified the assertion fails against the unfixed script. Same type guard on `.tag` as the previous commit, for the same reason: without it an unquoted numeric tag anywhere in a file silently discards every shape-3 ref in that file, and the swallowed stderr leaves no trace. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
06278ac18b |
fix(ci): mirror images whose digest is embedded in the values tag
The nightly's full-suite e2e failed at install: the e2e-image-prepuller
DaemonSet never became ready, because
ghcr.io/cozystack/cozystack/linstor-csi@sha256:2f314cb9... and
.../piraeus-server@sha256:39fdacec... both 404 on GHCR. They are not there
because nightly-mirror.sh never copied them.
collect_refs handled three values shapes, and the one most package Makefiles
actually write is not among them: a split map whose digest is embedded in the
tag, {repository, tag: <tag>@sha256:<digest>}, produced by a single yq call
setting .image.tag to "$(IMAGE_TAG)@$(digest)". It matches neither the
{repository, tag, digest} rule (no `digest` key) nor, usefully, the
single-string rule -- that one matches the bare tag value, which carries no
repository, so ref_repo() reduces "main@sha256:..." to the repository "main"
and the SRC_REGISTRY case drops it.
Being dropped there is silent, and the host rewrite at the end of the script
runs over every depth-2 values.yaml regardless of what was mirrored. So the
image was never copied AND its repository was still repointed at
DST_REGISTRY, leaving a reference to an image that was never pushed there --
exactly the dangling-ref failure the comment above that rewrite warns about,
reached by a path it did not anticipate. The 404 then surfaces at pull time,
one job later, as a DaemonSet rollout timeout that names no image.
This is not limited to linstor. Checking the :nightly tag on GHCR, the two
packages using a recognised shape (cilium, cozystack-operator) are present
and eight images across six packages using this shape are absent: kamaji,
kilo, linstor-csi, piraeus-server, linstor-gui, metallb-controller,
metallb-speaker, redis-operator. The prepuller only templates
kubeovn/linstor/cert-manager, so it surfaced two of the eight; the rest would
have failed later at HelmRelease install.
Latent since the script was added in
|
||
|
|
f5227ee60f |
fix(seaweedfs): close the 4.31 rename fallout on the 1.5.x→1.6 upgrade path (#3339)
## What this PR does Cozystack v1.5.0 bumped the vendored SeaweedFS chart 4.0.405 → 4.31.0, which renamed every workload after the Helm release (`<name>-system-*`). StatefulSet names are immutable, so upgrades through 1.5.x stood up a second, empty set beside the running one instead of renaming. #3282 pinned `fullnameOverride: seaweedfs` so 1.6 adopts running workloads in place — this PR closes the remaining holes on that adoption path, found by driving a disposable 3-node cluster through a real v1.4.5 → v1.5.3 → main upgrade with SeaweedFS tenants planted in every reachable state. - **The naming guard moves into `packages/system/seaweedfs`** — the render a platform upgrade actually re-renders: the `<name>-system` HelmRelease pulls this chart from a platform-managed ExternalArtifact, so the previous guard in `extra/seaweedfs` was never in the path. Upgrading a fresh-1.5.x tenant therefore created an empty chart-named set beside its live data and flipped the guard's own legacy-data signal. `extra/` keeps a sibling copy for operator visibility; a bats parity suite pins the two detection blocks byte-identical. - **The guard refuses instead of guessing when both naming generations exist.** Nothing durable distinguishes a duplicate that never served from one that served and crashed: claim timestamps invert during the recovery runbook's own re-bind, `readyReplicas: 0` is a snapshot, and Helm birth order answers "which generation is original", not "is the other one empty". Exactly one generation present is decidable, and the render proceeds (or refuses as class S) on its own. - **Cluster-scoped RBAC is named per namespace again.** 4.31 named four cluster-scoped objects after the release — identical for every tenant — so all tenants collided on one ClusterRole/ClusterRoleBinding and only the last-reconciled tenant's COSI provisioner kept its RBAC. Names return to `global.seaweedfs.serviceAccountName`; three of four are byte-identical to pre-4.31 and adopt in place. - **The `seaweedfs-db` hand-over runs for every instance name.** Migration 43 compared the owning release against the literal `seaweedfs-system`, so an instance named `foo` was skipped and its CNPG Cluster — the filer metadata for every object in that tenant's S3 — was pruned on the next reconcile, PVC included. The comparison now matches the `-system` suffix (shared `lib/seaweedfs-db-adopt.sh`), and new migration 53 re-runs the hand-over for clusters already past 43, before anything re-renders. - **`hack/seaweedfs-naming-audit.sh` + `docs/operations/seaweedfs-431-rename-recovery.md`** — what the guard's refusal points operators at: read-only classification (`L` / `S` / `MIXED`, naming the candidate duplicate from relative PV vintage, never a clock window) and the recovery procedures. **Scope.** The supported SeaweedFS deployment is the tenant module, which hardcodes the instance name (`packages/apps/tenant/templates/seaweedfs.yaml`); a tenant only enables or disables it. The runbook and its selectors are scoped to that name; instances created directly against the API under other names are classified by the audit but routed to escalation. Zone/pool keys of ~40 or more characters fall outside the guard's reconstruction — an accepted limit, recorded in `_naming.tpl` and the runbook. **Upgrade impact.** 1.4.x → 1.6: no manual steps — one generation, adopted in place. 1.5.x → 1.6 with SeaweedFS: migration 53 protects every `seaweedfs-db` first; then the upgrade **refuses** for any tenant holding both naming generations until the operator resolves the duplicate. This is deliberate — which generation holds the data is not decidable from inside a render, and guessing wrong destroys it. Release notes should present the refusal as expected behavior. **Testing.** 55 chart unit tests across both packages (including new MultiZone zone-component guard cases — the suite previously had no zone shapes), 11 audit bats, 8 guard-parity bats, migration bats. Validated end-to-end on a disposable 3-node cluster driven v1.4.5 → v1.5.3 → main with five tenants covering: never-saw-4.31 (renders untouched), fresh-1.5.x (refused as class S), wedged duplicate, split duplicate, and a non-default-named instance (its database survives only with this fix). The audit classifies all five correctly, and its two independent signals — revision-1 birth scheme and relative PV vintage — agree on every MIXED tenant. Related: #3282 (fullnameOverride pin), #3335 (etcd adoption backup gate — separate, also required for the 1.5.x→1.6 path). ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change Walked the trigger map against the diff: no package added/renamed under `packages/{apps,extra}/`, the `packages/core/platform/values.yaml` change is only the migrations `targetVersion` bump (no `spec.components.platform.values.*` key changes), no variant/bundle/component changes, no asset renames, no ApplicationDefinition semantic changes. ### Release note ```release-note fix(seaweedfs): the 1.6 upgrade no longer renames a SeaweedFS instance away from its data. The naming guard now runs in the chart a platform upgrade actually re-renders and refuses when both pre- and post-4.31 naming generations exist; hack/seaweedfs-naming-audit.sh and docs/operations/seaweedfs-431-rename-recovery.md guide recovery, and the refusal is expected for tenants that passed through 1.5.x. Cluster-scoped COSI RBAC is named per namespace again (the 4.31 release-based names collided across tenants), and the seaweedfs-db hand-over runs for every instance name — previously an instance not named `seaweedfs` had its filer metadata database pruned on upgrade; new migration 53 repairs clusters that already ran the old hand-over. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added fail-closed upgrade protections for SeaweedFS naming migrations (mixed/damaged classification), including safer behavior when cluster visibility is limited. * Added cluster-scoped RBAC uniqueness safeguards to prevent cross-tenant name collisions during rendering. * Improved SeaweedFS database adoption/repair migrations and strengthened post-delete cleanup ownership checks. * **Bug Fixes** * Hardened SeaweedFS 4.31 rename recovery and PV rebind flow, including reclaim policy preservation and long/non-default instance-name edge cases. * **Documentation** * Expanded the SeaweedFS 4.31 rename-recovery runbook with clarified auditing, verification, and escalation. * **Tests** * Added/expanded integration and Helm rendering tests for the above scenarios and refusal/fail-closed behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->v1.6.0-rc.3 |
||
|
|
8020656725 |
fix(ci): resolve nightly packages digest without image-specific skopeo ops (#3332)
## What this PR does
The Nightly workflow has **never completed a run**. Every scheduled run
since `NIGHTLY_ENABLED` was set has failed in `mirror` at "Resolve
source commit and version", ~80s in, with:
```
level=fatal msg="unsupported image-specific operation on artifact with type \"application/vnd.cncf.flux.config.v1+json\""
```
Nothing has ever been published — `cozy-installer:nightly` and
`cozystack-packages:nightly` both 404 on GHCR, and `cozystack-nocloud`
does not exist.
This fixes that failure and the next two behind it.
### 1. The digest lookup could never have worked
`cozystack-packages` is a **Flux OCI artifact** (config mediaType
`application/vnd.cncf.flux.config.v1+json`), not a container image. A
plain `skopeo inspect` parses the config blob as an image config and
fatals on it, so `skopeo inspect --format '{{.Digest}}'` was never
viable against this ref — the `--raw` call on the very next line already
acknowledged this.
Replaced with `skopeo inspect --raw` + `skopeo manifest-digest`, which
hashes the manifest bytes — exactly what the registry addresses the
artifact by, with no image-specific parsing.
This also collapses two registry fetches into one: the digest and the
`org.opencontainers.image.revision` annotation now come from the **same
bytes**, which strengthens the race the original comment describes —
there is no longer any window between pinning the tag and reading it.
### 2. `flux pull artifact` rejects a non-existent output path (×2)
`flux pull artifact --output _baked` fails with `invalid output path
"_baked": no such file or directory`. `_baked` is neither tracked nor
created by anything. `pull-requests.yaml` does `rm -rf` + `mkdir -p`
first; `nightly.yaml` omitted it. Since the workflow always died at step
1, this was never reached.
Same bug for `_rw` in the e2e staging step. Both now create the
directory first.
## Verification
- Reproduced the original fatal against the live artifact with skopeo
1.22.2.
- Ran the fixed resolve step end-to-end against the real registry:
returns the digest the registry reports, and a `sha` that matches the
current `main` HEAD.
- Reproduced the `_baked` failure and confirmed the fix extracts the
tree.
- `actionlint` clean; `zizmor --offline` clean; `shellcheck` clean on
changed lines.
Also checked and **not** changed, each disproven rather than assumed:
- `skopeo copy --multi-arch all` on a Flux artifact — works; `copy` is
mediaType-agnostic.
- The `awk -F@ '/artifact successfully pushed/'` digest capture —
matches current flux CLI output, and is the same idiom
`packages/core/installer/Makefile` already uses in production.
- `nightly-mirror.sh`'s `inspect --format Digest` — all mirrored refs
there are genuine image manifests, so it is correct as written.
- `make -C packages/core/talos talos-nocloud` — exists and is CI-proven
via `tags.yaml`.
## Honest scope
This fixes the `mirror` job's blockers. It does **not** promise a green
nightly on the first run: every step after `mirror` is still unproven,
having never executed. The most likely next wall is `build-disk` —
`cozystack-nocloud` is a brand-new package, so the first `oras push`
creates it, and the follow-up `oras tag` can hit `403 permission_denied`
until the package is linked (the risk the workflow already documents
in-line). The first-ever full e2e run may well surface its own issues.
Reviewing that follow-on separately seems better than growing this PR.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved nightly artifact mirroring and staging reliability.
* Prevented artifact pull failures by ensuring required output
directories are created before use.
* Enhanced retrieval of artifact version and source revision metadata.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
a15206502f |
ci(labeler): map cert-manager, external-secrets and metrics-server scopes (#3362)
## What this PR does PRs titled with the `cert-manager`, `external-secrets` or `metrics-server` scopes landed on `area/uncategorized` because the labeler's scope map had no entry for them — all three surfaced at once in the current port-collision series (#3359, #3360, #3361). metrics-server feeds the monitoring pipeline, so it maps to `area/monitoring`; cert-manager and external-secrets are platform services, so they map to `area/platform`. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Chores** - Improved pull request labeling automation. - Pull requests involving metrics, certificate management, and external secrets are now categorized more accurately with the appropriate area labels. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
bd770b2288 |
fix(hack): make the seaweedfs naming audit POSIX sh
hack/seaweedfs-naming-audit.bats sources the audit script at top level,
and cozytest.sh sources the converted test file into its own /bin/sh --
dash 0.5.12 on the Ubuntu CI runner. dash has no `pipefail`, so
`set -uo pipefail` aborted the runner before the first @test and took
the whole file down with it:
hack/cozytest.sh: 48: set: Illegal option -o pipefail
The bash shebang could not prevent this: `.` executes the file's
contents in the caller's shell, so the shebang is only a comment there.
The script was already POSIX apart from those two lines -- no arrays,
no [[, no local, no bashisms at all -- so the script becomes POSIX sh
rather than the runner becoming bash. That keeps the shell under test
and the shell at run time the same one, instead of testing under dash
while shipping under bash.
Dropping `pipefail` changes no behaviour: the script never sets -e and
never reads a pipeline's exit status, reporting failure as empty output
checked with [ -n ... ].
Verified under dash (ubuntu:24.04): 11/11 tests pass, and the script
runs clean against a live cluster, classifying five seaweedfs tenants
with the revision-1 scheme and the PV ages agreeing on every MIXED
case. shellcheck -s sh reports nothing.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
9b82d1bf36 |
test(seaweedfs): run the db-adopt migrations in the shell they ship with
Address review feedback from IvanHunters on packages/core/platform/images/migrations/migrations/43:23: The suite invoked the migrations as `sh "$MIG_DIR/<n>"`. On the CI runner /bin/sh is dash, which has no `set -o pipefail`, so both scripts aborted on their `set` line with "set: Illegal option -o pipefail" (exit 2) before doing any work, and the job went red. Production was unaffected — the migrations image is FROM alpine, so /bin/sh there is busybox ash — which is exactly why checking the runtime image missed it. Rather than change a shebang or drop pipefail, run the migrations the way run-migrations.sh does: by path, inside the image's own pinned alpine base, read out of the migrations Dockerfile so the interpreter under test cannot drift from the one they ship on. pipefail is load-bearing here — lib/cozystack-version.sh pipes the rendered manifest into `kubectl apply`, so without it a failed render stamps the version from empty input — and a shell that lacks it can prove nothing about that path. run_migration() returns the container's status explicitly: cozytest.sh's awk generator rewrites every bare `}` in column 0 into `return 0` + `}`, so a helper that falls off its own end reports success no matter what it ran, and every fail-closed assertion would have passed vacuously. 15/15 pass. Verified the interpreter under test is BusyBox v1.37.0 ash with pipefail supported, and that a fail-open mutation in lib/seaweedfs-db-adopt.sh turns the suite red. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
8355f3e36f |
docs(seaweedfs): correct the stale "D-wedged is adopted" comments
The naming guard now refuses every two-generation state, but two comments in extra/seaweedfs still described a D-wedged duplicate (zero ready replicas) as safe to adopt and rendering through. readyReplicas is a snapshot, not proof the duplicate never served, so it cannot be told apart from D-split and the guard refuses it too. Align the state summary in the seaweedfs template comment and the test header with the presence-only refusal behaviour. No classifier change -- comments only. Address review feedback from coderabbitai on packages/extra/seaweedfs/tests/fullname_override_test.yaml. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
a099b7e086 |
fix(seaweedfs): fail the cleanup Job on a database-owner lookup error
The post-delete cleanup hook read Cluster/seaweedfs-db's Helm owner with `|| true`, so a Forbidden, a timeout or any API error collapsed to an empty owner, took the "nothing to reclaim" branch, and let the Job complete -- permanently orphaning the keep-protected Cluster the hook exists to reclaim, with nothing else that will ever collect it. Only a genuine NotFound now counts as absence; every other error prints the failure and exits non-zero so the Job retries. This matches the fail-closed handling the delete path already had. Add a regression assertion that the ownership lookup no longer swallows failures. Address review feedback from coderabbitai on packages/extra/seaweedfs/templates/hooks/cleanup.yaml. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b4f98af940 |
docs(seaweedfs): fail closed in the rename runbook's reclaim-policy handling
The 1.5.x->1.6 rename recovery runbook deletes each volume PVC after switching its PV to Retain, but the switch was never gated on success. With no `set -e`, a failed `kubectl patch ... Retain` left the PV at Delete and the following `delete pvc` took the data with the claim. Gate the delete on both the annotation record and the Retain patch succeeding, aborting the re-bind loop otherwise, so no PVC is deleted while its PV is unprotected. In the Step 5 restore loop, chain the annotation removal after the restore patch with `&&` so a failed patch keeps the saved policy for a retry. Address review feedback from coderabbitai on docs/operations/seaweedfs-431-rename-recovery.md. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b16a51e60e |
test(seaweedfs): pin the naming guard's MultiZone zone-component classification
MultiZone (and Simple-with-pools) tenants render no plain `volume` component — extra/seaweedfs sets volume.enabled=false and the vendored chart emits one component per zone key with suffix `volume-<key>`, truncated at (62 - len(suffix)), which is SHORTER than the 56-char cut the guard's renamedVolumePrefix replays. The final review pass flagged this divergence as a guard bypass. For the supported instance the prefixes cannot diverge: the tenant module hardcodes the name `seaweedfs`, the fullname is 16 chars, and nothing truncates until a zone/pool key of ~40+ characters. Three new cases pin that supported path — legacy-only zones adopt, release-named-only zones refuse as class S, both generations refuse — since the suite previously had no zone shapes at all and the zone objects are the ONLY evidence on a MultiZone tenant. The divergence beyond that boundary (long instance names or absurd keys) is an accepted, documented limit, not a fix: decided in the 1.6 triage, recorded in _naming.tpl (both copies, kept in lockstep) and the runbook's Scope section. Instance names are fixed by the tenant module, so absurd keys are the only reachable trigger; revisit if instance naming is ever opened up. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
95fcc76902 |
docs(seaweedfs): scope the rename runbook to the supported instance name
The tenant module is the supported way to run SeaweedFS and it hardcodes the instance name: packages/apps/tenant/templates/seaweedfs.yaml creates the HelmRelease as `seaweedfs`, and a tenant can only enable or disable it. Every shell selector in this runbook assumes that name, so say so up front, and route instances created directly against the API under other names to escalation instead of through loops whose name filters provably cannot see truncated claims. The ~40-char zone/pool key limit of the guard's reconstruction is called out as an accepted limit for default-named instances too. Two content fixes ride along, both for findings the final review pass raised: - The emptiness check told the operator to kubectl exec the candidate's own volume pod — which for a D-wedged duplicate can never start; that is the definition of the class, so the check was unexecutable exactly where it matters most and every D-wedged tenant funneled to escalation. Mount each bound candidate claim read-only in a scratch pod instead (a Pending claim has no PV and is empty by construction). The weed shell cross-check stays on the authoritative master, which is running. - Step 1's description of the audit now matches the audit itself: direction is decided by the relative strictly-newer PV rule (the same precondition Step 2a enforces), and an interrupted re-bind reads as overlapping vintages with no candidate — not by a clock window anchored on first_deployed. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
24bcb0bb69 |
fix(seaweedfs): decide MIXED direction by relative PV vintage, not a clock window
The audit named the candidate duplicate by measuring each generation's oldest PV against the release's first_deployed, with an absolute 120-second window for the mid-rebind case. The shipped StorageClasses are WaitForFirstConsumer, so PVs are provisioned at pod-SCHEDULE time — on a cold cluster minutes after first_deployed — and a tenant interrupted mid-way through Step 2's re-bind (both generations sitting on original-vintage PVs) fell out of the window and into a branch decided by which volume index happened to provision first. The advice printed there routes the operator to a step that deletes the claims Step 2 had not yet re-bound. Replace the window with the relative rule runbook Step 2a already enforces: a generation is the candidate duplicate only when EVERY one of its bound PVs is strictly newer than every bound PV of the other generation. Overlapping or tying vintages name no candidate — finish Step 2 or escalate. first_deployed is still printed, as context only. The direction decision is now a pure function (classify_mixed_direction) so the bats suite drives it directly: clean duplicate both ways, interrupted re-bind, tie, and a no-clock case pinning the WaitForFirstConsumer regression. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
601f164c5f |
fix(seaweedfs): stop the naming guard guessing which volume generation holds the data
The guard told a class-S tenant from a D tenant by comparing PVC creationTimestamps,
on the premise that "PVCs are never recreated in place, so the older generation is
where the data was born". The premise is false, and this runbook is the counter-
example: Step 2's re-bind DELETES each release-named claim and recreates it under
the chart name against the same PV.
So a tenant interrupted part-way through Step 2 — after volume-0, before volume-1 —
has data1-seaweedfs-volume-0 created SECONDS ago holding real data, beside
data1-seaweedfs-system-volume-1 with the original timestamp also holding real data.
That is the exact inversion of the rule. Reproduced against the old template:
keeps its data on the release-named volume PVCs (created 2026-06-20T07:01:59Z,
OLDER than the chart-named data1-seaweedfs-volume-* claims created
2026-07-17T09:30:00Z) ... Remove the empty chart-named set ... (Step 2a, then Step 2)
Step 2a then runs `grep -E '^data1-seaweedfs-volume' | xargs kubectl delete pvc` —
deleting the claim Step 2 had just re-bound — and Step 2 had already restored the
PV's original reclaim policy, so on any Delete-policy StorageClass the volume and
the bytes go with it. The guard's own recovery path was a data-loss instruction.
Two more holes in the same classifier:
- The S-damaged branch was gated on $systemPVC, which requires the PVC name to
contain "seaweedfs". seaweedfs.componentName truncates the fullname to 56 chars
before appending -volume, so an instance name >= ~40 chars drops the chart name
off the PVC name — the very case fullname_override_named_test.yaml exists for.
Probed with `archive-of-quarterly-financial-statements-x1`: damaged + renamed set
Ready fell through to the D-split branch ("is running two live SeaweedFS sets ...
Stop the split"), sending the operator to quiesce the set holding ALL the data —
the exact misclassification the parent commit exists to remove. Damaged + not
Ready fired no branch at all and rendered through unguarded.
- readyReplicas == 0 was read as proof the duplicate never served. It is a
snapshot: a duplicate that crashed, was scaled down, or lost readiness AFTER
serving writes reads identically, and adoption strands whatever it wrote.
There is no sound signal here. Claim timestamps are mutable, StatefulSets are
recreated by the adoption hook, and liveness is a snapshot — nothing durable in the
object graph distinguishes "empty set a bad upgrade created" from "the data,
mid-re-bind". Rather than add a fourth heuristic, the guard now decides only what
is decidable:
legacy generation only (or nothing) -> render; adoption is correct
release-named generation only -> refuse, class S (Step 2)
BOTH generations -> refuse; an operator classifies
Both generation flags now OR the PVC and StatefulSet evidence, so the truncated
long-name tenant is seen via its label-matched StatefulSet.
The cost is bounded and lands only where it should. A duplicate exists only on a
tenant that passed through the broken 4.31 rename in 1.5.x — already damaged,
already needing an operator. A 1.4.x tenant upgrading straight to 1.6 never
renames, so it has one generation and is untouched. Confirmed on the stand:
tenant-l (1.4.5-era, never upgraded) is legacy-only and dry-runs clean; the four
both-generation tenants all already need an operator. What is lost is D-wedged
auto-adoption, which was resting on the readyReplicas assumption above.
Classification moves to the runbook, where a human has signals a template does not:
Step 1 now reads the ages of the PVs (durable — they survive the re-bind) via
.spec.volumeName instead of the claims, reports MIXED instead of picking, and walks
the operator through interrupted-Step-2 first, then strict age separation, then
verifying emptiness directly, then escalating rather than guessing. Step 2 stashes
the PV's original reclaim policy ON THE PV and leaves it Retain until Step 5
verifies the tenant, so a mis-step during recovery is survivable. Step 2a grows a
precondition that refuses unless every chart-named claim is bound to a strictly
newer PV.
Tests: 7 of 14 in the enforcing suite fail against the old template, including the
interrupted-Step-2 tenant and both truncated-long-name cases. hack/seaweedfs-guard-
parity.bats pins the two copies (system/ enforcing, extra/ sibling) against silent
drift — they cannot share a library, neither chart depends on cozy-lib, and "keep
these in sync" as a comment is the failure class this whole branch is about; it
also pins that creationTimestamp and readyReplicas never come back.
`make test` in both charts now runs helm template --is-upgrade and greps for the
refusal. The fail-closed canary — the headline safety property — had ZERO automated
coverage: guard_fail_closed_test.yaml passes with AND without the fix, because
helm-unittest 1.0.3 ignores release.isUpgrade. Verified the new target fails when
the canary is removed.
Review round 3 — the refusal was right; the justification was false and the
procedure that replaced the chart's judgement was broken on both axes.
THE COMMENT WAS WRONG. It said "nothing durable in the object graph distinguishes
the empty generation from the data mid-re-bind". A durable signal does exist, and
the reviewers were right to call that out: the Helm release history. Verified on
the stand — sh.helm.release.v1.<name>-system.v1 records the scheme the tenant was
BORN under (tenant-root/l/dsplit/named rev 1 render seaweedfs-master => pre-4.31;
tenant-fresh renders seaweedfs-system-master => 1.5.x), and info.first_deployed is
on every retained revision, so anchoring PV creationTimestamps against it survives
history pruning and the Step 2 re-bind (tenant-root: chart-named +10s vs
release-named +2084s; tenant-fresh: +5225s vs +6s).
BUT IT DOES NOT UNLOCK AUTO-ADOPTION, and that is the substantive disagreement
with the round-3 direction. Birth order answers "which generation is ORIGINAL".
Adoption needs "is the OTHER one EMPTY". Those come apart exactly where it
matters: D-wedged (duplicate never scheduled, safe to adopt) and D-split
(duplicate served writes, adopting strands them) are BOTH born pre-4.31 and are
therefore identical on every durable signal. Verified: tenant-root and
tenant-dsplit have the same rev-1 scheme and the same first_deployed deltas, yet
one duplicate served and the other did not. The only thing that ever separated
them was readyReplicas, which is a snapshot and not evidence. So the render keeps
refusing on presence, and the comment now says why — naming the signal, so nobody
re-derives the broken one.
The signal IS used where it belongs, in the audit.
- naming-guard.yaml / seaweedfs.yaml: match the release-named generation by
RECONSTRUCTING the name 4.31 gives this release's volume component
(templates/_naming.tpl, replaying seaweedfs.fullname + componentName), tested
FIRST because the prefixes nest. Fixes a blocker: an instance legitimately named
`seaweedfs-volume` renders seaweedfs-volume-system-volume and
data1-seaweedfs-volume-system-volume-0, which BOTH satisfy the chart-named
prefixes — a prefix-only guard read its live storage as legacy and would render
onto the empty chart-named claims, stranding the real generation. The parity
bats passed straight through it, because parity pins sameness, not correctness.
Reconstruction also removes the name-match blind spot at source: a truncated long
instance name now matches by PVC, not only by StatefulSet.
- hack/seaweedfs-naming-audit.sh + .bats: the classifier is a tested script now.
Two revisions of it shipped as an untested snippet inside the runbook and both
were wrong in ways that routed a live tenant into the step that strands its data:
a selector `^data1-(.*seaweedfs.*)-volume` that also matched the chart-named
claims, so the release-named range spanned BOTH generations, every tenant read as
"overlapping / mid-rebind", and a genuine S-damaged tenant was routed AWAY from
Step 2a into Step 2 — which deletes its data claim, fails to recreate it, then
deletes its StatefulSets; and a `grep seaweedfs` name match that reported
"L — nothing to do" for a long-named tenant the chart refuses. Both are covered.
The audit reports L/S/MIXED matching the guard, names the original generation
from both durable signals, detects MID-REBIND (both generations at
first_deployed), and states plainly that "original" is not "empty" and that
emptiness must be established from the duplicate's volume files before deleting.
A bats case renders _naming.tpl through helm and compares it to the script's
reconstruction, so the two implementations cannot drift.
- Runbook: Step 1 calls the script. Steps 3/4 were circular — Step 3 ended in an
upgrade the presence-gated guard refuses, and Step 4 told the operator to clear
duplicates AFTER an upgrade that cannot run until they are gone. Step 3 now
removes the duplicate BEFORE the upgrade and re-runs the audit to prove it; Step 4
starts from "every tenant reads L or S". D-wedged no longer claims to need
nothing. Step 3's "enumerate fids that resolve only on the duplicate" was not an
executable procedure and is now an explicit escalation: two masters allocate
volume IDs from independent sequences into one shared seaweedfs-db, so there is
no supported tool that reconciles two volume-ID spaces against one metadata
store. Step 2's re-bind restores the cleanup hook's labels on the recreated
claims, without which a recovered tenant leaks its volumes on a later delete.
Fixture corrected: data1-...-x1-system-volume-0 is not a name the chart renders;
componentName gives ...-x1-system-seaw-volume-0.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
47e24f01d1 |
fix(platform): hand seaweedfs-db over for every instance name, not just the default
Migration 43 performs the v1.5.0 db-split hand-over: it re-owns Cluster/seaweedfs-db from <name>-system to the new <name>-db release and stamps helm.sh/resource-policy: keep, so the <name>-system upgrade — whose post-split chart no longer renders the Cluster — does not prune it as a removed resource. It compared the owning release name against the literal "seaweedfs-system". SeaweedFS is a user-creatable kind, so an instance named `foo` is owned by `foo-system` and was silently skipped: no re-own, no keep. The <name>-system upgrade then deleted the Cluster, CNPG took its PVC with it, and the filer metadata — the index for every object in that tenant's S3 — was destroyed. Data loss, not an outage: the volume PVCs still hold the bytes, nothing can find them. Verified live on the upgrade stand, which is a natural control group. All five tenants sit in the SAME failed-upgrade loop (<name>-system last-deployed is rev 1, a pre-split revision whose manifest still contains the Cluster, so every retry recomputes the same deletion): tenant-root/dsplit/l instance `seaweedfs` owner=seaweedfs-db keep Cluster alive tenant-named instance `foo` no Cluster at all Cluster GONE The only difference is that migration 43 fired for the default-named instances and skipped `foo`. That also answers whether `keep` is sufficient: three tenants survive the identical prune loop with it. Correction to the reported mechanism: this is the 1.4->1.5 db split (PR #2601, v1.5.0), not a 1.5->1.6 change — v1.5.3 and main are identical on every db-split file. And the split does protect the object; it just only protected the instances someone happened to name `seaweedfs`. - lib/seaweedfs-db-adopt.sh: shared hand-over matching the `-system` SUFFIX, so `foo-system` -> `foo-db` exactly as `seaweedfs-system` -> `seaweedfs-db`. Idempotent, and refuses a release named literally `-system` rather than annotating an owner of `-db` that no release would claim. - Migration 43 sources it: covers clusters upgrading from before 43 (1.4.x). - Migration 53 (new) re-runs it: migrations never re-run, so any cluster already at >= 44 ran the hardcoded version and is still exposed. As a pre-upgrade hook it lands before <name>-system re-renders, closing the window on THIS upgrade. This is the one that matters for 1.5.x -> 1.6. - targetVersion 53 -> 54. Tests fail on unfixed code: the non-default-name case produces no ANNOTATE at all against the hardcoded 43, while the default-name case passes — which is precisely the bug's shape. They drive the real migration scripts against a fake kubectl, following hack/migration-50-etcd-adopt.bats. LIMIT, stated plainly: a Cluster that is already deleted cannot be recovered by either migration — there is nothing left to re-annotate and the PVC went with it. Such a tenant needs seaweedfs-db restored from a backup. Runbook Step 0 adds the ownership audit that distinguishes at-risk from already-lost, and records that <name>-db can report Ready while its Cluster is gone (it rendered fine; a later <name>-system prune removed it and Flux has not re-checked) — so the audit trusts `kubectl get cluster`, not the HelmRelease status. Review round 2 — two ways this still lost the database: - OWNERSHIP IS NOT SAFETY. The helper skipped any Cluster already owned by <name>-db, treating ownership as proof the hand-over was done. It is not: where the hand-over was skipped, <name>-system prunes the Cluster and <name>-db RECREATES it under its own ownership with NO keep, while <name>-system's prune baseline still lists it — so the next reconcile deletes it again. That is the observed delete/recreate loop, and the skip walked straight past it. Verified on the stand: tenant-l and tenant-root are seaweedfs-db-owned AND their <name>-system deployed revision (rev 1) still contains the Cluster, so only keep saves them; tenant-fresh is seaweedfs-db-owned without keep but was installed after the split, so its baseline never had one. Telling those apart needs the release's deployed manifest, which this script cannot read cheaply or reliably. The costs are asymmetric — a needless keep leaves an orphan on delete (now reclaimed by the cleanup hook), a missing one loses the database — so keep is stamped on every Cluster owned by either side of the split. - FAIL OPEN, THEN STAMP. `set -euo pipefail` does not abort on a failing command substitution in a `for` word-list: the loop ran zero times, the script continued, and stamp_cozystack_version ran anyway. Migrations never re-run, so one throttle, RBAC hiccup, or not-yet-established CNPG apiservice behind the pre-upgrade hook permanently left every at-risk tenant exposed with no later migration to catch them. Reproduced: against the old helper a failing fleet scan still logged STAMP 54. Now every kubectl failure is fatal EXCEPT the two that genuinely mean "nothing to do" — the CNPG resource type not being served (the fail-open that IS load-bearing: a cluster without CNPG must still upgrade) and a Cluster vanishing between scan and read. An unreadable annotation is no longer indistinguishable from an absent one, and the unowned case now warns instead of passing silently. - ORPHAN ON DELETE. `keep` is permanent (removal deferred to the 1.7 batch migrations) and also survives the <name>-db release's own uninstall, so `kubectl delete seaweedfs <name>` left the CNPG Cluster, its Postgres pods and its PVCs behind — the cleanup hook selects app.kubernetes.io/instance=<name>-system, which the db PVCs (cnpg.io/cluster=seaweedfs-db) do not carry. Migration 53 widened that from pre-1.5 default-named tenants to every instance name, so the hook now deletes the Cluster explicitly; its PVCs carry ownerReferences to it and follow. - Tests assert the stamped VERSION (44 / 54), not a bare "STAMP": a wrong number would loop run-migrations.sh forever and the old assertion could not tell. Review round 3 — the reclaim introduced a data-loss path of its own: - DELETING ONE APP DELETED ANOTHER APP'S DATABASE. The db chart hardcodes the Cluster name `seaweedfs-db` for every instance, and the reclaim was scoped by that name in the namespace rather than by what the release owns. templates/seaweedfs-db.yaml gates the <name>-db HelmRelease on `topology != "Client"`; the cleanup hook had no such gate. Verified by render: `helm template s3-remote . --set topology=Client` emits ZERO s3-remote-db HelmReleases yet still emitted the Role granting delete on clusters/seaweedfs-db AND `kubectl delete cluster.postgresql.cnpg.io -n tenant-root seaweedfs-db`. So in a namespace running a server instance plus a Client, deleting the CLIENT destroyed the SERVER's filer metadata — CNPG takes the PVC via ownerReferences — for an app the operator never touched. Reachable the same way via a second Simple instance, whose <name>-db loses Helm's ownership check but which still installs and still carries the hook. main's hook was instance-scoped; this was new, and mine. Now gated exactly as seaweedfs-db.yaml is, and the Job re-checks meta.helm.sh/release-name == <release>-db at run time before deleting. The gate stops the permission being granted where it can never be legitimate; the runtime check stops the delete where it can. - `|| echo` turned any API/authz/admission failure into exit zero, so backoffLimit never retried and hook-delete-policy removed the only cleanup Job while the keep-protected Cluster stayed orphaned. A failed reclaim now fails the Job. - TEST SHELL. The migrations run under /bin/sh = busybox ash (the image is FROM alpine, run-migrations.sh is #!/bin/sh), but the tests drove them with bash — so the fail-closed guarantees were asserted in a shell that never runs them, which matters more than usual given the bug they cover was a fail-open. The scripts are now invoked via `sh` and the fake kubectl is POSIX sh. Verified directly against the production base image (busybox 1.37.0 / alpine:3.24): `set -o pipefail` is supported, every script passes `ash -n`, and the failing-fleet-scan path exits non-zero without stamping under real ash. The recipe is in the bats header. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
ae93356ed1 |
fix(seaweedfs): name cluster-scoped RBAC per namespace, not after the pinned fullname
ClusterRoles and ClusterRoleBindings are cluster-scoped: one object per name, cluster-wide. This chart installs once per tenant, so every cluster-scoped name it renders must be unique per install. Four of them were not. Upstream 4.31 switched them from global.serviceAccountName to seaweedfs.fullname. That is fatal in combination with the #3282 pin: fullnameOverride: seaweedfs is what adopts the running NAMESPACED workloads across the rename, so fullname is by construction the SAME string in every tenant. Every SeaweedFS instance in the fleet therefore renders seaweedfs-objectstorage-provisioner, seaweedfs-rw-cr and their bindings — one object, five claimants. NOT a #3282 regression. Rendering v1.4.5, v1.5.3 and main for the same tenants: v1.4.5 tenant-root-seaweedfs-* tenant-named-seaweedfs-* unique v1.5.3 seaweedfs-system-* foo-system-seaweedfs-* COLLIDES (identical for every default-named instance; a non-default instance name was the only thing that escaped) main seaweedfs-* seaweedfs-* COLLIDES (nothing escapes: the pin removes the last distinguishing input) So the collision shipped in 1.5.0 with the 4.31 bump; #3282 widens it from "every default-named instance" to "every instance". Observed live on the upgrade stand: one ClusterRole/seaweedfs-objectstorage-provisioner annotated to a single owning tenant, four other tenants' COSI provisioners left with no RBAC, and tenant-dsplit's helm history recording Upgrade failed: original object ClusterRole "seaweedfs-objectstorage-provisioner" not found The binding is the sharp end: its subject names a ServiceAccount in the release namespace, so the single live ClusterRoleBinding grants cluster-wide secrets RBAC to whichever tenant reconciled last, and silently revokes it from the rest. Fix: put the four names back on global.seaweedfs.serviceAccountName, which extra/seaweedfs already sets to <namespace>-seaweedfs and which cosi-provisioner-sa-name.patch already uses for the ServiceAccount and the binding's subject — the cluster-scoped names were simply left behind. Three of the four come out byte-identical to pre-4.31, so a 1.4.x tenant adopts them in place; the fourth (rw-crb) replaces upstream's username-shaped system:serviceaccount:<sa>:default and is pruned from the release's own manifest. The namespaced pin is untouched: workloads still render seaweedfs-master/-filer/ -volume, so #3282's adoption is intact. Uniqueness now rests on a value, so assert it rather than trust it: templates/cluster-scoped-rbac-guard.yaml refuses to render when the name would not carry the release namespace. It sits behind the same namespace canary as the naming guard, because system/seaweedfs's own values.yaml ships a placeholder that only extra/seaweedfs overrides — a client-side render must not trip on it. Tests fail on unfixed code: cluster_scoped_names_test asserts two tenants get two names, and unpatched both render seaweedfs-objectstorage-provisioner (5 failed, 0 passed). They test the vendored chart templates directly, which is where the rename happens. The guard includes seaweedfs.compat before reading. extra/seaweedfs sets the OLD flat key global.serviceAccountName, and the shim is what folds it into the canonical global.seaweedfs.serviceAccountName; without it the guard compares against this chart's placeholder default and refuses every real upgrade. Caught by helm upgrade --dry-run=server against the stand, not by unit tests — the mocks never carry the flat key. Verified live with --dry-run=server on the stand: tenant-l (installed pre-4.31, never upgraded) already owns tenant-l-seaweedfs-objectstorage-provisioner and tenant-l-seaweedfs-rw-cr, and the fixed chart renders exactly those — adopted in place, no ownership error. tenant-root renders tenant-root-seaweedfs-* and also dry-runs clean. Leftovers on the release-based names that a 1.5.x tenant passed through are inert once every tenant is upgraded (no release renders them any more) but are not always pruned, since more than one release claimed them; the runbook adds the sweep. Review round 2: - The uniqueness guard used `contains .Release.Namespace $sa`, a substring test that accepts namespace `tenant-a` for service account `tenant-ab-seaweedfs` — a different namespace's. Anchored to hasPrefix "<namespace>-". It still asserts the VALUE the names are built from, not the rendered names, so it cannot catch the chart being reverted to name them after the pinned fullname; cluster_scoped_names_test.yaml covers that axis and is what actually bites. - Added the case that would have caught the compat bug: every test set the CANONICAL global.seaweedfs.serviceAccountName, while production sets the OLD FLAT global.serviceAccountName that only seaweedfs.compat folds in. Deleting the include left the suite passing 5/5 while every real upgrade refused. The new case sets the flat key; with the include removed it now fails with the production error. - Runbook sweep missed the fourth pre-4.31 name: 4.31 renamed the master-rw binding from upstream's username-shaped system:serviceaccount:<sa>:default to <sa>-rw-crb, and the old one carries neither suffix nor a tenant prefix, so the grep could not see it (live on the stand as system:serviceaccount:tenant-l-seaweedfs:default). Also documents the rolling- upgrade window: Helm prunes by name without checking ownership, so the first tenant onto the fixed chart deletes shared objects a not-yet-upgraded tenant is still bound through — COSI provisioning 403s until they reconcile, S3 traffic unaffected. Review round 3: - hasPrefix "<namespace>-" was still too loose. It accepts `tenant-a-b-seaweedfs` for namespace `tenant-a` — which is ALSO the natural value for namespace `tenant-a-b`, so two namespaces would accept one string and collide on the very object this guard exists to keep unique. extra/seaweedfs sets exactly `<namespace>-seaweedfs`, so assert equality with it and nothing looser. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |