Promotion rewrote the rc version substring across the depth-2 package
values.yaml plus packages/apps/kubernetes/images/*.tag alone, on the
premise that the kubernetes app was the only one whose .tag files carry
the cozystack version. Nine other .tag files do, and system/multus seds
its ref straight into a vendored upstream manifest. v1.6.0 was therefore
staged with 33 of 55 first-party refs still reading v1.6.0-rc.4.
promote-retag.sh and nightly-mirror.sh had the same blind spot from the
other direction: both scanned only the depth-2 values.yaml, so those
images were never retagged to the stable version (30 refs selected
before, 42 after) and never mirrored to the public registry for a
nightly. A retag miss merely leaves an image short of a tag, since the
digest still resolves inside one registry; a mirror miss is worse,
because the host rewrite walks the same file list and a ref that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.
The three call sites each carried their own idea of where a ref can
live, which is how they drifted apart. Move that knowledge into
hack/lib/image-refs.sh as the single enumeration -- image_ref_files()
for the files, collect_image_refs() for the refs inside them -- and have
all three source it, so a new storage location is declared once and
reaches every consumer.
The rewrite itself moves out of promote-rc.yaml into
hack/promote-rewrite-tags.sh. Being workflow-inline is why this was only
observable by cutting a release: there was nothing to unit test. It now
also asserts its own postcondition, scanning wider than it rewrites and
failing the promotion when any ref survives in a location the
enumeration does not know about, rather than shipping it quietly.
Component-versioned images (kamaji's v0.19.0-cozystack.N) and
third-party pass-through images do not ride the cozystack version line
and are deliberately left alone; a test pins that so a future fix here
cannot over-reach. capi-providers-cpprovider is a documented known gap:
its ref is also embedded in a gzipped copy that the chart actually
ships, and rewriting only the readable copy would diverge the two.
docs/agents/image-refs.md records the contract -- three tag classes,
three storage shapes, the invariants, and which consumer does what.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address non-blocking review feedback from lexfrei on
hack/nightly-mirror.sh:81 and hack/promote-retag.sh:84.
`registry`, `repository` and `digest` are joined into a ref by string
concatenation with no type guard, so a non-string value could abort the
whole per-file expression. Because collect_refs swallows both stderr and
status, that abort silently discards every co-located owned ref — the
same per-file blast radius the `tag` guard already prevents.
The mechanism differs from `tag`'s, though, and the narrower reading is
worth recording. These keys are never fed to test(), so the "cannot
match with !!int" abort does not apply to them: yq coerces int, bool,
null and seq on `+`, and each of those merely yields a ref the
ownership filter discards. Only a !!map actually aborts, raising
"!!str () cannot be added to a !!map". The guard is therefore narrow
insurance against a nested-map typo in a vendored chart, not protection
against unquoted scalars.
Implemented as select() rather than a test(), so an offending map drops
only itself and its neighbours in the same file still resolve.
The nightly fixture gains a file pairing a map-typed `repository` with
an owned ref, asserting the latter is still mirrored; removing the
guard turns it red. Selection over the committed tree is unchanged at
30 repositories.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from lexfrei and IvanHunters on
hack/promote-retag.sh:84 and hack/nightly-mirror.sh:81.
The shape-3 rule assumed the registry host lives inside `repository`.
Two cozystack-built images do not satisfy that and were emitted
host-less, so the ownership filter dropped them — the same silent
emit-then-drop this PR fixes for the other shapes:
- keycloak-operator splits the host into a sibling `registry` key
({registry: ghcr.io, repository: cozystack/cozystack/keycloak-
operator, tag: latest@sha256:...}), and its own Makefile writes
exactly that layout.
- kubeovn keeps the host in the document-level
global.registry.address with a bare `repository: kubeovn`. It is
built and pushed to $REGISTRY by the cozystack/kubeovn-chart
wrapper, whose `make image` writes all three keys.
Neither has received a 1.x release tag: on GHCR keycloak-operator
carries only `latest`, and kubeovn's newest cozystack-versioned tag is
v1.14.11-v0.38.8, from before the 1.0 versioning switch — while cilium,
which uses an already-covered shape, has v1.5.0 through v1.6.0-rc.2.
Shapes 2 and 3 now rejoin an optional `registry` sibling, and a new
shape-4 rule handles the chart-global layout. An audit of every
@sha256: digest under packages/*/*/values.yaml against the selector
output confirms the remaining five misses are third-party or sit
outside $REGISTRY. Dry-run over the committed tree: 28 -> 30
repositories, exactly these two gained, nothing lost.
Also correct two comments that overstated coverage. The header no
longer presents its list as the shapes the build is limited to — the
list is empirical and charts are vendored — and the ownership-filter
comment no longer names kilo, kube-ovn and keycloak-operator as
third parties not pushed to $REGISTRY. All three are pushed there, and
that wrong comment is part of why the missing tags went unnoticed.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The nightly's full-suite e2e failed at install: the e2e-image-prepuller
DaemonSet never became ready, because
ghcr.io/cozystack/cozystack/linstor-csi@sha256:2f314cb9... and
.../piraeus-server@sha256:39fdacec... both 404 on GHCR. They are not there
because nightly-mirror.sh never copied them.
collect_refs handled three values shapes, and the one most package Makefiles
actually write is not among them: a split map whose digest is embedded in the
tag, {repository, tag: <tag>@sha256:<digest>}, produced by a single yq call
setting .image.tag to "$(IMAGE_TAG)@$(digest)". It matches neither the
{repository, tag, digest} rule (no `digest` key) nor, usefully, the
single-string rule -- that one matches the bare tag value, which carries no
repository, so ref_repo() reduces "main@sha256:..." to the repository "main"
and the SRC_REGISTRY case drops it.
Being dropped there is silent, and the host rewrite at the end of the script
runs over every depth-2 values.yaml regardless of what was mirrored. So the
image was never copied AND its repository was still repointed at
DST_REGISTRY, leaving a reference to an image that was never pushed there --
exactly the dangling-ref failure the comment above that rewrite warns about,
reached by a path it did not anticipate. The 404 then surfaces at pull time,
one job later, as a DaemonSet rollout timeout that names no image.
This is not limited to linstor. Checking the :nightly tag on GHCR, the two
packages using a recognised shape (cilium, cozystack-operator) are present
and eight images across six packages using this shape are absent: kamaji,
kilo, linstor-csi, piraeus-server, linstor-gui, metallb-controller,
metallb-speaker, redis-operator. The prepuller only templates
kubeovn/linstor/cert-manager, so it surfaced two of the eight; the rest would
have failed later at HelmRelease install.
Latent since the script was added in 4777273db (2026-06-26) -- the nightly
never completed a full run before today, so the mirror step had never been
exercised end to end.
The new rule type-guards `.tag` before test(). Without it a non-string tag
aborts the whole expression ("cannot match with !!int"), and since the
invocation swallows stderr and status that abort would silently discard every
shape-3 ref in the SAME file -- reintroducing this very bug in any file that
happens to carry an unquoted `tag: 1.24` next to a cozystack image. Rule 1 is
immune because its select(tag == "!!str") already precedes its test().
The test fixture is why this passed green: _make_tree() covered the
single-string, {repository,tag,digest} and platformSource shapes, a layout
only cilium actually uses, and omitted the dominant one. It now builds that
shape too, co-locates a numeric third-party tag with a cozystack shape-3 ref
to pin the type guard, and asserts the planned copy carries the REPOSITORY
rather than the bare tag -- a rule matching the tag alone would plan a copy
from "main@sha256:..." and be dropped as non-SRC_REGISTRY, so asserting only
the destination would not catch the regression.
Verified each assertion fails against the unfixed script and passes with it,
and that a realistic baked linstor tree now plans both copies.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
collect_refs scans "$TREE"/*/*/values.yaml (depth-2), but the host rewrite
used `find "$TREE" -name values.yaml` (any depth). A values.yaml nested
deeper than group/pkg would then have its image host rewritten to GHCR
without its image ever being mirrored there, leaving a dangling ref. Rewrite
over the same depth-2 glob so the rewritten set is exactly the mirrored set
(this also matches the existing --dry-run message).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/nightly-mirror.sh copies the latest main build's cozystack-owned
component images from the CI build registry (OCIR) to the public release
registry (GHCR) by digest — no rebuild — then rewrites the baked package
tree's image hosts so the GHCR install closure is self-contained. It
reuses promote-retag.sh's three-shape ref enumeration, filters to
cozystack-owned refs, and deliberately excludes the cozystack-packages
artifact (rebuilt downstream from the rewritten tree).
Dry-run bats test under the cozytest harness guards the ref selection,
the cozystack-packages exclusion, third-party filtering, and the host
rewrite plan. shellcheck clean.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Myasnikov Daniil