The image target stamped only `tag`, leaving the committed
`registry: ghcr.io` in place, so a build against the CI registry would
render the public host with a digest that exists only in the build
registry. Stamping the build host into `registry` instead would split
the source registry across two keys, where the nightly mirror's
substring host rewrite cannot reach it and the published tree would
keep pointing at the private registry.
Set `registry: ""` and keep the whole host in `repository`, stamped
by the image target. The upstream chart joins the two keys, so the
rendered image string is unchanged, and the ref now has the same
shape as every other stamped reference the mirror and promotion
already handle.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Without --raw, skopeo inspect resolves a multi-arch index to the child
manifest for the host platform and reports that digest. The tree pins
the index digest, so once main publishes indexes every mirrored image
would fail its own post-copy verification. Hash the raw manifest
instead, which is the digest the tree pins for an index and a single
manifest alike.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The --dry-run line advertised `sed -i 's|...|g'`, an idiom the real path
stopped using once the edit moved through a temp file, and it printed
only the first of the two expressions the rewrite applies -- understating
what the run touches and re-describing a command the script no longer
runs. Report the rewrite by intent instead, covering both the `<host>/`
prefix and the bare-host scalar. Its test asserted the old `s|...|g`
string, so it encoded the same stale idiom; retarget it at the new
message and guard against `sed -i` reappearing in the plan.
Also tighten two comments that overstate what BSD sed does: the `\?`
note claimed the quantifier merely fails to match a bare host, but
`["']\?` reads as "a quote then a literal ?" so the whole expression is
dead on macOS, matching neither shape; and the `cat`-over-temp-file note
now records that `cat` truncates before it writes, an interrupt-window
trade taken to avoid `mv` stamping mktemp's 0600 onto rewritten files.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
nightly-mirror.sh and promote-rewrite-tags.sh, and the fixture builder
in promote-rewrite-tags_test.bats, rewrite files with `sed -i`. BSD sed
reads the argument after -i as a mandatory backup suffix, so `sed -i -e`
and `sed -i -E` consume the following flag as that suffix: the edit
either errors or silently no-ops, leaving `make unit-tests` red on macOS
while CI stays green on GNU sed.
Route each edit through a temp file instead. The nightly-mirror rewrite
also relied on the GNU `\?` optional quantifier, which BSD BRE reads as a
literal `?` so a bare host never matched; replace it with the POSIX
`\{0,1\}`, equivalent on both seds.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The nightly host rewrite is a literal "<src-registry>/" substring
replace, so it only reaches a host sitting contiguously in front of the
repository. kubeovn keeps its host in global.registry.address with the
bare repository in a sibling key: no trailing slash, no match. The
image was mirrored to the public registry and the tree kept pointing at
the private build registry.
The gap was dormant while kubeovn's image came from a separate
repository, because the baked ref carried the public host and the
ownership filter skipped it. Building the image in-tree stamps the
build-registry host into that key and turns it live, so fix it here.
Add an end-of-line-anchored expression that fires only when the source
registry is the complete value; a contiguous ref continues past the
host, so the two expressions cannot both fire on one reference.
keycloak-operator splits at a different boundary and stays unfixed --
neither of its keys holds the source registry as a whole. Documented,
and dormant while no CI path rebuilds it.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Promotion rewrote the rc version substring across the depth-2 package
values.yaml plus packages/apps/kubernetes/images/*.tag alone, on the
premise that the kubernetes app was the only one whose .tag files carry
the cozystack version. Nine other .tag files do, and system/multus seds
its ref straight into a vendored upstream manifest. v1.6.0 was therefore
staged with 33 of 55 first-party refs still reading v1.6.0-rc.4.
promote-retag.sh and nightly-mirror.sh had the same blind spot from the
other direction: both scanned only the depth-2 values.yaml, so those
images were never retagged to the stable version (30 refs selected
before, 42 after) and never mirrored to the public registry for a
nightly. A retag miss merely leaves an image short of a tag, since the
digest still resolves inside one registry; a mirror miss is worse,
because the host rewrite walks the same file list and a ref that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.
The three call sites each carried their own idea of where a ref can
live, which is how they drifted apart. Move that knowledge into
hack/lib/image-refs.sh as the single enumeration -- image_ref_files()
for the files, collect_image_refs() for the refs inside them -- and have
all three source it, so a new storage location is declared once and
reaches every consumer.
The rewrite itself moves out of promote-rc.yaml into
hack/promote-rewrite-tags.sh. Being workflow-inline is why this was only
observable by cutting a release: there was nothing to unit test. It now
also asserts its own postcondition, scanning wider than it rewrites and
failing the promotion when any ref survives in a location the
enumeration does not know about, rather than shipping it quietly.
Component-versioned images (kamaji's v0.19.0-cozystack.N) and
third-party pass-through images do not ride the cozystack version line
and are deliberately left alone; a test pins that so a future fix here
cannot over-reach. capi-providers-cpprovider is a documented known gap:
its ref is also embedded in a gzipped copy that the chart actually
ships, and rewriting only the readable copy would diverge the two.
docs/agents/image-refs.md records the contract -- three tag classes,
three storage shapes, the invariants, and which consumer does what.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address non-blocking review feedback from lexfrei on
hack/nightly-mirror.sh:81 and hack/promote-retag.sh:84.
`registry`, `repository` and `digest` are joined into a ref by string
concatenation with no type guard, so a non-string value could abort the
whole per-file expression. Because collect_refs swallows both stderr and
status, that abort silently discards every co-located owned ref — the
same per-file blast radius the `tag` guard already prevents.
The mechanism differs from `tag`'s, though, and the narrower reading is
worth recording. These keys are never fed to test(), so the "cannot
match with !!int" abort does not apply to them: yq coerces int, bool,
null and seq on `+`, and each of those merely yields a ref the
ownership filter discards. Only a !!map actually aborts, raising
"!!str () cannot be added to a !!map". The guard is therefore narrow
insurance against a nested-map typo in a vendored chart, not protection
against unquoted scalars.
Implemented as select() rather than a test(), so an offending map drops
only itself and its neighbours in the same file still resolve.
The nightly fixture gains a file pairing a map-typed `repository` with
an owned ref, asserting the latter is still mirrored; removing the
guard turns it red. Selection over the committed tree is unchanged at
30 repositories.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from lexfrei and IvanHunters on
hack/promote-retag.sh:84 and hack/nightly-mirror.sh:81.
The shape-3 rule assumed the registry host lives inside `repository`.
Two cozystack-built images do not satisfy that and were emitted
host-less, so the ownership filter dropped them — the same silent
emit-then-drop this PR fixes for the other shapes:
- keycloak-operator splits the host into a sibling `registry` key
({registry: ghcr.io, repository: cozystack/cozystack/keycloak-
operator, tag: latest@sha256:...}), and its own Makefile writes
exactly that layout.
- kubeovn keeps the host in the document-level
global.registry.address with a bare `repository: kubeovn`. It is
built and pushed to $REGISTRY by the cozystack/kubeovn-chart
wrapper, whose `make image` writes all three keys.
Neither has received a 1.x release tag: on GHCR keycloak-operator
carries only `latest`, and kubeovn's newest cozystack-versioned tag is
v1.14.11-v0.38.8, from before the 1.0 versioning switch — while cilium,
which uses an already-covered shape, has v1.5.0 through v1.6.0-rc.2.
Shapes 2 and 3 now rejoin an optional `registry` sibling, and a new
shape-4 rule handles the chart-global layout. An audit of every
@sha256: digest under packages/*/*/values.yaml against the selector
output confirms the remaining five misses are third-party or sit
outside $REGISTRY. Dry-run over the committed tree: 28 -> 30
repositories, exactly these two gained, nothing lost.
Also correct two comments that overstated coverage. The header no
longer presents its list as the shapes the build is limited to — the
list is empirical and charts are vendored — and the ownership-filter
comment no longer names kilo, kube-ovn and keycloak-operator as
third parties not pushed to $REGISTRY. All three are pushed there, and
that wrong comment is part of why the missing tags went unnoticed.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The nightly's full-suite e2e failed at install: the e2e-image-prepuller
DaemonSet never became ready, because
ghcr.io/cozystack/cozystack/linstor-csi@sha256:2f314cb9... and
.../piraeus-server@sha256:39fdacec... both 404 on GHCR. They are not there
because nightly-mirror.sh never copied them.
collect_refs handled three values shapes, and the one most package Makefiles
actually write is not among them: a split map whose digest is embedded in the
tag, {repository, tag: <tag>@sha256:<digest>}, produced by a single yq call
setting .image.tag to "$(IMAGE_TAG)@$(digest)". It matches neither the
{repository, tag, digest} rule (no `digest` key) nor, usefully, the
single-string rule -- that one matches the bare tag value, which carries no
repository, so ref_repo() reduces "main@sha256:..." to the repository "main"
and the SRC_REGISTRY case drops it.
Being dropped there is silent, and the host rewrite at the end of the script
runs over every depth-2 values.yaml regardless of what was mirrored. So the
image was never copied AND its repository was still repointed at
DST_REGISTRY, leaving a reference to an image that was never pushed there --
exactly the dangling-ref failure the comment above that rewrite warns about,
reached by a path it did not anticipate. The 404 then surfaces at pull time,
one job later, as a DaemonSet rollout timeout that names no image.
This is not limited to linstor. Checking the :nightly tag on GHCR, the two
packages using a recognised shape (cilium, cozystack-operator) are present
and eight images across six packages using this shape are absent: kamaji,
kilo, linstor-csi, piraeus-server, linstor-gui, metallb-controller,
metallb-speaker, redis-operator. The prepuller only templates
kubeovn/linstor/cert-manager, so it surfaced two of the eight; the rest would
have failed later at HelmRelease install.
Latent since the script was added in 4777273db (2026-06-26) -- the nightly
never completed a full run before today, so the mirror step had never been
exercised end to end.
The new rule type-guards `.tag` before test(). Without it a non-string tag
aborts the whole expression ("cannot match with !!int"), and since the
invocation swallows stderr and status that abort would silently discard every
shape-3 ref in the SAME file -- reintroducing this very bug in any file that
happens to carry an unquoted `tag: 1.24` next to a cozystack image. Rule 1 is
immune because its select(tag == "!!str") already precedes its test().
The test fixture is why this passed green: _make_tree() covered the
single-string, {repository,tag,digest} and platformSource shapes, a layout
only cilium actually uses, and omitted the dominant one. It now builds that
shape too, co-locates a numeric third-party tag with a cozystack shape-3 ref
to pin the type guard, and asserts the planned copy carries the REPOSITORY
rather than the bare tag -- a rule matching the tag alone would plan a copy
from "main@sha256:..." and be dropped as non-SRC_REGISTRY, so asserting only
the destination would not catch the regression.
Verified each assertion fails against the unfixed script and passes with it,
and that a realistic baked linstor tree now plans both copies.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
collect_refs scans "$TREE"/*/*/values.yaml (depth-2), but the host rewrite
used `find "$TREE" -name values.yaml` (any depth). A values.yaml nested
deeper than group/pkg would then have its image host rewritten to GHCR
without its image ever being mirrored there, leaving a dangling ref. Rewrite
over the same depth-2 glob so the rewritten set is exactly the mirrored set
(this also matches the existing --dry-run message).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/nightly-mirror.sh copies the latest main build's cozystack-owned
component images from the CI build registry (OCIR) to the public release
registry (GHCR) by digest — no rebuild — then rewrites the baked package
tree's image hosts so the GHCR install closure is self-contained. It
reuses promote-retag.sh's three-shape ref enumeration, filters to
cozystack-owned refs, and deliberately excludes the cozystack-packages
artifact (rebuilt downstream from the rewritten tree).
Dry-run bats test under the cozytest harness guards the ref selection,
the cozystack-packages exclusion, third-party filtering, and the host
rewrite plan. shellcheck clean.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Aleksei Sviridkin