From 4fcc111142a9883b01dc738a850ffbd5d775b346 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 19 Aug 2026 11:13:22 +0500 Subject: [PATCH 1/2] fix(release): compare host-less image refs on digest alone hack/verify-promoted-packages.sh reported "promotion changed the container repository/digest set" for a v1.6.2 promotion that moved no container bytes at all: all 48 real repo@digest refs and all 48 digests were byte-identical between the rc artifact and the promotion candidate. The six entries it flagged as changed repositories were not repositories. They were bare tag strings -- v1.6.2-rc.1 on the rc side, v1.6.2 on the candidate side -- pairing one-to-one by identical digest. hack/lib/image-refs.sh emits two entries per shape-3 split map: the correctly joined repository@digest, plus shape 1's recursive scrape of the raw .tag scalar, which carries no repository. The library documents that degeneracy and leaves host-less refs to its callers; promote-retag.sh drops them, the verifier had only a */cozystack-packages exclusion. So `${image%:*}` -- a no-op on a string holding no colon -- handed the tag over as the repository, and promotion rewrites exactly those tags. Set an empty repository for a ref with no `/` ahead of its digest, so it compares on the digest, which is the whole container identity such a ref carries and the only part promotion must not move. Dropping these refs instead would be wrong rather than blunter: packages/system/kuberture carries an `image:` map with a `tag:` and no `repository:`, so shape 1 is the only rule that sees its digest, and skipping it would silently stop proving that digest unchanged. Reproduced over the release-1.6 packages tree with the shape-3 tags rewritten rc-to-stable: before, the six pairs above diverge; after, both sides normalize to 60 identical entries, kuberture's digest included and kube-ovn's real repository still visible through its shape-4 join. Signed-off-by: Myasnikov Daniil --- hack/verify-promoted-packages.sh | 39 +++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/hack/verify-promoted-packages.sh b/hack/verify-promoted-packages.sh index bbd625ff3..26089e418 100755 --- a/hack/verify-promoted-packages.sh +++ b/hack/verify-promoted-packages.sh @@ -206,7 +206,44 @@ normalized_refs() { digest="${raw##*@}" image="${without_digest##*/}" if [ "$without_digest" = "$image" ]; then - repo="${image%:*}" + # No `/` anywhere ahead of the digest, so nothing in this ref names a + # registry host or a repository path — it is a bare scalar sitting in + # front of a digest, and the collector emits those by design. Shape 3 + # (`repository:` on one key, `tag: @sha256:` on another) is + # the dominant shape in the tree, and shape 1's recursive descent sees + # that `tag` scalar on its own, so every shape-3 map arrives here twice: + # once correctly joined, once as the bare tag. hack/lib/image-refs.sh + # documents that degeneracy and leaves host-less refs to its callers — + # promote-retag.sh drops them through its ownership filter, and this + # function had no equivalent. `${image%:*}` is then a no-op (a tag holds + # no colon) and the TAG arrives as the repository. Promotion rewrites + # exactly those tags, which is how a promotion that moved no container + # bytes at all reported six changed "repositories" whose digests were + # identical on both sides. + # + # Compare such a ref on its digest alone. The digest is the whole of the + # container identity a host-less ref carries, and it is the only part + # promotion must not move; the empty repository yields a leading `@`, + # which no real repository name can produce, so these cannot collide with + # a host-bearing entry. Where a repository name does exist behind one of + # them — kube-ovn's `global.images.kubeovn`, whose host lives in + # `global.registry.address` — the same image also arrives host-bearing + # from shape 4, so a repository change stays visible through that entry. + # + # Dropping host-less refs outright would be wrong rather than merely + # blunter: packages/system/kuberture/values.yaml carries an `image:` map + # with a `tag:` and no `repository:`, so shape 1 is the only rule that + # ever sees its digest. Skipping it would silently stop proving that + # digest is unchanged — a hole in the very guard this branch repairs. + # + # Do NOT reach for promote-retag.sh's `${REGISTRY}/` ownership filter to + # do this job. The verify job's REGISTRY names the private build registry + # while both artifacts under comparison live on the public one, so that + # filter drops every ref; and the emptiness guard above tests the RAW + # collection rather than the filtered set, so the proof would then pass + # by comparing two empty sets — failing open on the one invariant it + # exists to establish. + repo="" else repo="${without_digest%/*}/${image%:*}" fi From 2cb8c743cb0f5371fd6271928bde26ef60ca0146 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 19 Aug 2026 11:16:29 +0500 Subject: [PATCH 2/2] test(release): cover the split-map and repository-less ref shapes No fixture in verify-promoted-packages_test.bats built a shape-3 split map -- `repository:` on one key, `tag: @sha256:` on another -- which hack/lib/image-refs.sh itself calls the dominant shape in the tree. The existing fixture's single-string ref carries a registry host, so its rc-to-stable rewrite normalizes through the other branch and passes whether or not host-less refs are handled, which is why the suite looked covered while a clean promotion could still be rejected. Add two cases. The first promotes a split map whose tag moves from rc to stable with the repository and digest held constant; reverting the normalization fix reddens it on "promotion changed the container repository/digest set", diffing v9.9.9-rc.3@sha256:eee... against v9.9.9@sha256:eee... -- the same tag-as-repository pair, with identical digests, that CI reported. The second pins the property a blunter fix would break. A digest-only `image:` map with no `repository:` (the kuberture shape) is seen by no rule but shape 1, so it must still contribute its digest: the case moves that digest and nothing else, and asserts the rejection comes from the digest set rather than a neighbouring leg. Replacing the fix with an outright `continue` over host-less refs makes the verifier exit 0 on that changed digest and reddens the case at its exit-status assertion. Negative assertions are counted rather than `!`-negated: under hack/cozytest.sh a `! grep -q` cannot fail, so it would be a comment shaped like an assertion. Signed-off-by: Myasnikov Daniil --- hack/verify-promoted-packages_test.bats | 105 ++++++++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/hack/verify-promoted-packages_test.bats b/hack/verify-promoted-packages_test.bats index 1d75f89db..1f6bad367 100644 --- a/hack/verify-promoted-packages_test.bats +++ b/hack/verify-promoted-packages_test.bats @@ -365,3 +365,108 @@ EOF [ "$rc" -ne 0 ] grep -q 'promotion changed the container repository/digest set' "$tmp/err" } + +# Shape 3 — `repository:` on one key, `tag: @sha256:` on another — +# is the shape hack/lib/image-refs.sh itself calls the dominant one, and the one +# no fixture above builds. The collector emits TWO entries for such a map: the +# correctly joined repository@digest, plus shape 1's recursive scrape of the +# bare `tag` scalar, which names no repository at all. Promotion rewrites +# exactly that scalar, so normalization keeping it — treating the tag as the +# repository — makes every clean promotion look as though it moved container +# bytes. The single-string ref in _make_verify_fixture cannot reach that path: +# it carries a registry host, so it normalizes through the other branch and its +# own rc-to-stable rewrite passes either way. That is why the suite looked +# covered while the dominant shape went untested. +_write_split_map() { + cat > "$1" < "$tmp/out" 2> "$tmp/err" || rc=$? + if [ "$rc" -ne 0 ]; then + echo "verification exited $rc" >&2 + cat "$tmp/err" >&2 + return "$rc" + fi + + grep -q "stable tags, identical package tree, and the rc artifact's container digests" "$tmp/out" +} + +# The property a blunter fix would break. packages/system/kuberture/values.yaml +# carries an `image:` map with a `tag:` and no `repository:` — shape 2 wants a +# `digest` key and shape 3 a `repository` key, so shape 1's scrape of the bare +# tag scalar is the ONLY rule that ever sees that image's digest. Discarding +# host-less refs rather than comparing them on their digest would silently stop +# proving it unchanged, which is a hole in the same guard being repaired here. +# Asserted from the rejecting direction, because that is the direction able to +# go red: the digest moves and nothing else does, so the digest-set comparison +# is the only leg that can fire — artifact and release tree stay byte-identical, +# no rc string appears in the candidate, and the installer values match. +_write_repository_less_map() { + cat > "$1" < "$tmp/out" 2> "$tmp/err" || rc=$? + + [ "$rc" -ne 0 ] + grep -q 'promotion changed the container repository/digest set' "$tmp/err" + # The rejection has to come from the digest set and not from a neighbouring leg + # rejecting a tree this test never perturbed, or it would still pass with the + # digest dropped from the comparison entirely. Counted rather than negated with + # `!`, which suppresses errexit and would pass regardless. + for _other in 'contain different files' 'differs from release tree' \ + 'still carries rc image references' 'installer values differ'; do + count="$(grep -c "$_other" "$tmp/err" || true)" + [ "${count:-0}" -eq 0 ] + done +}