mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-26 21:17:32 +09:00
Aleksei Sviridkin
43bf2768c2
The shared image-ref enumeration had no rule for the upstream ingress-nginx values layout, where the repository sits in `image` and the digest in its own `digest` key with no `@`. Both ingress-nginx refs in the tree (the controller and the default backend) were therefore invisible to the promote retag, the nightly mirror, the multi-arch gate and the promoted-packages verifier. The new rule requires a complete sha256 digest, so upstream's unpinned default (`digest: ""`) and digest-less image maps emit nothing. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
221 lines
13 KiB
Bash
221 lines
13 KiB
Bash
# shellcheck shell=sh
|
|
# Shared enumeration of where cozystack vendors its image references.
|
|
#
|
|
# Sourced by hack/promote-rewrite-tags.sh, hack/promote-retag.sh,
|
|
# hack/nightly-mirror.sh, hack/verify-promoted-packages.sh,
|
|
# hack/stitch-multiarch.sh and hack/verify-multiarch.sh. It exists because the
|
|
# first three call sites each grew their own idea of where a ref can live, and
|
|
# drifted: promote-retag and nightly-mirror scanned only the depth-2
|
|
# values.yaml, while the promote workflow's tag rewrite scanned those plus
|
|
# packages/apps/kubernetes/images/*.tag alone. Every ref stored in any OTHER
|
|
# images/*.tag file was therefore invisible to all three — never retagged to
|
|
# the stable version in the registry, never mirrored to GHCR for a nightly, and
|
|
# left carrying the rc version string in a promoted release tree. The verifier
|
|
# is a later consumer that has only ever read this enumeration: it compares the
|
|
# stable candidate's refs against the rc artifact's to prove promotion moved no
|
|
# container bytes, so a shape it could not see would be a proof that passed over
|
|
# the images it skipped. See docs/agents/image-refs.md for the contract this
|
|
# file implements; a new storage location must be added here and nowhere else.
|
|
#
|
|
# Three storage shapes exist, and all are first-class:
|
|
#
|
|
# 1. <root>/<group>/<pkg>/values.yaml — the YAML sub-shapes listed below
|
|
# 2. <root>/<group>/<pkg>/images/*.tag — a plain file holding one ref
|
|
# 3. an explicitly declared file (IMAGE_REF_EXTRA_FILES) whose producer sed's
|
|
# a ref straight into it because the package is not values-driven
|
|
#
|
|
# The .tag files are consumed by templates via .Files.Get rather than through
|
|
# values, which is why they are easy to forget; they are otherwise ordinary
|
|
# first-party refs and carry the cozystack version exactly like values.yaml.
|
|
#
|
|
# <root> is "packages" for the committed tree and the extracted artifact
|
|
# directory for a baked tree; both use the same <group>/<pkg> layout below it.
|
|
|
|
# Root-relative files that hold a first-party ref outside the two globs above.
|
|
# Kept as an explicit list rather than a templates/** glob: that glob would
|
|
# sweep in every vendored upstream chart template and every Helm-templated
|
|
# `image:` line, where a blind rewrite corrupts a value `make update`
|
|
# regenerates. An entry here is a deliberate statement that some package's
|
|
# `image:` target sed's a ref into a file it vendors from upstream.
|
|
#
|
|
# system/multus is the only one today: its templates/ is the upstream
|
|
# multus-daemonset-thick.yml fetched by `make update`, and multus's Makefile
|
|
# `image:` target sed's the built ref into every `multus-cni` `image:` line
|
|
# inside it.
|
|
#
|
|
# KNOWN GAP, deliberately not listed: system/capi-providers-cpprovider stamps
|
|
# its kamaji ref into files/control-plane-components.yaml AND into the
|
|
# files/components.gz built from it, and the chart ships the .gz
|
|
# (templates/configmaps.yaml reads it with .Files.Get). Declaring only the
|
|
# readable .yaml would rewrite the copy nothing consumes and leave it diverged
|
|
# from the copy that does, which is worse than leaving both alone. Nothing is
|
|
# currently owed to it: kamaji is component-versioned (v0.19.0-cozystack.N), so
|
|
# no tag rewrite is due, and the image is still retagged and mirrored via
|
|
# images/cluster-api-control-plane-provider-kamaji.tag, which carries the same
|
|
# repo@digest. The residual is that a nightly's kamaji ConfigMap keeps the
|
|
# private build-registry host, since the mirror's host rewrite cannot reach
|
|
# inside a gzip. Fixing it means decompress/rewrite/recompress with `gzip -n`
|
|
# to stay reproducible — see docs/agents/image-refs.md.
|
|
# Assigned unconditionally rather than via ${VAR:-default}: this list is
|
|
# safety-critical for the retag and the mirror, neither of which has a
|
|
# postcondition that would notice it being wrong. An inherited environment
|
|
# variable able to silently NARROW it is a hazard with no matching use case.
|
|
# Word splitting is intended (the list is whitespace-separated), which also
|
|
# means a path containing whitespace cannot be represented here.
|
|
IMAGE_REF_EXTRA_FILES="system/multus/templates/multus-daemonset-thick.yml"
|
|
|
|
# Emit every ref-bearing file under <root>, one per line.
|
|
#
|
|
# Both globs are depth-anchored rather than a `find`: a deeper values.yaml
|
|
# belongs to a vendored upstream chart (packages/*/*/charts/**) whose images
|
|
# the build neither pushes nor stamps, and rewriting one would corrupt a
|
|
# vendored default that `make update` regenerates. No .tag file lives under a
|
|
# charts/ subtree today, so the images/*.tag glob is exact rather than merely
|
|
# conservative.
|
|
image_ref_files() {
|
|
_ir_root="${1:?image_ref_files: <root> required}"
|
|
for _ir_f in "$_ir_root"/*/*/values.yaml; do
|
|
[ -f "$_ir_f" ] && printf '%s\n' "$_ir_f"
|
|
done
|
|
for _ir_f in "$_ir_root"/*/*/images/*.tag; do
|
|
[ -f "$_ir_f" ] && printf '%s\n' "$_ir_f"
|
|
done
|
|
for _ir_f in $IMAGE_REF_EXTRA_FILES; do
|
|
[ -f "$_ir_root/$_ir_f" ] && printf '%s\n' "$_ir_root/$_ir_f"
|
|
done
|
|
return 0
|
|
}
|
|
|
|
# Emit every "<repo>[:<tag>]@sha256:<digest>" ref found under <root>.
|
|
#
|
|
# The YAML shapes below are the ones present in the tree today, found by
|
|
# auditing every @sha256: digest under the depth-2 values.yaml against this
|
|
# selector's output — NOT a closed set the build is known to be limited to.
|
|
# Charts are vendored from upstream and their values layout is theirs to
|
|
# change, so treat the list as empirical and re-run that audit when a package
|
|
# is added:
|
|
# 1. single string <repo>:<tag>@sha256:<digest> (e.g. .cozystackAPI.image)
|
|
# 2. split map {[registry,] repository, tag, digest} (e.g. .cilium.image)
|
|
# 3. split map {[registry,] repository, tag: <tag>@sha256:<digest>}
|
|
# (e.g. .linstorCSI.image)
|
|
# 4. chart-global global.registry.address + global.images.<n>.{repository, tag}
|
|
# (kube-ovn's wrapper chart)
|
|
# 5. OCI artifact {platformSourceUrl: oci://<repo>, platformSourceRef: digest=sha256:<digest>}
|
|
# 6. split map {[registry,] image, [tag,] digest}
|
|
# (upstream ingress-nginx's layout, e.g. its .controller.image)
|
|
#
|
|
# The optional `registry` sibling in shapes 2/3/6 and the whole of shape 4 exist
|
|
# because the host does not always live inside the repository key. When it does not,
|
|
# the rule must rejoin it: a host-less ref reaches the caller's ownership
|
|
# filter looking third-party and is dropped, which is the silent-skip failure
|
|
# the shape-3 rule was added to fix.
|
|
#
|
|
# Shape 3 is the dominant one: most package Makefiles set `.image.tag` to
|
|
# "$(IMAGE_TAG)@$(digest)" in a single yq call instead of maintaining a
|
|
# separate `digest` key. It matches neither shape 2 (no `digest` key) nor,
|
|
# usefully, shape 1 — that rule sees only the bare tag value, which carries no
|
|
# repository, so a caller's ref_repo() reduces "<tag>@sha256:<digest>" to
|
|
# "<tag>" and the ownership filter drops it.
|
|
#
|
|
# The `tag == "!!str"` guard is load-bearing: yq's test() aborts the expression
|
|
# on a non-string tag ("cannot match with !!int"), and since these invocations
|
|
# swallow stderr and status, that abort would silently drop every shape-3 ref
|
|
# in the same file. `tag: 1.24` unquoted in a neighbouring third-party block is
|
|
# enough. Rule 1 is immune — its select(tag == "!!str") precedes its test().
|
|
#
|
|
# `registry`, `repository` and `digest` are guarded the same way against a
|
|
# different failure: they are string-concatenated rather than test()ed, and yq
|
|
# coerces most scalars on `+`, so only a !!map aborts ("!!str () cannot be
|
|
# added to a !!map") and takes the whole file's expression with it. Being a
|
|
# select() rather than a test(), the guard drops just the offending map.
|
|
# Emit every ref found in ONE file. Split out from collect_image_refs so a test
|
|
# can apply the same shape knowledge to an independently discovered file list:
|
|
# what drifted historically was which FILES each consumer looked at, not how a
|
|
# ref is spelled inside one, so an oracle wants to vary the former and hold the
|
|
# latter. Shape coverage is pinned separately by per-shape tests, which is why
|
|
# sharing this half does not weaken the suite.
|
|
#
|
|
# YAML is parsed; anything else is scraped textually. A .tag file has no YAML
|
|
# structure to speak of, and a declared extra file may be a manifest whose ref
|
|
# yq would find anyway — running both on it would merely duplicate, and the
|
|
# callers dedup.
|
|
collect_refs_from_file() {
|
|
_ir_f="${1:?collect_refs_from_file: <file> required}"
|
|
[ -f "$_ir_f" ] || return 0
|
|
|
|
case "$_ir_f" in
|
|
*.yaml|*.yml)
|
|
_collect_yaml_shapes "$_ir_f"
|
|
;;
|
|
*)
|
|
# Select on the digest rather than emitting whole lines so a blank line
|
|
# or a comment a future file may carry cannot reach the caller as a bogus
|
|
# ref. grep's no-match exit 1 is swallowed for the same reason the yq
|
|
# calls swallow theirs: an unmatched file is normal, not an error.
|
|
grep -Eo "[^[:space:]\"']+@sha256:[0-9a-f]{64}" "$_ir_f" 2>/dev/null || true
|
|
;;
|
|
esac
|
|
return 0
|
|
}
|
|
|
|
collect_image_refs() {
|
|
_ir_root="${1:?collect_image_refs: <root> required}"
|
|
image_ref_files "$_ir_root" | while IFS= read -r _ir_each; do
|
|
collect_refs_from_file "$_ir_each"
|
|
done
|
|
|
|
# Declared extras additionally get the textual scrape, even though most are
|
|
# YAML and were already parsed above. Parsing ALONE would narrow what this
|
|
# shape means: an extra is a manifest vendored from upstream, so it can carry
|
|
# a ref inside a block scalar (a ConfigMap embedding a whole deployment, say),
|
|
# where yq returns the enclosing block rather than the ref and the caller's
|
|
# ownership filter then discards it — and it can stop parsing entirely, when
|
|
# `make update` pulls a version yq chokes on or when the package patches Helm
|
|
# syntax into the vendored file. Both are silent skips, which is the failure
|
|
# mode this library exists to remove. Running both and letting the callers'
|
|
# existing dedup absorb the overlap costs nothing.
|
|
#
|
|
# The multus entry is in the second state today: its staging init container is
|
|
# wrapped in Helm conditionals, so yq refuses the whole file and this scrape is
|
|
# not a backstop for it but the only leg that reports anything. One consequence
|
|
# is worth knowing when reading the tests: the completeness oracle in
|
|
# hack/promote-rewrite-tags_test.bats builds its expected side with
|
|
# collect_refs_from_file, so for this file that comparison is inert -- it can
|
|
# neither fail nor pass on it. What pins the multus entry against the real tree
|
|
# is `image_ref_files enumerates all three storage shapes` for the path and
|
|
# `collect_image_refs finds refs the depth-2 values.yaml glob misses` for the
|
|
# ref, both in that same file.
|
|
for _ir_f in $IMAGE_REF_EXTRA_FILES; do
|
|
[ -f "$_ir_root/$_ir_f" ] || continue
|
|
grep -Eo "[^[:space:]\"']+@sha256:[0-9a-f]{64}" "$_ir_root/$_ir_f" 2>/dev/null || true
|
|
done
|
|
|
|
return 0
|
|
}
|
|
|
|
_collect_yaml_shapes() {
|
|
_ir_f="$1"
|
|
# shape 1
|
|
yq -r '.. | select(tag == "!!str") | select(test("@sha256:[0-9a-f]{64}"))' "$_ir_f" 2>/dev/null || true
|
|
# shape 2. The sub("^/"; "") is what makes `registry` optional: absent, it
|
|
# alternates to "" and leaves a leading slash on the join, stripped back to
|
|
# the bare repository the rule emitted before registry was handled.
|
|
yq -r '.. | select(tag == "!!map") | select(has("repository") and has("digest")) | select(.repository | tag == "!!str") | select(.digest | tag == "!!str") | select((.registry // "") | tag == "!!str") | (((.registry // "") + "/" + .repository) | sub("^/"; "")) + "@" + .digest' "$_ir_f" 2>/dev/null || true
|
|
# shape 3
|
|
yq -r '.. | select(tag == "!!map") | select(has("repository") and has("tag")) | select(.tag | tag == "!!str") | select(.tag | test("@sha256:[0-9a-f]{64}")) | select(.repository | tag == "!!str") | select((.registry // "") | tag == "!!str") | (((.registry // "") + "/" + .repository) | sub("^/"; "")) + "@" + (.tag | sub(".*@"; ""))' "$_ir_f" 2>/dev/null || true
|
|
# shape 4. Scoped to global.images rather than a recursive descent: the
|
|
# host is a document-level key, so binding it to a map found anywhere in
|
|
# the file would attach kube-ovn's registry to unrelated repositories.
|
|
# $reg is a yq binding, not a shell variable — the single quotes are
|
|
# required, so SC2016's "expressions don't expand" is inverted here.
|
|
# shellcheck disable=SC2016
|
|
yq -r '(.global.registry.address // "") as $reg | select($reg != "") | select($reg | tag == "!!str") | .global.images[] | select(tag == "!!map") | select(has("repository") and has("tag")) | select(.tag | tag == "!!str") | select(.tag | test("@sha256:[0-9a-f]{64}")) | select(.repository | tag == "!!str") | $reg + "/" + .repository + "@" + (.tag | sub(".*@"; ""))' "$_ir_f" 2>/dev/null || true
|
|
# shape 5
|
|
yq -r '.. | select(tag == "!!map") | select(has("platformSourceUrl") and has("platformSourceRef")) | (.platformSourceUrl | sub("^oci://"; "")) + "@" + (.platformSourceRef | sub("^digest="; ""))' "$_ir_f" 2>/dev/null || true
|
|
# shape 6. Requiring a whole digest, not just the key, is what keeps
|
|
# upstream's unpinned default (`digest: ""`) from emitting "<image>@". The
|
|
# `!!str` guard on digest has to come before test(), as in shape 3.
|
|
yq -r '.. | select(tag == "!!map") | select(.image | tag == "!!str") | select(.digest | tag == "!!str") | select(.digest | test("^sha256:[0-9a-f]{64}$")) | select((.registry // "") | tag == "!!str") | (((.registry // "") + "/" + .image) | sub("^/"; "")) + "@" + .digest' "$_ir_f" 2>/dev/null || true
|
|
return 0
|
|
}
|