## What this PR does
A local build that asks to load images into docker no longer pushes
them. `hack/common-envs.mk` set `PUSH` and `LOAD` with plain `:=`, and a
makefile assignment beats the environment, so `LOAD=1 PUSH=0 make image`
quietly pushed a multi-arch build. Both now take the environment value,
and an empty one falls back to the default. I also made `PUSH` default
to 0 when `LOAD=1`, so `LOAD=1` alone only loads. Before this, `make
image LOAD=1` pushed and loaded at once. An explicit `PUSH` from the
environment or the command line still wins, and `OCI_EXPORT_DIR` still
forces both off. Every buildx target gets the values through
`BUILDX_ARGS`, so the fix lives in that one file.
The Talos installer was the one image that ignored `PUSH`. It is
published with skopeo, not buildx, so it never saw `--push=$(PUSH)`. Its
registry copies now run only when `PUSH=1`.
A build that pushes nothing no longer writes `PUSHED_TAGS_LOG`. That log
tells the multi-arch stitch which tags to move, so it must hold only
tags that were really pushed.
The cache helper lost an argument nobody used. No caller passes the
second argument of `cache-args`, and an explicit tag there would bypass
`CACHE_TAG`, so the amd64 and arm64 builds of that image would share one
cache ref.
CI sets neither `PUSH` nor `LOAD`, so CI builds do the same thing as
before. Each fix has a `make --dry-run` test in
`hack/common-envs_test.bats`.
Closes#4499Closes#4504Closes#4551
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
The ccp skills pass `PUSH=1` explicitly, which still works. The website
docs do not mention `PUSH`, `LOAD` or `cache-args`.
### Release note
```release-note
fix(build): `LOAD` and `PUSH` set in the environment are now honoured, `LOAD=1` no longer pushes unless `PUSH=1` is also set, and the Talos installer is no longer published when `PUSH=0`.
```
PUSHED_TAGS_LOG fed every tag to hack/stitch-multiarch.sh whether or
not the build pushed it, so a PUSH=0 build, and now a plain LOAD=1
build, handed the stitch tags that were never published.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A plain := assignment beats an environment variable, so
`LOAD=1 PUSH=0 make image` pushed a multi-arch build instead of
loading one. An empty value from the environment falls back to the
default rather than reaching buildx as `--push=`. PUSH now also
defaults to 0 under LOAD=1, so a local load never publishes unless
PUSH=1 is asked for. The OCI_EXPORT_DIR block still overrides an
environment PUSH or LOAD.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
No caller passes the second argument any more, and an explicit tag
would bypass CACHE_TAG, so the amd64 and arm64 legs of such an image
would write the same mode=max cache ref and evict each other's layers.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A transient proxy.golang.org HTTP/2 stream reset fails an image build
outright: Go's default GOPROXY falls back to direct only on a 404 or
410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a
build arg from hack/common-envs.mk, declared by every Go builder stage,
so any proxy error falls back to fetching from the origin. The value
comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may
name a mirror that buildkit cannot reach. The alpine golang images
carry no git, so in their stages a module hosted on GitHub cannot be
fetched directly and the retry is the cover.
Explicit `go mod download` steps retry three times with a growing
pause. The loop is inline, in one form the contract test pins: the
images build from different contexts, so no shared script can be
copied in, and a named build context in the shared buildx arguments
makes buildx refuse every Dockerfile pinned to a frontend older than
dockerfile:1.4.
The two alpine stages that left the download to `go build` get the
same download step, because without git neither the fallback nor a
retry covered them.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The arm64 images are built on native arm64 runners, apart from the
amd64 build, so each build pushes a single-platform image. The stitch
joins every first-party image with its -arm64 twin into one index,
moves every tag this build pushed for it to that index, and repins
the tree on the index digest, so one ref serves both architectures.
Most refs in the tree name only the build's IMAGE_TAG, while under
PUBLISH_VERSIONED=1 the build also pushes the component version, so
the refs alone cannot say which tags to move. The image-tags macro
therefore appends every tag it pushes to PUSHED_TAGS_LOG when that is
set, and the stitch moves exactly those, plus IMAGE_TAG and the tag a
ref carries. It reads no other tag: another build's tag that sits on
the same amd64 bytes is not this build's to move, and scanning every
tag of every repository would cost thousands of registry reads per
release.
It reads refs through hack/lib/image-refs.sh like every other
consumer, rewrites only the files that enumeration names, and then
searches the whole tree outside charts/, gzip payloads included, for
any old digest it re-pointed. A ref whose digest is also pinned where
the rewrite cannot reach is skipped instead of half rewritten.
The kamaji control-plane provider's chart ships its ref only inside
files/components.gz, built from a plain copy with gzip -n, which no
text search sees. The stitch rewrites both copies, recompressing with
gzip -n so the result is what the package's image target would have
written, rather than leaving that image amd64 only.
An image with no arm64 twin (talos, the e2e sandbox), or a pair that
is not one amd64 and one arm64 image, is skipped and named. A
registry read answered with a rate limit or a 5xx is retried a
bounded number of times; any other registry error fails the run at
once rather than reading as a missing arm64 image.
On a rerun after a stitch that failed partway, the build tag can
already sit on an index of the pinned image and an arm64 image, not
necessarily this run's if the arm64 leg rebuilt it. The stitch repins
on that index instead of skipping the image, and copies it to any of
the build's other tags still left on the amd64 image. A tag moved to
anything else fails the run, since the tree then no longer matches
the registry.
MATRIX_ARCH=arm64 makes hack/build-matrix.sh leave out the e2e
sandbox, which pins linux/amd64: an arm64 leg building it would push
a second amd64 image under its -arm64 tag.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A native arm64 leg builds the same images next to the amd64 build and
must not overwrite the cache that build owns. CACHE_TAG moves the
default cache ref, so the leg can write buildcache-arm64 without every
write evicting the amd64 cache layers, and without a change to each
package Makefile. An explicit per-call cache tag still wins.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
PLATFORM defaulted to empty, so buildx built for the host alone. On an
arm64 workstation `make image` published an arm64-only digest, which
amd64 nodes then fail to pull; the cilium pin shipped exactly that
until it was rebuilt by hand. A build now targets linux/amd64 and
linux/arm64 unless PLATFORM is set, and the digest a package stamps is
the index digest.
LOAD=1 keeps building for the host only. The classic docker image store
cannot load a multi-platform index, and loading is the documented local
workflow.
This changes local pushes: a `make image` on the default docker driver
without the containerd image store now fails with "Multi-platform build
is not supported for the docker driver", where it used to build for
the host. Such a build needs a docker-container builder (BUILDER), or
LOAD=1 or an explicit PLATFORM for a single architecture.
talos and testing pin linux/amd64 with `override`, placed before the
include because common-envs.mk expands BUILDX_ARGS immediately. The
Talos assets are amd64: matchbox bakes the kernel and initramfs into
its image, and the e2e sandbox boots the disk under qemu-system-x86_64.
An arm64 variant of either would still serve or boot amd64 files.
CI pins linux/amd64 on every image build. Nothing there sets up QEMU,
fork builds export a single-platform archive, and tags.yaml builds on
the default docker driver, so the default would break or emulate those
jobs. Native arm64 builds belong in a leg of their own.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three package Makefiles spelled out their docker buildx flags instead
of using $(BUILDX_ARGS) from hack/common-envs.mk: flux-plunger,
kubeovn-plunger and both dashboard images. kubeovn-plunger landed just
before the change that moved the other Makefiles onto BUILDX_ARGS,
flux-plunger was modelled on it, and dashboard kept its own flags when
that change touched it.
Those targets ignored PLATFORM, SBOM and BUILDX_EXTRA_ARGS. On an arm64
workstation the plungers came out as arm64 images, which fail on amd64
nodes with "exec format error"; CI builds on amd64 and never saw it.
Dashboard and metallb hardcoded --platform=linux/amd64 instead; for
metallb that sat next to $(BUILDX_ARGS), so setting PLATFORM gave buildx
two platforms and a multi-platform build.
All of them now build through $(BUILDX_ARGS) alone. CI sets only
BUILDER, which these targets already honoured, and runs on amd64, so
its builds are unchanged. Local builds of dashboard and metallb now
follow the host architecture like every other package: on an arm64
machine build them for an amd64 cluster with PLATFORM=linux/amd64.
A bats test dry-runs every package image target with a probe platform
and a marker in BUILDX_EXTRA_ARGS, and requires each buildx call to
carry the marker, so $(BUILDX_ARGS) is really passed, and that
platform and no other.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Belt-and-suspenders for the B2-class trap lexfrei flagged as latent in
image-tags. PUBLISH_* are 0 on fork PRs, so today the export path never emits a
versioned/floating tag — but if it ever did:
- image-tags: multiple --tag under `--output type=oci` write >1 manifest into
the archive's index.json, and `skopeo copy oci-archive:<f>` then refuses it
("more than one image in oci"). Suppress the versioned/floating tags when
OCI_EXPORT_DIR is set (macro-level $(if …), so `make -n` prints only the
taken branch).
- packages/core/talos: gate its versioned/floating `skopeo copy docker://…`
release copies on an empty OCI_EXPORT_DIR too.
The release (non-export) path is unchanged — versioned + :latest still emitted.
Covered by a new common-envs_test.bats case that forces PUBLISH_*=1 under
export.
Reported by gemini-code-assist on #3262.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Fork PRs build images to per-image OCI archives instead of pushing,
gated on OCI_EXPORT_DIR. The `--output type=oci` was appended only inside
the `image-tags` macro, so `packages/system/capi-providers-cpprovider`,
whose `image:` recipe uses a hardcoded `--tag` and never calls
`image-tags`, ran buildx with no exporter under export mode (PUSH/LOAD
forced to 0). No archive was written, and the fork build's
`if-no-files-found: error` upload then failed the whole build. Its
`--cache-to type=inline` also needs an image exporter.
- hack/common-envs.mk: extract the OCI-output snippet from `image-tags`
into a reusable `oci-output <image-name>` macro so any buildx call can
opt in; `image-tags` now ends with `$(call oci-output,$(1))` (no-op for
the ~33 conforming packages). Move `comma :=` above its first user.
- packages/system/capi-providers-cpprovider/Makefile: add
`$(call oci-output,cluster-api-control-plane-provider-kamaji)` (archive
base name equals the OCIR repo the privileged push keys on) and drop the
inline cache under export (`--cache-to type=inline` gated off).
- .github/workflows/pull-requests.yaml: wrap the `E2E Tests` check-run
creation in try/catch. On error, `core.setFailed` so the required merge
gate fails loudly and fail-closed instead of silently succeeding.
Repo-wide audit of every `docker buildx build` reachable from the root
`build:` matrix confirms capi was the only recipe bypassing both
`image-tags` and `oci-output`.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
When OCI_EXPORT_DIR is set, every image builds to a per-image OCI archive
(<dir>/<name>.oci.tar) instead of being pushed, and PUSH/LOAD are forced
off. This lets a credential-less build (a fork PR's pull_request run)
produce the images as artifacts that a privileged workflow later pushes
to the registry by digest. The image digest is captured via
--metadata-file regardless of output type, so the values.yaml/.tag refs
baked during the build match exactly what the privileged run pushes.
The output flag is threaded through the image-tags macro, which every
package's buildx call already uses, so no per-package Makefile change is
needed.
Refs: #3257
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The full-matrix PR run validated the decomposition functionally (green
install + 21/22 app tests on the matrix-built tree), stronger evidence
than the file-set parity diff would give — and it avoids running the
serial make build the parity job exists to compare against. SBOM stays
capability-only (common-envs.mk) until an OCIR referrers probe is run
when SBOM is actually pursued.
Refs #2937
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the single serial Build job (one ephemeral VM running `make
build`, ~26 images one after another, wedging on the shared buildkit
under concurrent load — #2937) with a fan-out:
plan changed files + hack/build-matrix.sh -> scoped matrix
checks unit + controller Go tests (decoupled from e2e)
build one package per job, own docker-container buildx,
uploads a per-package digest patch fragment
build-talos talos image + nocloud disk (its own leg; e2e needs it)
finalize apply all fragments, then build the installer LAST (its
flux-push bundles the whole patched tree) and emit the
unified pr.patch e2e consumes unchanged
build-matrix.sh scopes the matrix to the packages a PR touches (full set
on shared-dependency changes) and fans seaweedfs out to
objectstorage-controller. common-envs.mk gains an SBOM=1 flag wiring
buildx --sbom, kept OFF until the OCIR referrers probe confirms support.
.github/actionlint.yaml registers the custom ephemeral runner labels.
Refs #2937
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
On release builds IMAGE_TAG = github.ref_name = vX.Y.Z, and Makefiles
that pass v$(COZYSTACK_VERSION) as the versioned arg evaluate to the
same vX.Y.Z on the tag commit. The macro then emitted
`--tag :vX.Y.Z --tag :vX.Y.Z --tag :latest`. docker buildx dedupes
internally so there is no functional impact, but the build logs carried
the redundant flag for every affected package.
Replace the empty-string guard with filter-out so the versioned tag is
also skipped when it would duplicate IMAGE_TAG.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The api/apps/v1alpha1/* subtags share the same commit as the release
tags. git describe --exact-match picks the first match alphabetically,
returning api/apps/v1alpha1/vX.Y.Z instead of vX.Y.Z, which produces
invalid Docker image tags.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
