Files
cozystack/.github/workflows/build-main.yaml
Aleksei Sviridkin 2af8b0c229 ci(build): fall back to direct, retry and cache Go module downloads (#4660)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

One transient error from proxy.golang.org is enough to fail an image
build or the `Unit & controller tests` job today. The error is an HTTP/2
stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or
falls back after it. This PR adds three things: a direct fallback, a
bounded retry and a module cache.

Image builds now fall back to fetching modules from their origin when
the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a
build arg from `hack/common-envs.mk`, and every Go builder stage
declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error.
With Go's default comma it does that only on a 404 or 410. The value
comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because
buildkit can't reach a mirror on localhost or a `file://` proxy. Every
workflow job that runs Go on the runner sets the same value as env.

Explicit module downloads in Dockerfiles are retried three times with a
growing pause. The loop is inline, in one form that the contract test
pins. I tried a shared script from a named build context first, but
buildx refuses any `--build-context` for the four Dockerfiles still
pinned to the `dockerfile:1.2` frontend. The images also build from
different contexts, so no single file can be copied in. Raising those
pins changes how those images build and needs a real build to check, so
I left it for a follow-up. Two alpine stages, token-proxy and
shadowbox's `ss-server`, left the download to `go build`. The alpine
golang images have no git, so the direct fallback can't fetch for them,
and they now get the same download step. The runner jobs use
`hack/go-mod-download.sh` with the same bound.

The unit test job now starts from a module cache instead of downloading
everything. It restores `GOMODCACHE` with `actions/cache`, keyed on
every `go.sum` in the tree, and then prefetches all tracked modules
through the retry, so a cache miss still works. A cache saved by a PR
run is visible only to that PR, but one saved on the default branch can
be restored by any PR
([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)).
That is why the build cache warmer on main writes the same key before
its image build. This way the first run of a PR, which is the one that
flakes, starts warm. The warmer restores only an exact key, so old
module versions don't pile up, and its Go steps can't fail the image
build. Both runner labels are Linux, so the key matches. Both also check
out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the
same cache path, but I have not seen it on the warmer's runner yet. The
release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env
only, no cache.

`hack/go-module-fetch-contract.bats` covers all three. Each test is red
on main, and I checked each guard with a mutation.

Closes #4591

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Reliability**
- Go module downloads now fall back to direct access when the public
proxy encounters errors.
- Builds and automated checks retry failed module downloads, helping
them recover from temporary network issues.
- **Performance**
- CI caches Go modules to reduce repeated downloads and speed up builds
and tests.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:47 +03:00

156 lines
6.6 KiB
YAML

name: Build cache (main)
# Warms the shared mode=max build cache so PR builds start hot.
#
# PR builds (pull-requests.yaml) read the cache but never write it
# (WRITE_CACHE unset -> 0). Only this serialized main build writes
# CACHE_REGISTRY/<img>:buildcache, so concurrent PR builds never race on the
# cache manifest -- the 409 collisions PR #2711 fixed for image tags would
# otherwise reappear on the cache ref. Cache is mode=max so all multistage
# `builder` stages are cached, not just the final stage as with inline.
#
# This job builds on a docker-container buildx builder, NOT the host's default
# embedded `docker` driver: `--cache-to type=registry,mode=max` is unsupported
# on the docker driver unless the daemon runs the containerd image store, so on
# a classic store the warmer would otherwise fail to export any cache. The
# container driver also gives this build its own buildkit (separate bbolt), so
# the warmer never contends with PR builds on the shared embedded buildkit.
env:
# Same per-CI registry as pull-requests.yaml; the cache is co-located here.
REGISTRY: iad.ocir.io/idyksih5sir9/cozystack
on:
push:
branches: [main]
paths-ignore:
- 'docs/**'
# Only the newest main commit's cache matters; cancel older in-flight warmers.
concurrency:
group: build-main-cache
cancel-in-progress: true
# Top-level token defaults to read-only; the warm-cache job below narrows to the
# packages:write scope it actually needs to push the build cache.
permissions:
contents: read
jobs:
warm-cache:
name: Warm build cache
# Runs on the ephemeral pool, not the dedicated self-hosted runner, so the
# build path no longer depends on that host (#2937 decommission). Serial
# `make build` is CPU-bound, so use the large shape; the warmer is a
# background main-push job, latency is not critical.
runs-on: oracle-vm-24cpu-96gb-x86-64
timeout-minutes: 120
env:
# The pipe falls back to direct on any proxy error (hack/common-envs.mk).
GOPROXY: "https://proxy.golang.org|direct"
permissions:
contents: read
packages: write
steps:
- name: Checkout code
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
fetch-depth: 0
fetch-tags: true
# This privileged workflow has packages:write and an OCIR push token;
# it never pushes git, so don't leave the checkout's GITHUB_TOKEN in
# .git/config where a later build step could read it.
persist-credentials: false
# Writes the Go module cache pull-requests.yaml's unit test job restores:
# an entry saved on main is readable from every PR, while one a PR run
# saves is visible to that PR alone. It is saved before the image build so
# a cancelled or failed build still leaves it, and none of these steps may
# stop the build, which does not use it. Only an exact key is restored: a
# partial match saved under the new key would carry every superseded
# module version forward forever.
- name: Locate Go module cache
id: gomodcache
continue-on-error: true
run: echo "dir=$(go env GOMODCACHE)" >> "$GITHUB_OUTPUT"
- name: Restore Go module cache
id: gomod-restore
continue-on-error: true
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.gomodcache.outputs.dir }}
key: go-mod-${{ runner.os }}-${{ hashFiles('**/go.sum') }}
- name: Download Go modules
id: gomod-download
if: steps.gomod-restore.outputs.cache-hit != 'true'
continue-on-error: true
run: |
for mod in $(git ls-files go.mod '*/go.mod' | xargs -n1 dirname); do
(cd "$mod" && "$GITHUB_WORKSPACE/hack/go-mod-download.sh")
done
- name: Save Go module cache
if: steps.gomod-download.outcome == 'success'
continue-on-error: true
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.gomodcache.outputs.dir }}
key: go-mod-${{ runner.os }}-${{ hashFiles('**/go.sum') }}
# Ephemeral runners lack the flux CLI the installer's image-packages step
# shells out to.
- name: Set up flux
uses: fluxcd/flux2/action@04acaec6161ac4fb1a82ffafa88901c03271d34f # v2.8.6
with:
version: "2.8.6"
- name: Set up Docker config
run: |
if [ -d ~/.docker ]; then
cp -r ~/.docker "${{ runner.temp }}/.docker"
fi
- name: Login to OCIR
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ secrets.OCIR_USER }}
password: ${{ secrets.OCIR_TOKEN }}
registry: iad.ocir.io
env:
DOCKER_CONFIG: ${{ runner.temp }}/.docker
# docker-container driver: required for `--cache-to type=registry,mode=max`
# (unsupported on the host's default docker driver) and gives this build an
# isolated buildkit so it never contends with PR builds on the shared one.
- name: Set up Buildx (docker-container driver)
id: buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
with:
driver: docker-container
env:
# buildx stores builder instances under $DOCKER_CONFIG/buildx/instances/.
# Create the builder in the SAME config the build step reads
# (runner.temp/.docker, seeded above and holding the OCIR auth); without
# this the builder lands in the default ~/.docker and the build step fails
# with `no builder "<name>" found`.
DOCKER_CONFIG: ${{ runner.temp }}/.docker
- name: Build all images and write mode=max cache
run: make build
env:
DOCKER_CONFIG: ${{ runner.temp }}/.docker
# Build on the docker-container builder created above (BUILDX_ARGS adds
# --builder=$(BUILDER)); buildx forwards the OCIR auth from DOCKER_CONFIG
# to the container builder for cache reads/writes and image pushes.
BUILDER: ${{ steps.buildx.outputs.name }}
# common-envs.mk defaults to amd64+arm64. Nothing here sets up QEMU,
# and on this x86-64 runner the arm64 half would be emulated anyway;
# CI builds amd64 until arm64 has a native leg of its own.
PLATFORM: linux/amd64
# WRITE_CACHE=1 turns on --cache-to (mode=max) for every image.
WRITE_CACHE: '1'
# Publish a floating :main handle; do not move :latest (releases do).
IMAGE_TAG: main
PUBLISH_VERSIONED: '0'
PUBLISH_FLOATING: '0'