Commit Graph
16 Commits
Author SHA1 Message Date
Aleksei Sviridkin 2af8b0c229 ci(build): fall back to direct, retry and cache Go module downloads (#4660)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

One transient error from proxy.golang.org is enough to fail an image
build or the `Unit & controller tests` job today. The error is an HTTP/2
stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or
falls back after it. This PR adds three things: a direct fallback, a
bounded retry and a module cache.

Image builds now fall back to fetching modules from their origin when
the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a
build arg from `hack/common-envs.mk`, and every Go builder stage
declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error.
With Go's default comma it does that only on a 404 or 410. The value
comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because
buildkit can't reach a mirror on localhost or a `file://` proxy. Every
workflow job that runs Go on the runner sets the same value as env.

Explicit module downloads in Dockerfiles are retried three times with a
growing pause. The loop is inline, in one form that the contract test
pins. I tried a shared script from a named build context first, but
buildx refuses any `--build-context` for the four Dockerfiles still
pinned to the `dockerfile:1.2` frontend. The images also build from
different contexts, so no single file can be copied in. Raising those
pins changes how those images build and needs a real build to check, so
I left it for a follow-up. Two alpine stages, token-proxy and
shadowbox's `ss-server`, left the download to `go build`. The alpine
golang images have no git, so the direct fallback can't fetch for them,
and they now get the same download step. The runner jobs use
`hack/go-mod-download.sh` with the same bound.

The unit test job now starts from a module cache instead of downloading
everything. It restores `GOMODCACHE` with `actions/cache`, keyed on
every `go.sum` in the tree, and then prefetches all tracked modules
through the retry, so a cache miss still works. A cache saved by a PR
run is visible only to that PR, but one saved on the default branch can
be restored by any PR
([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)).
That is why the build cache warmer on main writes the same key before
its image build. This way the first run of a PR, which is the one that
flakes, starts warm. The warmer restores only an exact key, so old
module versions don't pile up, and its Go steps can't fail the image
build. Both runner labels are Linux, so the key matches. Both also check
out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the
same cache path, but I have not seen it on the warmer's runner yet. The
release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env
only, no cache.

`hack/go-module-fetch-contract.bats` covers all three. Each test is red
on main, and I checked each guard with a mutation.

Closes #4591

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Reliability**
- Go module downloads now fall back to direct access when the public
proxy encounters errors.
- Builds and automated checks retry failed module downloads, helping
them recover from temporary network issues.
- **Performance**
- CI caches Go modules to reduce repeated downloads and speed up builds
and tests.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:47 +03:00
Aleksei Sviridkin 6df5d076b1 ci(workflows): pin flux, yq and helm-unittest installs
Every flux install fetched fluxcd.io/install.sh and ran it as root, so
whatever that host served at the moment executed on runners that later
hold registry credentials. Use the upstream setup action instead, pinned
by commit and given an explicit version: it checks the release tarball
against that release's checksums file and runs no script fetched at run
time.

nightly and the fork e2e lane still took yq from the floating latest
release. They now use the pinned, checksum-verified install the
promotion steps already have.

The helm-unittest plugin embeds its own Helm and was installed at
whatever version was newest, so a plugin release could change chart
test results on a PR that touched no chart. Pin it, and replace a
different version a persistent runner may carry.

A contract test scans every workflow for these three tools, so an
unpinned flux, yq or helm-unittest install copied from an older step
fails the unit lane.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 11:02:18 +03:00
Aleksei Sviridkin 5eee22fe28 ci(workflows): cache Go modules and fall back to direct in Go jobs
The unit and controller test job has no module cache and runs on an
ephemeral runner, so every run fetches every module from
proxy.golang.org, and one transient stream reset fails the job.

It now restores the module cache keyed on every go.sum in the tree and
prefetches the modules of every tracked go.mod through a bounded retry,
hack/go-mod-download.sh. The cache is written from main by the build
cache warmer under the same key, because a cache saved by a PR run is
visible to that PR only, while one saved on the default branch is
readable from every PR.

Every job that runs Go on the runner sets
GOPROXY=https://proxy.golang.org|direct so any proxy error falls back
to the origin. The release jobs get only that, not a cache.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 19:37:00 +03:00
Aleksei Sviridkin 6d759cc73e feat(build): build package images for amd64 and arm64 (#4498)
## What this PR does

Package images can now be built for amd64 and arm64 in one `make image`,
and a published build is multi-arch unless `PLATFORM` says otherwise.
This is the build half of arm64 support from #1961 and #903. CI still
publishes amd64 only until it gets a native arm64 leg, which comes in a
separate PR.

Builder stages now run on the build platform and compile for the target
one, so Go never runs under emulation. Under emulation it is slow and it
crashes: the migration-controller Dockerfile already records a "bad
pointer in Go heap" from exactly that. On top of this, `PLATFORM`
defaults to `linux/amd64,linux/arm64`. A build on an arm64 workstation
can't publish an arm64-only digest any more. That happened once already,
the cilium pin was arm64-only until 3f36a1b45 rebuilt it by hand.

Two images were only right because they were emulated. On a native
builder they would ship the wrong architecture. token-proxy ran `go
build` with no GOARCH, and kube-ovn calls upstream `make build-go`,
which hardcodes `GOARCH=amd64`. Both are fixed now. The new bats test
wants every compiling stage on `$BUILDPLATFORM` and actually using
`TARGETARCH`. A bare `ARG TARGETARCH` does not count, this is how
token-proxy got through.

Some things stay amd64. talos and testing pin it with `override`,
because matchbox serves amd64 Talos assets and the sandbox runs
`qemu-system-x86_64`. `LOAD=1` builds for the host only, since the
classic docker image store can't load an index. Every CI build job pins
`PLATFORM: linux/amd64`, so CI output does not change.

This changes local builds. A `make image` on the default docker driver
with the classic image store now fails with "Multi-platform build is not
supported for the docker driver", where before it built for the host.
Use a `docker-container` builder (`BUILDER=<name>`), or pass
`PLATFORM=linux/amd64` (or `LOAD=1`) to build one architecture.

I checked it on an arm64 host with a docker-container builder.
cozystack-api, dashboard, kubeovn, platform-migrations, cilium and
capi-providers-cpprovider build and push as amd64+arm64 indexes.
Binaries pulled from each platform are x86-64 and aarch64:
cozystack-api, token-proxy, the kamaji provider manager, cilium-agent
and all four kube-ovn binaries. piraeus-server builds for both
platforms, but the push to my test registry failed on registry 500s. The
capi-providers-cpprovider inline cache works with a multi-platform push.
Not covered is running the arm64 images on an arm64 cluster, CI has
none.

This is stacked on #4485 and targets its branch, so the diff here is
only the two commits on top.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#711
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(build): package images build for linux/amd64 and linux/arm64 by default, with Go builders cross-compiling on the build platform; a local `make image` now needs a docker-container builder, or PLATFORM=linux/amd64 / LOAD=1 for a single architecture. CI still publishes amd64.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Build Improvements**
* Container image builds now target both amd64 and arm64 by default when
building without loading images locally.
* CI and release builds are pinned to amd64; Talos and testing images
remain amd64-only.
* Cross-platform builds now compile on the builder’s native platform
while targeting the requested architecture.
* **Documentation**
* Clarified platform defaults and architecture requirements for local
and CI builds.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 19:43:47 +03:00
Aleksei Sviridkin 084ac47237 feat(build): build published images for amd64 and arm64 by default
PLATFORM defaulted to empty, so buildx built for the host alone. On an
arm64 workstation `make image` published an arm64-only digest, which
amd64 nodes then fail to pull; the cilium pin shipped exactly that
until it was rebuilt by hand. A build now targets linux/amd64 and
linux/arm64 unless PLATFORM is set, and the digest a package stamps is
the index digest.

LOAD=1 keeps building for the host only. The classic docker image store
cannot load a multi-platform index, and loading is the documented local
workflow.

This changes local pushes: a `make image` on the default docker driver
without the containerd image store now fails with "Multi-platform build
is not supported for the docker driver", where it used to build for
the host. Such a build needs a docker-container builder (BUILDER), or
LOAD=1 or an explicit PLATFORM for a single architecture.

talos and testing pin linux/amd64 with `override`, placed before the
include because common-envs.mk expands BUILDX_ARGS immediately. The
Talos assets are amd64: matchbox bakes the kernel and initramfs into
its image, and the e2e sandbox boots the disk under qemu-system-x86_64.
An arm64 variant of either would still serve or boot amd64 files.

CI pins linux/amd64 on every image build. Nothing there sets up QEMU,
fork builds export a single-platform archive, and tags.yaml builds on
the default docker driver, so the default would break or emulate those
jobs. Native arm64 builds belong in a leg of their own.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:14:12 +03:00
IvanHunters 95e68bad86 ci(workflows): use a literal match in the flux version guards
The guard ran `grep -qx "flux: v${FLUX_VERSION}"`, where the dots in the
version are regex metacharacters, so a client printing `flux: v2X8X6`
would satisfy it. Use `grep -Fqx` for a fixed-string match in the build
toolchain guards this change added.

Also correct the step comments that still read "install if absent
(idempotent)": the step now installs whenever the pinned version is not
the one present, not only when flux is missing.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:58 +03:00
IvanHunters edbc4f1b3e ci(workflows): pin flux version in build toolchain steps
The "Set up build toolchain" steps installed flux with a bare
`curl https://fluxcd.io/install.sh | sudo bash`, letting the installer
resolve its floating "latest". latest had advanced to v2.9.5, and a
matrix job on PR #4429 failed when that release's checksums transiently
4xx'd during install.

Pin FLUX_VERSION to 2.8.6 and guard on the installed client version, the
same shape the "Set up promotion toolchain" steps already use, so the
flux CLI is fixed and reproducible across CI instead of following latest.
This does not remove the checksum host's transient errors; it removes the
uncontrolled version bump. Related to the yq floating-latest pin in #3640.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:47 +03:00
renovate[bot] a7dfbe6f8c chore(deps): update container image & action digests
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:04 +03:00
myasnikovdaniil 48a3cfd341 Merge branch 'main' into ci/scorecard-hardening
Signed-off-by: myasnikovdaniil <60174387+myasnikovdaniil@users.noreply.github.com>
2026-07-07 16:49:29 +05:00
Andrei Kvapil 75452f36a3 ci(actions): pin third-party GitHub Actions to commit SHAs
Pin the remaining floating version tags in the build-main and ui-test
workflows to immutable commit SHAs, preserving the original tag in a
trailing comment so Renovate can keep the pins up to date.

Supply-chain hardening against tag hijacking/mutation; no functional
change — each SHA is the commit the tag currently resolves to.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-07 11:28:53 +02:00
Myasnikov DaniilandClaude Opus 4.8 a0bd53f84d ci: pin workflow actions, scope token permissions, add zizmor gate
build-main.yaml and ui-test.yaml (the latter added by the in-tree console
move) landed without following the repo's workflow conventions: no top-level
permissions block and actions referenced by moving tags. Together they zero out
two OpenSSF Scorecard checks the rest of the tree already satisfies:

  - Token-Permissions: scored on the weakest workflow, so one workflow with an
    undeclared (default-write) top-level token drops the whole check to 0.
  - Pinned-Dependencies: the six tag-pinned action refs count as unpinned.

Fix both files (top-level read-only token; job-level packages:write kept where
needed; all action refs pinned to the SHA already used elsewhere in the tree)
and add a durable gate so it cannot recur:

  - .github/zizmor.yml enables the two Scorecard-relevant audits (unpinned-uses,
    excessive-permissions), clean tree-wide as of this commit. The remaining
    audits carry pre-existing debt and are disabled with a TODO to burn down and
    re-enable one at a time.
  - .pre-commit-config.yaml runs zizmor (offline) on changed workflows.
  - .github/workflows/zizmor.yml runs the same audit tree-wide on every PR that
    touches a workflow, failing on any reintroduced regression.

Also wire scorecard.yml to an optional SCORECARD_TOKEN secret: the default
GITHUB_TOKEN cannot read branch-protection settings, so the Branch-Protection
check is scored on partial data. The token is a no-op until the secret is added.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 11:31:13 +05:00
Myasnikov Daniil 1ebf7f15a4 ci: run selector bats in checks, harden flux install, 3-way fragment apply
- Run hack/build-matrix_test.bats and hack/select-e2e_test.bats in the checks
  job so a selector parse regression is caught in CI rather than shipping.
- curl -fsSL the flux install so an HTTP error page is not piped to sudo bash.
- git apply --3way in finalize so two fragments touching adjacent lines of a
  shared file merge instead of the second apply aborting the whole step.
- Add least-privilege permissions: {contents: read} to the plan and checks jobs.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-25 10:21:06 +05:00
Myasnikov DaniilandClaude Opus 4.8 3a653e087b ci(build): move the cache warmer off the self-hosted runner
build-main.yaml now warms the mode=max cache on the ephemeral pool
(oracle-vm-24cpu) instead of the dedicated self-hosted runner, so no
build path depends on that host (#2937 decommission). Adds an idempotent
flux install for the installer's image-packages step and bumps the
timeout for the serial build on the ephemeral shape.

The self-hosted runner remains referenced only by the `debug`-label
breakpoint path in e2e; fully de-registering it is a separate infra step.

Refs #2937

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-22 20:08:52 +05:00
Myasnikov DaniilandClaude Opus 4.8 e7abea9576 fix(ci): create buildx builder in the build step's DOCKER_CONFIG
The "Set up Buildx" step in build-main.yaml ran without DOCKER_CONFIG, so
docker/setup-buildx-action created the docker-container builder under the
default ~/.docker. The build step runs with DOCKER_CONFIG=$RUNNER_TEMP/.docker
(seeded before the builder existed and holding the OCIR auth), so it could not
find the builder and failed immediately with `no builder "builder-<uuid>"
found`. The post-job `docker buildx rm` still succeeded because the action's
cleanup runs under the default config where the builder actually lived.

Set DOCKER_CONFIG on the Set up Buildx step so the builder instance is created
in the same config the build step reads.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-22 11:39:36 +05:00
Myasnikov DaniilandClaude Opus 4.8 f3914d2053 ci(build): disable credential persistence in build-cache checkout
The build-main cache warmer is a privileged workflow (packages:write + an OCIR
push token) and never pushes git, so set persist-credentials: false to keep the
checkout's GITHUB_TOKEN out of .git/config where a later build step could read
it. (SHA-pinning the actions is handled repo-wide in #2849.)

Addresses non-blocking follow-up 7 on #2948.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-20 16:35:50 +05:00
Myasnikov DaniilandClaude Opus 4.8 d3958c5bcb refactor(build): mode=max registry cache with a main-only warmer (#2938)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 10:40:05 +05:00