<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
One transient error from proxy.golang.org is enough to fail an image
build or the `Unit & controller tests` job today. The error is an HTTP/2
stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or
falls back after it. This PR adds three things: a direct fallback, a
bounded retry and a module cache.
Image builds now fall back to fetching modules from their origin when
the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a
build arg from `hack/common-envs.mk`, and every Go builder stage
declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error.
With Go's default comma it does that only on a 404 or 410. The value
comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because
buildkit can't reach a mirror on localhost or a `file://` proxy. Every
workflow job that runs Go on the runner sets the same value as env.
Explicit module downloads in Dockerfiles are retried three times with a
growing pause. The loop is inline, in one form that the contract test
pins. I tried a shared script from a named build context first, but
buildx refuses any `--build-context` for the four Dockerfiles still
pinned to the `dockerfile:1.2` frontend. The images also build from
different contexts, so no single file can be copied in. Raising those
pins changes how those images build and needs a real build to check, so
I left it for a follow-up. Two alpine stages, token-proxy and
shadowbox's `ss-server`, left the download to `go build`. The alpine
golang images have no git, so the direct fallback can't fetch for them,
and they now get the same download step. The runner jobs use
`hack/go-mod-download.sh` with the same bound.
The unit test job now starts from a module cache instead of downloading
everything. It restores `GOMODCACHE` with `actions/cache`, keyed on
every `go.sum` in the tree, and then prefetches all tracked modules
through the retry, so a cache miss still works. A cache saved by a PR
run is visible only to that PR, but one saved on the default branch can
be restored by any PR
([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)).
That is why the build cache warmer on main writes the same key before
its image build. This way the first run of a PR, which is the one that
flakes, starts warm. The warmer restores only an exact key, so old
module versions don't pile up, and its Go steps can't fail the image
build. Both runner labels are Linux, so the key matches. Both also check
out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the
same cache path, but I have not seen it on the warmer's runner yet. The
release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env
only, no cache.
`hack/go-module-fetch-contract.bats` covers all three. Each test is red
on main, and I checked each guard with a mutation.
Closes#4591
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Reliability**
- Go module downloads now fall back to direct access when the public
proxy encounters errors.
- Builds and automated checks retry failed module downloads, helping
them recover from temporary network issues.
- **Performance**
- CI caches Go modules to reduce repeated downloads and speed up builds
and tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The image build ran go build only, and go build never compiles a
_test.go, so nothing ran the tests of the code the patches change. Run
go test in the builder stage on the packages the patches touch. The list
is read from the patches themselves, so a new patch is covered without
an edit to the Dockerfile.
Each package is tested from its own directory rather than as a ./...
path of the root module, because internal/ is a module of its own and
the root module cannot name ./internal/runtime.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A transient proxy.golang.org HTTP/2 stream reset fails an image build
outright: Go's default GOPROXY falls back to direct only on a 404 or
410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a
build arg from hack/common-envs.mk, declared by every Go builder stage,
so any proxy error falls back to fetching from the origin. The value
comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may
name a mirror that buildkit cannot reach. The alpine golang images
carry no git, so in their stages a module hosted on GitHub cannot be
fetched directly and the retry is the cover.
Explicit `go mod download` steps retry three times with a growing
pause. The loop is inline, in one form the contract test pins: the
images build from different contexts, so no shared script can be
copied in, and a named build context in the shared buildx arguments
makes buildx refuse every Dockerfile pinned to a frontend older than
dockerfile:1.4.
The two alpine stages that left the download to `go build` get the
same download step, because without git neither the fallback nor a
retry covered them.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Go 1.26.4 carries standard-library advisories in net/http, net/url,
crypto/tls, encoding/asn1 and html/template that govulncheck reports
as reachable from the first-party binaries. The shipped stdlib comes
from the builder image, so the builder tags move together with the
go directive; the digests are the multi-arch indexes that golang:1.27,
1.27-alpine and 1.27-bookworm resolve to at 1.27.1.
api/apps/v1alpha1 keeps go 1.26.4. It is a published types-only
module, no binary here links its stdlib, and raising its directive
would force every consumer onto Go 1.27.
linstor-csi v1.11.2 pins grpc v1.81.0, which no longer builds on
Go 1.27: golang.org/x/net drops http2.TrailerPrefix there. Its image
now raises grpc to v1.83.2 before building, as the kamaji image does
for its advisories; the upstream unit tests pass on it.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Completes the Go-toolchain modernization for first-party images — the
build-time counterpart to #2852, which raised the root and `api/`
modules to Go 1.26.4.
1. **First-party image Go modules → go 1.26.4** (`token-proxy`,
`kube-ovn-webhook`, `kubevirt-csi-driver`): raise the `go` directive and
refresh dependencies. `kube-ovn-webhook` moves `k8s.io/*` 0.26 → 0.36.2
(only long-stable admission/core API is used); `kubevirt-csi-driver`
keeps its existing replace-based k8s alignment at 0.36.0 (main's
refactor owns that module's pin) and raises the `go` directive, with the
`golang.org/x/net` 0.56.0 security bump — which clears the reachable
GO-2026-5026 — reapplied. Adds two `.gitignore` files so the
locally-built binaries can't be committed.
2. **Go builder images → `golang:1.26`** across 8 images still on
EOL/behind toolchains (golang 1.20, 1.20.6, 1.23.4, 1.24, 1.25):
`cluster-autoscaler`, `kubevirt-cloud-provider`, `redis-operator`,
`multus-cni`, `keycloak-operator`, `objectstorage`, `kilo`,
`linstor-csi`. The builder image tag — not the go.mod minimum — sets the
standard-library version baked into the binary, so these still shipped
unpatched stdlib on a plain rebuild.
Verified: `GOOS=linux go build`/`go vet` pass for all three modules and
`go mod tidy` is a no-op; the three biggest builder jumps
(redis-operator, kubevirt-cloud-provider, cluster-autoscaler — all
formerly Go 1.20.x/1.23.x) were verified with a local `docker build`;
the remaining five (1.24/1.25 → 1.26) are covered by CI image builds.
### Release note
```release-note
chore(platform): raise the Go toolchain to 1.26 across first-party images to clear standard-library security advisories
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated several container build environments to a newer Go release for
consistency across Kubernetes-related components.
* Refreshed module and dependency versions in a few services to keep
builds current.
* Expanded ignore rules for generated image artifacts so build outputs
aren’t tracked.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
While the backend Bucket is not ready the controller returns an error to
force a requeue, but that error is only logged at klog.V(3) and is
suppressed at default verbosity, so an operator cannot see why the claim
is stuck. Record a WaitingForBucket Warning event on the BucketClaim on
each requeue (event aggregation folds the repetition) and still return
the error so the requeue happens as before. Pin the requeue contract in
the test by asserting the error carries the stable "not ready yet"
substring, not merely that it is non-nil.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The controller and the per-driver sidecar both run single-replica, so leader
election provides no high availability -- it only governs how quickly a freshly
recreated pod takes over. With the upstream 150s lease duration a recreated
provisioner pod stalls reconciliation for up to 150s while it waits out the
stale lease still held by the previous, now-deleted pod. Lower the lease timings
to the standard Kubernetes values (15s/10s/2s) so a recreated pod resumes
reconciliation in roughly 15s.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The dynamic-provisioning path re-reads the live Bucket after create and copies
its readiness onto the BucketClaim, but only during a reconcile. Nothing
re-drives that reconcile afterwards: the controller does not watch Buckets, and
the generic controller drops no-op resync deltas (reflect.DeepEqual), so once
the BucketClaim stops changing the listener is never re-invoked. The Bucket's
false->true transition, performed asynchronously by the sidecar on a different
object, is therefore reflected onto the claim only if the Bucket happened to be
ready during that single reconcile -- a race the sidecar usually loses, leaving
bucketReady frozen at false and BucketAccess never granted.
Return an error while the backend Bucket is not ready so the work queue
requeues the claim with exponential backoff; each retry re-reads the live
Bucket until its readiness has been propagated, converging deterministically
instead of depending on a resync that never fires.
Add a unit test pinning the contract (requeue while not ready, propagate true
once ready) and adjust the existing status-update conflict-retry test, which
pinned the old never-requeue behavior.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Several first-party images still built their Go binaries with EOL or
behind-latest toolchains (golang 1.20, 1.20.6, 1.23.4, 1.24, 1.25), so a
plain rebuild kept shipping standard-library versions that the current
1.26 toolchain fixes. Raise every Go builder image (FROM and the
builder_image ARG defaults) to golang:1.26 across cluster-autoscaler,
kubevirt-cloud-provider, redis-operator, multus-cni, keycloak-operator,
objectstorage, kilo and linstor-csi.
This is the build-time counterpart to raising the go directive in the
first-party Go modules: the toolchain that compiles the binary, not the
go.mod minimum, determines the standard-library version baked into the
image.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The vendored COSI controller (v0.2.2) sets only BucketClaim.status.bucketName
in the dynamic-provisioning path and hardcodes bucketReady=false, never
re-reading the Bucket afterwards. The controller does not watch Buckets, so the
readiness transition the sidecar performs (false->true) is never reflected on
the BucketClaim: bucketReady stays false forever and BucketAccess is never
granted, leaving provisioned buckets without credentials.
Re-read the live Bucket after create and propagate its readiness, so the
periodic resync converges the BucketClaim to ready. This matches the behavior
the rewritten upstream controller already has on its main branch.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Carries an upstream-bound patch against COSI v0.2.2's BucketAccess
sidecar reconciler so it survives the optimistic-concurrency race with
the central Bucket reconciler running in the same process.
In `sidecar/pkg/bucketaccess/bucketaccess_controller.go`, the `Add` path
does:
```go
bucket, _ := buckets().Get(...)
// ...
if controllerutil.AddFinalizer(bucket, BABucketFinalizer) {
buckets().Update(ctx, bucket, ...) // <-- 409 here
}
```
Between `Get` and `Update` the Bucket reconciler in the same binary can
mutate the object and bump `resourceVersion`, producing:
```
Operation cannot be fulfilled on buckets.objectstorage.k8s.io "bucket-...":
the object has been modified; please apply your changes to the latest
version and try again
```
surfaced to users as a `FailedGrantAccess` event on the BucketAccess.
This patch wraps the Bucket finalizer add in
`retry.RetryOnConflict(retry.DefaultRetry, ...)` and re-`Get`s the
Bucket inside the closure so each retry mutates the freshest version.
Patch is held as `91-bucketaccess-conflict-retry.diff` and consumed at
image build time (re-introduces `git` to the source stage and the `COPY
patches /patches` + `git apply /patches/*.diff` steps that existed
before `c29d501b` dropped them when 89/90 were upstreamed). The
convention is identical: drop the local patch the moment the upstream PR
merges and ships in a tagged release.
### Release note
~~~release-note
[objectstorage-controller] Carry a patch that retries on Bucket update
conflict during BucketAccess reconcile, eliminating the "object has been
modified" FailedGrantAccess event seen in CI.
~~~
## Test plan
- [x] `make image` for `system/objectstorage-controller` and verify
`values.yaml` picks up the new digest.
- [x] Re-run the cozystack-e2e suite that emitted the
`FailedGrantAccess: ... object has been modified` event on
`tenant-test/BucketAccess/bucket-test-admin` and confirm it no longer
fires.
- [ ] Verify `kubectl describe bucketaccess` reports
`AccessGranted=true` on first reconcile, no requeue from the conflict
path.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enhanced bucket access update reliability with improved conflict
handling during concurrent operations. The system now automatically
retries operations and refetches data to prevent update conflicts.
* **Chores**
* Improved Docker build process with automatic patch application support
for source customization.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Upstream COSI v0.2.2's BucketAccess reconciler does a Get->mutate->Update
on the parent Bucket and surfaces "Operation cannot be fulfilled ... the
object has been modified" as a FailedGrantAccess event when it races
against the Bucket reconciler in the same controller process. Wrap the
mutation in retry.RetryOnConflict so the reconcile loop refreshes and
retries instead of leaking the conflict to users.
Carried as 91-bucketaccess-conflict-retry.diff until upstreamed
(cf. 89-reconciliation.diff and 90-bucket-name.diff, both dropped
in c29d501b once merged upstream).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replace commit hash with v0.2.2 tag for better readability.
GitHub archive URLs work with tags identically.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Update COSI controller/sidecar from commit c2f6e65 to v0.2.2
(f75d475) and remove two patches that have been merged upstream:
- 89-reconciliation.diff: fix reconciliation logic for existing
and deleted objects (PR #89)
- 90-bucket-name.diff: prefix bucket names with bucket- using
UID only (PR #90)
Both patches were contributed by kvaps and included in the
upstream v0.2.2 release (2025-12-08).
Also remove git from Dockerfile dependencies as it was only
needed for applying patches.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Aleksei Sviridkin