A Sentinel that restarts on the same pod IP, as after a node reboot,
comes back with the operator's initial sentinel.conf, which monitors a
127.0.0.1 placeholder and carries no auth-pass. Its peers still know
that address and keep publishing hello messages to it, and once a
failover has raised the config epoch the restarted Sentinel adopts the
current master address from them before the operator looks at it. The
operator then finds the right master address and skips
NewSentinelMonitor, the only call that sends SENTINEL SET auth-pass. The
master answers that Sentinel with NOAUTH, it reports the master as
s_down, and the operator's resulting SENTINEL RESET on every check keeps
the empty auth-pass, so the instance runs with one Sentinel fewer until
someone deletes the pod.
RestoreSentinel now sends the RedisFailover's password with SENTINEL SET
auth-pass after the reset. It runs only when the Sentinel already looks
unhealthy, and the reset has just dropped its links, so a Sentinel that
had the password loses nothing. Without auth there is nothing to send.
The patch file sorts after tls.diff because it is written on top of it;
the Dockerfile applies patches/*.diff in glob order.
Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com>
Assisted-by: LLM
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
One transient error from proxy.golang.org is enough to fail an image
build or the `Unit & controller tests` job today. The error is an HTTP/2
stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or
falls back after it. This PR adds three things: a direct fallback, a
bounded retry and a module cache.
Image builds now fall back to fetching modules from their origin when
the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a
build arg from `hack/common-envs.mk`, and every Go builder stage
declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error.
With Go's default comma it does that only on a 404 or 410. The value
comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because
buildkit can't reach a mirror on localhost or a `file://` proxy. Every
workflow job that runs Go on the runner sets the same value as env.
Explicit module downloads in Dockerfiles are retried three times with a
growing pause. The loop is inline, in one form that the contract test
pins. I tried a shared script from a named build context first, but
buildx refuses any `--build-context` for the four Dockerfiles still
pinned to the `dockerfile:1.2` frontend. The images also build from
different contexts, so no single file can be copied in. Raising those
pins changes how those images build and needs a real build to check, so
I left it for a follow-up. Two alpine stages, token-proxy and
shadowbox's `ss-server`, left the download to `go build`. The alpine
golang images have no git, so the direct fallback can't fetch for them,
and they now get the same download step. The runner jobs use
`hack/go-mod-download.sh` with the same bound.
The unit test job now starts from a module cache instead of downloading
everything. It restores `GOMODCACHE` with `actions/cache`, keyed on
every `go.sum` in the tree, and then prefetches all tracked modules
through the retry, so a cache miss still works. A cache saved by a PR
run is visible only to that PR, but one saved on the default branch can
be restored by any PR
([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)).
That is why the build cache warmer on main writes the same key before
its image build. This way the first run of a PR, which is the one that
flakes, starts warm. The warmer restores only an exact key, so old
module versions don't pile up, and its Go steps can't fail the image
build. Both runner labels are Linux, so the key matches. Both also check
out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the
same cache path, but I have not seen it on the warmer's runner yet. The
release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env
only, no cache.
`hack/go-module-fetch-contract.bats` covers all three. Each test is red
on main, and I checked each guard with a mutation.
Closes#4591
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Reliability**
- Go module downloads now fall back to direct access when the public
proxy encounters errors.
- Builds and automated checks retry failed module downloads, helping
them recover from temporary network issues.
- **Performance**
- CI caches Go modules to reduce repeated downloads and speed up builds
and tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The image build ran go build only, and go build never compiles a
_test.go, so nothing ran the tests of the code the patches change. Run
go test in the builder stage on the packages the patches touch. The list
is read from the patches themselves, so a new patch is covered without
an edit to the Dockerfile.
Two of the files the patches edit are upstream integration tests behind
the integration build tag, which go test compiles none of, so each
directory is also vetted with that tag. Doing it per derived directory
rather than for the one directory those tests live in today keeps a
later patch to a tagged test elsewhere covered.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A transient proxy.golang.org HTTP/2 stream reset fails an image build
outright: Go's default GOPROXY falls back to direct only on a 404 or
410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a
build arg from hack/common-envs.mk, declared by every Go builder stage,
so any proxy error falls back to fetching from the origin. The value
comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may
name a mirror that buildkit cannot reach. The alpine golang images
carry no git, so in their stages a module hosted on GitHub cannot be
fetched directly and the retry is the cover.
Explicit `go mod download` steps retry three times with a growing
pause. The loop is inline, in one form the contract test pins: the
images build from different contexts, so no shared script can be
copied in, and a named build context in the shared buildx arguments
makes buildx refuse every Dockerfile pinned to a frontend older than
dockerfile:1.4.
The two alpine stages that left the download to `go build` get the
same download step, because without git neither the fallback nor a
retry covered them.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The remaining Go builder stages, plus the kube-ovn, console and
piraeus-server builders, ran on the target platform. A multi-platform
build therefore ran the whole compiler under QEMU: slow, and for Go
unreliable, as the migration-controller Dockerfile already records a
"found bad pointer in Go heap" crash from exactly that. The builders
now run on $BUILDPLATFORM and produce the target architecture from
TARGETARCH.
Two builders only produced the right architecture because they were
emulated. token-proxy ran `go build` with no GOOS/GOARCH, and kube-ovn
calls upstream `make build-go`, which hardcodes GOARCH=amd64 even on an
arm64 build. On a native builder both would have shipped host binaries
inside the other architecture's image, and nothing fails until a node
of that architecture starts the container. token-proxy now passes the
target explicitly; kube-ovn picks build-go or its arm64 twin
build-go-arm from TARGETARCH.
The console builder emits static JS and linstor-server's debian/control
declares every package Architecture: all, so both run natively without
a target.
A bats test requires every stage that compiles to sit on a BUILDPLATFORM
FROM and to read TARGETARCH or TARGETPLATFORM on a RUN or ENV line. A
bare ARG declaration does not count, which is what let token-proxy
through.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Go 1.26.4 carries standard-library advisories in net/http, net/url,
crypto/tls, encoding/asn1 and html/template that govulncheck reports
as reachable from the first-party binaries. The shipped stdlib comes
from the builder image, so the builder tags move together with the
go directive; the digests are the multi-arch indexes that golang:1.27,
1.27-alpine and 1.27-bookworm resolve to at 1.27.1.
api/apps/v1alpha1 keeps go 1.26.4. It is a published types-only
module, no binary here links its stdlib, and raising its directive
would force every consumer onto Go 1.27.
linstor-csi v1.11.2 pins grpc v1.81.0, which no longer builds on
Go 1.27: golang.org/x/net drops http2.TrailerPrefix there. Its image
now raises grpc to v1.83.2 before building, as the kamaji image does
for its advisories; the upstream unit tests pass on it.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
freshworks-oss/redis-operator v3.3.5 has no TLS support. The operator
work that packages/apps/redis renders against was ported upstream as
freshworks-oss/redis-operator#85 and is not in a release yet, so carry it
here over the v3.3.5 tarball until one lands.
Three pieces, and TLS is silently off unless all three are present:
- images/redis-operator/patches/tls.diff is the source half. The
Dockerfile already applies patches/*.diff, and the shell expands that
to labels.diff then tls.diff; they touch disjoint files, so the order
does not matter, but it is alphabetically stable either way.
- patches/crd-tls.diff teaches the vendored CRD about spec.tls. Without
it the apiserver prunes spec.tls from every RedisFailover the redis
chart renders and the patched binary waits for a field it never sees.
- patches/rbac-tls.diff grants Create, Get and Update on Secrets, so the
operator can publish the CA-only Secret named by
spec.tls.caCertSecretName. Those are the only Secret calls in the
patched source: upstream v3.3.5 only ever reads one, and the patch
adds Create and Update on a Secret it addresses by name, so nothing
lists, watches, patches or deletes.
tls.diff is cut from that PR's commit 2ad2ed8d1 rebased onto v3.3.5, and
its header pins that SHA so the next refresh can be diffed against it.
Beyond the TLS listener it carries the renewal roll: on every reconcile the operator stamps a hash of the TLS
Secret's tls.crt and ca.crt into the pod template of the Redis
StatefulSet and the Sentinel Deployment, so a renewed certificate rolls
Redis through the operator's own roller and Sentinel through its
Deployment. The effective spec.tls.authClients value is folded into
the same hash, so changing it rolls the pods the same way instead of
landing at the next unrelated restart, and the operator presents its
client certificate whenever the Secret carries one, independent of that
field, so neither direction of the change locks it out of the instance
it reconciles. It also mounts the TLS volume 0440 when the pod declares an
fsGroup, rejects a caCertSecretName that names the TLS Secret itself,
and waits for the TLS material before it writes the ConfigMaps and the
workloads, so a first TLS install starts its pods once.
tls.enabled is fixed at creation. Redis replication and Sentinel cannot
straddle plaintext and TLS: a replica restarted onto the other protocol
cannot sync from the master, and Sentinels restarted onto it fail over
to such a replica while the master still takes writes. The rule that
refuses the change is a CEL transition rule on the RedisFailover spec;
it lives in the CRD, so it ships here through patches/crd-tls.diff
together with the schema block it references. The cut carries source
only, and the Dockerfile strips the tarball's manifests/, charts/,
example/ and .github/ before applying the patches, so the image builds
from source alone and the two upstream tests that read the CRD YAML skip
instead of failing against v3.3.5's stale copy; the CRD half is
patches/crd-tls.diff.
cert-manager Certificates get no grant. The patched operator manages a
Certificate only under spec.tls.certManager, and packages/apps/redis
pins the other mode: it always supplies spec.tls.certificateSecret, so
that code path is unreachable as the platform ships it. A comment at
the rule site records the decision and names the verbs the rule would
need, so a re-vendor review reads the absence as deliberate.
The chart patches go through the mechanism the packages that customise a
vendored chart already use — reloader, grafana-operator, ingress-nginx,
seaweedfs and kamaji all re-apply a patches/*.diff at the end of update:
— because `rm -rf charts` discards a hand-edit and the next `make update`
would take TLS back out without failing.
tls.diff also carries one change that is not TLS and takes effect with
TLS off: MakeSlaveOfWithPort connects to the replica on its own port
rather than on the master's. It changes a dial, not a pod template, so
nothing rolls when this image ships; the probes dial localhost only
under TLS, and the plaintext pod templates stay byte-identical to
v3.3.5. The patch header names it, so a reader of patches/ does not
take the file for TLS-only.
The grant is narrower than the ported operator's own chart asks for:
that one also grants patch on Secrets, and create, delete, get, list,
patch, update and watch on Certificates. None of that is reachable from
the source as shipped here.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When TLS is enabled the operator publishes a CA-only Secret derived from
the leaf TLS Secret, but the ClusterRole only granted get on secrets, so
reconciling a TLS-enabled RedisFailover failed with a forbidden error and
the CA secret was never created. Grant the operator full management of
secrets, matching the other resources it owns.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The freshworks-oss operator authenticates the CLI in its generated
readiness and shutdown scripts by exporting the engine's auth env var
before invoking it. For the Valkey engine it used VALKEYCLI_AUTH, but
valkey-cli honours only REDISCLI_AUTH (its --help documents that name and
no VALKEYCLI_AUTH). With auth enabled the readiness probe's valkey-cli
call therefore returned NOAUTH, the role never resolved, and the data
pods never became Ready.
Patch valkeyEngine.CLIAuthEnvName to REDISCLI_AUTH so auth-enabled Valkey
instances pass readiness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The freshworks-oss fork exits fatal at startup when operator_group_id is
unset, unlike the archived spotahome operator. Pin a stable non-empty value
via extraEnvVars so the operator boots instead of CrashLooping and blocking
the install.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Build the freshworks-oss v3.3.5 operator image (with our label-preservation
patch) and pin its digest in values.yaml, replacing the placeholder tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
spotahome/redis-operator was archived 2026-06-11 (read-only). Switch the
operator to the actively-maintained freshworks-oss/redis-operator fork, which
keeps the same RedisFailover CRD and API group (databases.spotahome.com) and
adds a native Valkey engine (spec.engine: Redis|Valkey) — so managed Redis is
unaffected and managed Valkey no longer needs a Redis-compat shim image.
- re-vendor the chart from oci://ghcr.io/freshworks-oss/charts/redis-operator
- build the operator image from freshworks-oss v3.3.5 source
- re-target the label/annotation-preservation patch onto the freshworks module
The operator image is built out-of-band (make -C packages/system/redis-operator
image) and its digest pinned in values.yaml; a maintainer must rebuild and pin
before merge (tag is a placeholder here).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Several first-party images still built their Go binaries with EOL or
behind-latest toolchains (golang 1.20, 1.20.6, 1.23.4, 1.24, 1.25), so a
plain rebuild kept shipping standard-library versions that the current
1.26 toolchain fixes. Raise every Go builder image (FROM and the
builder_image ARG defaults) to golang:1.26 across cluster-autoscaler,
kubevirt-cloud-provider, redis-operator, multus-cni, keycloak-operator,
objectstorage, kilo and linstor-csi.
This is the build-time counterpart to raising the go directive in the
first-party Go modules: the toolchain that compiles the binary, not the
go.mod minimum, determines the standard-library version baked into the
image.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Moves the pinned Alpine runtime bases of the first-party images forward to
alpine:3.24 to clear the published musl / zlib advisories carried by the stale
layers:
- mariadb-backup: 3.20 -> 3.24
- cozystack-operator: 3.22 -> 3.24
- platform-migrations: 3.22 -> 3.24
- grafana-dashboards: 3.22 -> 3.24
- kubeovn-webhook: 3.21.3 -> 3.24 (also drops the over-specific patch pin in
favour of the minor-float convention used elsewhere)
- redis-operator: latest -> 3.24 (pins the previously-floating runtime base)
These rebuild on the release asset pipeline; the digests pinned in values.yaml
are re-pinned there on the next build, matching how the rest of the ~30
first-party images pick up the shared base.
Side effect for platform-migrations: its kubectl / helm come from the Alpine
repo, so the base bump moves them 1.33 -> 1.36 and 3.18 -> 3.19. Both are
current upstream stable and compatible with the management cluster; for the
cozystack migration scripts (apply / get / label / helm release ops) the newer
kubectl is in skew with the modern control plane.
The build-only `alpine AS source` stages (keycloak-operator,
objectstorage-controller) are left as-is: their final images are distroless, so
no Alpine layer ships. The s3manager base is owned by the separate bucket bump,
and kilo's base moves in its own image rebuild.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
This patch moves the build of the Redis operator into the Cozystack
organization and patches it to prevent overwriting third-party labels on
owned resources.
```release-note
[redis-operator] Move operator into tree and patch it to retain
third-party labels on owned resources, reducing noisy traffic to the API
server.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This change is aimed at improving the development experience.
- The option `make delete` has been added.
- Added check for `NAME` and `NAMESPACE` variables
- Now, any package (not just system ones) can include options such as
make show, make diff, make apply.
- Applications from packages/extra require explicit specification of the
`NAMESPACE`.
- Applications from packages/apps require explicit specification of both
`NAME` and `NAMESPACE`.
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Yasar Rahman