The multus chart takes its resources from values, but the platform
renders the cozystack.multus Package with stageCniPlugins alone, and an
edit to that child Package is reverted by the platform reconcile. A
running cluster therefore had no way to raise the multus limits.
networking.multus.resources is passed to the Package only when set, so
an unset key leaves the chart default in charge. A value that is not a
map stops the render here; the multus chart keeps refusing a render
without a memory limit or request.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
kube-multus serves the CNI call for every pod sandbox on its node, and
its 100m CPU limit was hardcoded in the patched upstream template.
After a CNI outage a node had ~600 dead sandboxes and 220 pods waiting:
multus was throttled in every CFS period, calls outlived kubelet's
two-minute CRI deadline, each retry grew the queue, and the node never
recovered. With the CPU limit lifted the 1Gi memory limit OOM-looped;
2 CPU and 4Gi cleared the backlog in about three minutes.
Resources now come from values, defaulting to the same requests, no
CPU limit and a 4Gi memory limit. The memory limit stays so the pod
cgroup keeps memory.max and remains outside the Talos OOM handler's
candidate set.
Since an explicit null in a values override could drop either memory
key, the render now fails without limits.memory or requests.memory.
Without the request it defaults to the limit, and a node with less
allocatable memory than that is left without multus.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
One transient error from proxy.golang.org is enough to fail an image
build or the `Unit & controller tests` job today. The error is an HTTP/2
stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or
falls back after it. This PR adds three things: a direct fallback, a
bounded retry and a module cache.
Image builds now fall back to fetching modules from their origin when
the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a
build arg from `hack/common-envs.mk`, and every Go builder stage
declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error.
With Go's default comma it does that only on a 404 or 410. The value
comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because
buildkit can't reach a mirror on localhost or a `file://` proxy. Every
workflow job that runs Go on the runner sets the same value as env.
Explicit module downloads in Dockerfiles are retried three times with a
growing pause. The loop is inline, in one form that the contract test
pins. I tried a shared script from a named build context first, but
buildx refuses any `--build-context` for the four Dockerfiles still
pinned to the `dockerfile:1.2` frontend. The images also build from
different contexts, so no single file can be copied in. Raising those
pins changes how those images build and needs a real build to check, so
I left it for a follow-up. Two alpine stages, token-proxy and
shadowbox's `ss-server`, left the download to `go build`. The alpine
golang images have no git, so the direct fallback can't fetch for them,
and they now get the same download step. The runner jobs use
`hack/go-mod-download.sh` with the same bound.
The unit test job now starts from a module cache instead of downloading
everything. It restores `GOMODCACHE` with `actions/cache`, keyed on
every `go.sum` in the tree, and then prefetches all tracked modules
through the retry, so a cache miss still works. A cache saved by a PR
run is visible only to that PR, but one saved on the default branch can
be restored by any PR
([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)).
That is why the build cache warmer on main writes the same key before
its image build. This way the first run of a PR, which is the one that
flakes, starts warm. The warmer restores only an exact key, so old
module versions don't pile up, and its Go steps can't fail the image
build. Both runner labels are Linux, so the key matches. Both also check
out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the
same cache path, but I have not seen it on the warmer's runner yet. The
release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env
only, no cache.
`hack/go-module-fetch-contract.bats` covers all three. Each test is red
on main, and I checked each guard with a mutation.
Closes#4591
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Reliability**
- Go module downloads now fall back to direct access when the public
proxy encounters errors.
- Builds and automated checks retry failed module downloads, helping
them recover from temporary network issues.
- **Performance**
- CI caches Go modules to reduce repeated downloads and speed up builds
and tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The other patched Go images now test the packages their patches touch
in the builder stage. The one package the multus patch touches,
pkg/multus, creates network namespaces in its tests, which a build
container is not allowed to do, so that image keeps building without a
test step. The Dockerfile records the reason, so the absence reads as a
decision and not as an omission.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A transient proxy.golang.org HTTP/2 stream reset fails an image build
outright: Go's default GOPROXY falls back to direct only on a 404 or
410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a
build arg from hack/common-envs.mk, declared by every Go builder stage,
so any proxy error falls back to fetching from the origin. The value
comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may
name a mirror that buildkit cannot reach. The alpine golang images
carry no git, so in their stages a module hosted on GitHub cannot be
fetched directly and the retry is the cover.
Explicit `go mod download` steps retry three times with a growing
pause. The loop is inline, in one form the contract test pins: the
images build from different contexts, so no shared script can be
copied in, and a named build context in the shared buildx arguments
makes buildx refuse every Dockerfile pinned to a frontend older than
dockerfile:1.4.
The two alpine stages that left the download to `go build` get the
same download step, because without git neither the fallback nor a
retry covered them.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Talos v1.14.2 ships DRBD 9.3.4 and ZFS 2.4.4. The v1.13 line tops
out at DRBD 9.3.3, which lacks the 9.3.4 fixes for a sender thread
pinning a CPU while its connection is down, for two-phase-commit
retries and hangs that leave drbdadm disconnect stuck, and for two
nodes ending UpToDate with different data after a reconnect.
Talos v1.14 no longer publishes ghcr.io/siderolabs/installer, so the
imager profiles take installer-base as the base installer, which is
also what upstream's own v1.14 imager profiles use.
BREAKING CHANGE: Talos v1.14 no longer loads kernel modules on demand,
so every node must list drbd_transport_tcp in machine.kernel.modules
before it is upgraded, or DRBD cannot connect to its peers.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Go 1.26.4 carries standard-library advisories in net/http, net/url,
crypto/tls, encoding/asn1 and html/template that govulncheck reports
as reachable from the first-party binaries. The shipped stdlib comes
from the builder image, so the builder tags move together with the
go directive; the digests are the multi-arch indexes that golang:1.27,
1.27-alpine and 1.27-bookworm resolve to at 1.27.1.
api/apps/v1alpha1 keeps go 1.26.4. It is a published types-only
module, no binary here links its stdlib, and raising its directive
would force every consumer onto Go 1.27.
linstor-csi v1.11.2 pins grpc v1.81.0, which no longer builds on
Go 1.27: golang.org/x/net drops http2.TrailerPrefix there. Its image
now raises grpc to v1.83.2 before building, as the kamaji image does
for its advisories; the upstream unit tests pass on it.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The cni-plugins-staging-contract test pins the kube-ovn release whose
CNI_PLUGINS_VERSION was last checked against multus's, and bumping
kube-ovn to v1.15.26 trips it while the marker still reads v1.15.10.
kube-ovn v1.15.26's base image (dist/images/Dockerfile.base) still uses
CNI_PLUGINS_VERSION=v1.9.1, identical to multus, so portmap and macvlan
stay in lockstep across the two daemonsets. Only the marker moves; the
plugins pin and per-arch checksums are unchanged.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The Talos userspace OOM handler (v1.12+) ranks cgroups with
memory_max.hasValue() ? 0.0 :
{Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current
and discards every cgroup scoring zero, so only a memory limit takes a
pod out of the victim set. Selection is decoupled from the trigger: a
tenant workload thrashing against its own limit drives node memory PSI,
and a limit-free system DaemonSet is killed for it. metallb-speaker and
linstor-satellite were the observed victims.
Set requests and limits on the node DaemonSets that had none. Requests
come from 14d peak working set measured on a 3-node dev cluster; limits
are deliberately loose, sized as a runaway backstop rather than a fitted
ceiling, because several of these scale with cluster size that a dev
cluster understates and because undersizing converts a rare
pressure-driven kill into a deterministic one. A limit costs nothing
until it is hit -- only the request reserves.
memory.max lands on the POD cgroup only when EVERY container in the pod
carries a limit, so sidecars and init containers are covered too; one
limit-free container returns the whole pod to the victim set.
Two DaemonSets are not chart-rendered and needed their own seam:
linstor-satellite via the LinstorSatelliteConfiguration podTemplate, and
virt-handler via KubeVirt customizeComponents.patches, since virt-operator
owns that DaemonSet. Note linstor-controller stays deliberately limit-free
-- it is a JVM that must not be OOM-killed mid-operation, and cluster_test
pins that.
Two things the multus manifest needed beyond the values change.
customize-deployment.patch is the source for that template, so the
limits are carried there rather than only in the generated file, which
`make update` would discard. And the reason kube-multus had no memory
limit was that this patch DELETED upstream's 50Mi rather than raising
it -- 50Mi being well under the 230Mi measured peak -- which is what
removed memory.max from the pod cgroup in the first place.
Not covered, because upstream hardcodes the containers with no resources
knob: the four frr-k8s cp-* init containers, the kube-ovn hostpath-init
and install-cni init containers, and cozy-proxy, whose vendored chart
exposes no resources value at all. hami and kilo are left alone as
optional components with no usage measurements behind them. All of these
still get a limit from the default LimitRange the operator maintains in
every system namespace.
Every limit is pinned by a helm-unittest assertion, and every assertion
was mutation-tested: changing the rendered value turns its suite red.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The check that multus is pointed at the cilium conflist matched its
JSON fragment anywhere in the rendered daemon config. The same text
nested inside any other key satisfies it, so a config whose effective
multusMasterCNI names a different conflist passes. Multus uses that
value as written when it is non-empty -- it delegates to whatever is
named, so a wrong-but-existing name silently routes through the other
CNI instead of cilium, which is the state this assertion exists to
prevent. Auto-discovery happens only when the value is empty, and a
name that does not exist fails manager startup rather than falling
back.
Anchor it to the top level, where the template writes its keys. That
pins the position rather than parsing the document; deciding which key
is effective needs a JSON parser, which helm-unittest does not have.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Multus-backed NetworkAttachmentDefinitions delegate to the upstream
containernetworking reference plugins (bridge, macvlan, ipvlan, ...),
and nothing guarantees a node has them in the directory Multus
searches. On k3s they live under /var/lib/rancher/k3s/data/cni while
Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails
there with `failed to find plugin "bridge" in path [/opt/cni/bin]`.
Bundle 14 of the release's plugins into the multus image and add an
install-cni-plugins init container that installs them into the host
/opt/cni/bin, mirroring the existing install-multus-binary container.
loopback, dhcp, tap and dummy are left out; the Dockerfile records why.
Gate it on a new chart value, stageCniPlugins, resolved per bundle and
on for both. Talos bundles only bridge, firewall, flannel, host-local,
loopback and portmap, so the other ten are missing there exactly as on
generic Linux; the four that overlap are rewritten with the same
upstream version, v1.9.1 on both sides, rather than an older one. A
marker in the image Dockerfile records the Talos release that pin was
compared against, and a test holds it to the release the platform
ships. multus therefore moves out of system.common-packages into the
two variant branches, the way linstor already does for talos.enabled.
networking.stageCniPlugins overrides the variant default in either
direction.
Where staging is on the install is unconditional: a plugin is replaced
on every pod recreation, not only when the destination is missing. A
destination that is already a directory is reported and left alone,
since replacing it is not something this can do safely. Turning the
value off afterwards does not undo it; nothing here removes or
restores a plugin.
The container is not privileged and takes no mount propagation. It
runs in the spc_t SELinux domain, without which an enforcing node
refuses the write. Each plugin is installed under a temporary name and
swapped in with a rename: /opt/cni/bin is shared with cilium and
kube-ovn. The plugin tarball is checksummed before extraction.
A plugin that cannot be installed is reported on stderr and to
/dev/termination-log, and the container still exits 0: Multus stays the
node's primary CNI and nothing removes its config, so an init container
that never completes would leave the shim with no daemon behind it.
The manifest change lives in customize-deployment.patch so it survives
make update.
BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI
plugins in /opt/cni/bin on existing nodes, Talos included -- four of the
names it writes are ones the Talos node image also carries. Opt out
before upgrading: declining afterwards does not restore what was
replaced.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Re-vendors the multus-cni thick-mode daemonset from upstream v4.2.3 to v4.3.0
and rebuilds the cozystack image (multi-arch, golang:1.26 builder; v4.3.0's
go.mod and the published security fixes warrant the stdlib bump from 1.24). The
fix-del-without-cache and customize-deployment patches both re-apply cleanly to
v4.3.0. The rebuilt image is pinned by digest in the daemonset.
Folds the previously hand-edited `multusMasterCNI: "05-cilium.conflist"`
CNI-ordering fix into customize-deployment.patch so it survives every make
update (it was a raw daemonset edit that the re-vendor would otherwise wipe,
re-introducing the kube-ovn-before-cilium startup race).
Adds helm-unittest coverage (the package shipped none): asserts the daemonset
runs the cozystack-built image and that the daemon-config pins multusMasterCNI
to the cilium conflist.
Adaptations:
- images/multus-cni/Dockerfile: builder golang:1.24 -> 1.26, ARG VERSION v4.2.3 -> v4.3.0
- templates/multus-daemonset-thick.yml: re-vendored v4.3.0 + customize patch; pin rebuilt multi-arch image
- patches/customize-deployment.patch: carry the multusMasterCNI CNI-ordering fix
- Makefile: add test target
- tests/multus_test.yaml: new helm-unittest coverage
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## Summary
- Pin multusMasterCNI to `05-cilium.conflist` to prevent race condition
at boot
- Fixes issue where multus auto-detects wrong conflist, bypassing Cilium
CNI chain
## Problem
During node boot (e.g. Talos upgrade), multus auto-detects the master
CNI conflist by scanning
the CNI config directory. If kube-ovn writes `10-kube-ovn.conflist`
before Cilium writes
`05-cilium.conflist`, multus selects the wrong file. Pods on the
affected node then bypass
the Cilium chain entirely, have no Cilium endpoint, and their traffic
gets blocked by
cluster-wide network policies.
## Fix
Set `multusMasterCNI: "05-cilium.conflist"` in the multus daemon config
to explicitly
pin the master CNI file, eliminating the race condition.
## Upstream
- Issue: https://github.com/k8snetworkplumbingwg/multus-cni/issues/1499
- Tracking: https://github.com/cozystack/cozystack/issues/2310
## Test plan
- [x] Verified `multusMasterCNI` JSON key matches upstream struct tag
- [ ] Deploy and verify `00-multus.conf` points to `05-cilium.conflist`
after node reboot
When CNI ADD never completes, multus DEL fails because delegate plugins
require runtime state that only exists after a successful ADD. This
creates a deadlock where containerd cannot release sandbox name
reservations, permanently blocking pod creation on the affected node.
Build a custom multus-cni image with a patch that returns success on
DEL when no cache file exists (ADD was never completed).
Upstream PR: https://github.com/k8snetworkplumbingwg/multus-cni/pull/1498
Tracking issue: https://github.com/cozystack/cozystack/issues/2310
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Aleksei Sviridkin