Files
cozystack/packages/system/flux-shard-operator
Aleksei Sviridkin 548d33f01f build(images): fall back to direct and retry Go module downloads
A transient proxy.golang.org HTTP/2 stream reset fails an image build
outright: Go's default GOPROXY falls back to direct only on a 404 or
410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a
build arg from hack/common-envs.mk, declared by every Go builder stage,
so any proxy error falls back to fetching from the origin. The value
comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may
name a mirror that buildkit cannot reach. The alpine golang images
carry no git, so in their stages a module hosted on GitHub cannot be
fetched directly and the retry is the cover.

Explicit `go mod download` steps retry three times with a growing
pause. The loop is inline, in one form the contract test pins: the
images build from different contexts, so no shared script can be
copied in, and a named build context in the shared buildx arguments
makes buildx refuse every Dockerfile pinned to a frontend older than
dockerfile:1.4.

The two alpine stages that left the download to `go build` get the
same download step, because without git neither the fallback nor a
retry covered them.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 19:37:00 +03:00
..

flux-shard-operator

Spreads tenant HelmReleases across multiple helm-controller shards so a noisy tenant (e.g. a HelmRelease stuck in remediation) cannot degrade the others. The damage of a saturated helm-controller is per-pod / per-informer, so horizontal sharding with per-shard label-scoped informers is what isolates both worker starvation and cache-lag blast radius.

What it does

The operator has three parts, all served by one Deployment (leader-elected controllers, webhook on every replica):

  1. Shard runtime. Reconciles shardCount helm-controller Deployments (helm-controller-shard<i>, --watch-label-selector=sharding.fluxcd.io/key=shard<i>) in the flux namespace, cloned from the flux-aio flux Deployment's helm-controller container and sanitised (no host networking, no localhost cross-container wiring, no inherited corporate-proxy env, and a startupProbe guarding a slow start). The helm-controller image and feature-gates are inherited from flux-aio automatically. Deployments beyond shardCount are pruned once they drain, and the legacy hand-rolled flux-tenants Deployment is retired once no HelmRelease carries sharding.fluxcd.io/key=tenants.

  2. Placement controller. Owns the tenant→shard assignment. The unit of placement is the tenant: all HelmReleases of one tenant (parent tenant-<id> plus everything in namespace tenant-<id>) carry the same shard label, so a noisy tenant's blast radius is bounded to its shard's co-residents. Tenants are distributed greedy least-loaded, weighted by HelmRelease count (N tenants over N shards land exactly 1 per shard). The assignment is recorded as the internal.cozystack.io/flux-shard label on the tenant namespace; HelmRelease labels remain the source of truth on restarts. Moves are paced and deleting tenants are never moved. Watches are metadata-only, so the controller does not decode the helm-controller status-patch firehose.

  3. Mutating webhook (CREATE-only, failurePolicy: Ignore). Stamps the tenant's shard onto every HelmRelease at admission, so each one is born on the correct shard regardless of creation path. On webhook outage or before the first assignment, HelmReleases keep their legacy tenants key until the placement controller relabels them — graceful degradation, never blocked creation.

System (non-tenant) HelmReleases are untouched: flux-aio keeps reconciling everything without a shard key (!sharding.fluxcd.io/key).

Parameters

Common parameters

Name Description Value
fluxShardOperator.image Container image ghcr.io/cozystack/cozystack/flux-shard-operator:*
fluxShardOperator.debug Enable debug logging false
fluxShardOperator.replicas Operator replica count 2
fluxShardOperator.shardCount Number of helm-controller shards: auto sizes from the HelmRelease count, an integer pins it auto
fluxShardOperator.rebalanceThreshold Load spread ratio above which tenants are rebalanced 0.25
fluxShardOperator.pinnedTenants Map of tenant namespace to shard, pins heavy tenants to dedicated shards {}
fluxShardOperator.shard.concurrent --concurrent of each shard helm-controller 5
fluxShardOperator.shard.resources Resources of each shard helm-controller (empty values inherit flux-aio) {requests: {cpu: 100m, memory: 64Mi}, limits: {memory: 1Gi}}

Autosizing

With shardCount: auto (the default) the operator drives the shard count from the tenant HelmRelease count: K = clamp(ceil(H/100), 1, min(16, T)), applied with hysteresis anchored on the currently provisioned shards (scale up eagerly once a shard runs >120 HR, scale down lazily only under 60 HR/shard) so K does not flap around sizing boundaries. The ~100 HR/shard target leaves headroom for existing tenants growing their HelmRelease count without an immediate reshard. Set an integer to pin the count explicitly.

Bootstrap on fresh installs

On a fresh cluster the first tenant helm-controller appears through a runtime chain: cert-manager issues the webhook/serving certificate, the operator starts, clones helm-controller-shard0 from flux-aio, and the placement controller assigns tenants once shard0 reports Ready (assignments only ever target ready shards). Until then tenant HelmReleases stay on their born sharding.fluxcd.io/key=tenants label and are simply not reconciled yet — the same install-order dependency as other cert-manager-gated components (capi-operator, capi-providers), surfaced as the platform HelmReleases not turning Ready. To debug a stalled bootstrap, walk the chain in order: cert-manager webhook up → flux-shard-operator Deployment Ready (image pull, cert mount) → helm-controller-shard0 Ready → HelmReleases relabeled off tenants.

Telemetry

The operator exports cozy_flux_shard_load, cozy_flux_shard_tenants, cozy_flux_shard_helmreleases, cozy_flux_shard_pending_moves, cozy_flux_shard_moves_total and cozy_flux_shard_recommended_count (the raw autosizing recommendation before hysteresis). Metrics are served plain-HTTP on :8080 and the operator pod carries prometheus.io/scrape annotations; gauges are populated by the leader replica.