A transient proxy.golang.org HTTP/2 stream reset fails an image build outright: Go's default GOPROXY falls back to direct only on a 404 or 410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a build arg from hack/common-envs.mk, declared by every Go builder stage, so any proxy error falls back to fetching from the origin. The value comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may name a mirror that buildkit cannot reach. The alpine golang images carry no git, so in their stages a module hosted on GitHub cannot be fetched directly and the retry is the cover. Explicit `go mod download` steps retry three times with a growing pause. The loop is inline, in one form the contract test pins: the images build from different contexts, so no shared script can be copied in, and a named build context in the shared buildx arguments makes buildx refuse every Dockerfile pinned to a frontend older than dockerfile:1.4. The two alpine stages that left the download to `go build` get the same download step, because without git neither the fallback nor a retry covered them. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
flux-shard-operator
Spreads tenant HelmReleases across multiple helm-controller shards so a noisy tenant (e.g. a HelmRelease stuck in remediation) cannot degrade the others. The damage of a saturated helm-controller is per-pod / per-informer, so horizontal sharding with per-shard label-scoped informers is what isolates both worker starvation and cache-lag blast radius.
What it does
The operator has three parts, all served by one Deployment (leader-elected controllers, webhook on every replica):
-
Shard runtime. Reconciles
shardCounthelm-controller Deployments (helm-controller-shard<i>,--watch-label-selector=sharding.fluxcd.io/key=shard<i>) in the flux namespace, cloned from the flux-aiofluxDeployment's helm-controller container and sanitised (no host networking, no localhost cross-container wiring, no inherited corporate-proxy env, and a startupProbe guarding a slow start). The helm-controller image and feature-gates are inherited from flux-aio automatically. Deployments beyondshardCountare pruned once they drain, and the legacy hand-rolledflux-tenantsDeployment is retired once no HelmRelease carriessharding.fluxcd.io/key=tenants. -
Placement controller. Owns the tenant→shard assignment. The unit of placement is the tenant: all HelmReleases of one tenant (parent
tenant-<id>plus everything in namespacetenant-<id>) carry the same shard label, so a noisy tenant's blast radius is bounded to its shard's co-residents. Tenants are distributed greedy least-loaded, weighted by HelmRelease count (N tenants over N shards land exactly 1 per shard). The assignment is recorded as theinternal.cozystack.io/flux-shardlabel on the tenant namespace; HelmRelease labels remain the source of truth on restarts. Moves are paced and deleting tenants are never moved. Watches are metadata-only, so the controller does not decode the helm-controller status-patch firehose. -
Mutating webhook (CREATE-only,
failurePolicy: Ignore). Stamps the tenant's shard onto every HelmRelease at admission, so each one is born on the correct shard regardless of creation path. On webhook outage or before the first assignment, HelmReleases keep their legacytenantskey until the placement controller relabels them — graceful degradation, never blocked creation.
System (non-tenant) HelmReleases are untouched: flux-aio keeps reconciling everything without a shard key (!sharding.fluxcd.io/key).
Parameters
Common parameters
| Name | Description | Value |
|---|---|---|
fluxShardOperator.image |
Container image | ghcr.io/cozystack/cozystack/flux-shard-operator:* |
fluxShardOperator.debug |
Enable debug logging | false |
fluxShardOperator.replicas |
Operator replica count | 2 |
fluxShardOperator.shardCount |
Number of helm-controller shards: auto sizes from the HelmRelease count, an integer pins it |
auto |
fluxShardOperator.rebalanceThreshold |
Load spread ratio above which tenants are rebalanced | 0.25 |
fluxShardOperator.pinnedTenants |
Map of tenant namespace to shard, pins heavy tenants to dedicated shards | {} |
fluxShardOperator.shard.concurrent |
--concurrent of each shard helm-controller |
5 |
fluxShardOperator.shard.resources |
Resources of each shard helm-controller (empty values inherit flux-aio) | {requests: {cpu: 100m, memory: 64Mi}, limits: {memory: 1Gi}} |
Autosizing
With shardCount: auto (the default) the operator drives the shard count from the tenant HelmRelease count: K = clamp(ceil(H/100), 1, min(16, T)), applied with hysteresis anchored on the currently provisioned shards (scale up eagerly once a shard runs >120 HR, scale down lazily only under 60 HR/shard) so K does not flap around sizing boundaries. The ~100 HR/shard target leaves headroom for existing tenants growing their HelmRelease count without an immediate reshard. Set an integer to pin the count explicitly.
Bootstrap on fresh installs
On a fresh cluster the first tenant helm-controller appears through a runtime chain: cert-manager issues the webhook/serving certificate, the operator starts, clones helm-controller-shard0 from flux-aio, and the placement controller assigns tenants once shard0 reports Ready (assignments only ever target ready shards). Until then tenant HelmReleases stay on their born sharding.fluxcd.io/key=tenants label and are simply not reconciled yet — the same install-order dependency as other cert-manager-gated components (capi-operator, capi-providers), surfaced as the platform HelmReleases not turning Ready. To debug a stalled bootstrap, walk the chain in order: cert-manager webhook up → flux-shard-operator Deployment Ready (image pull, cert mount) → helm-controller-shard0 Ready → HelmReleases relabeled off tenants.
Telemetry
The operator exports cozy_flux_shard_load, cozy_flux_shard_tenants, cozy_flux_shard_helmreleases, cozy_flux_shard_pending_moves, cozy_flux_shard_moves_total and cozy_flux_shard_recommended_count (the raw autosizing recommendation before hysteresis). Metrics are served plain-HTTP on :8080 and the operator pod carries prometheus.io/scrape annotations; gauges are populated by the leader replica.