Run go fix over every first-party module: interface{} becomes any,
hand-written loops become slices, maps, min/max and range-over-int,
errors.As becomes errors.AsType, pointer helpers become new(expr) and
are inlined at their call sites, and so on. Every rewrite keeps
behaviour.
Three rewrites stay out. embedlit flattens embedded-field literals
such as ObjectMeta, syntax only Go 1.27 accepts, which would break
backports to release branches still on Go 1.26. omitzero drops
omitempty from struct fields, and openapi-gen reads omitempty as
optional, so metadata, spec and status would turn required in the
published schemas. And errorsastype rewrote an errors.As whose
target, apierrors.APIStatus, does not implement error, which
errors.AsType rejects at compile time.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Name all six proxy env spellings in the sanitisation list, and replace
the unestablished 'external HTTPS call' / 'flux-aio does not restart'
claims with the grounded in-cluster mechanism: after the
KUBERNETES_SERVICE_HOST drop above, the shard talks to the management
apiserver over the kubelet-injected ClusterIP, which NO_PROXY=.svc does
not match, so that startup call is what stalls behind an unreachable
proxy. Comment-only; no behaviour change.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Model the source liveness probe as flux-aio actually ships it (a bare httpGet
with no timing fields) and tighten the startupProbe assertions:
- assert the complete normalised budget (delay/period/success/failure) instead
of only FailureThreshold, so a later PeriodSeconds regression cannot silently
restore a short crashloop window;
- assert TimeoutSeconds is inherited from the source probe rather than a
hardcoded value, so forcing it stricter is caught;
- assert the startupProbe is a DeepCopy of liveness, not an alias, which would
otherwise stamp the 30-failure startup budget onto liveness and produce a
never-failing liveness probe.
The alias assertion is non-vacuous: it fails when the DeepCopy is regressed to
a plain alias.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The BuildShardDeployment doc comment and the operator README are the only
record of how far the cloned shard container has diverged from flux-aio.
Add the two steps this fix introduced (dropping the inherited corporate-proxy
env and deriving a startupProbe) so the list stays the authoritative account
and does not invite re-adding what was deliberately removed.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Review follow-ups:
- Do not force the cloned startupProbe TimeoutSeconds to 1: keep the value
inherited from the liveness handler, otherwise a future flux-aio with a
larger liveness timeout would get a stricter startup probe and could
recreate the crashloop. Covered by a regression guard (liveness timeout 5s
must be inherited).
- Soften the proxy-strip rationale to the A/B-confirmed mechanism (blocking
external HTTPS call at startup) and note the remote spec.kubeConfig
exception that does not apply to cozystack in-cluster guest apiservers.
- Correct the test fixture comment: the proxy env is operator/patch-level, not
installer-injected (the installer injects only KUBERNETES_SERVICE_*).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The cloned shard Deployment inherits only a liveness probe (~30s window,
no startupProbe), so a controller still syncing caches on startup is killed
before it serves /healthz and never recovers. Derive a startupProbe from the
liveness handler with a generous budget so a slow start is not fatal, while
liveness still catches a wedged running pod.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
A sharded helm-controller inherits HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the
flux-aio helm-controller container. A standalone shard needs no external
egress (source-controller fetches artifacts), so behind an unreachable
corporate proxy it makes a blocking startup HTTPS call that never completes,
the manager never serves /healthz, and the liveness probe crashloops the pod
forever, freezing every HelmRelease carrying a sharding key. flux-aio itself
survives only because it starts once and never restarts.
Drop the proxy env (and the now-pointless NO_PROXY) when cloning, alongside
the existing localhost/KubePrism sanitisation, so the shard talks to the
in-cluster apiserver directly.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Native Go fuzzing over the flag parsers in config.go, asserting the
invariants that matter rather than just no-panic:
- ParseShardIndex round-trips with ShardName (accepted input is the canonical
name of the returned index; index is non-negative).
- ShardName round-trips back through ParseShardIndex for every valid index.
- ParseShardCount: a non-auto success yields count >= 1; auto yields 0.
- ParsePinnedTenants: every entry in a success has a non-empty key mapped to a
valid shard name.
Adds an OSS-Fuzz-detectable fuzzing signal for the OpenSSF Scorecard Fuzzing
check while covering parsers that sit on the sharding hot path.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BuildShardDeployment clones the flux-aio pod spec, which carries a required podAntiAffinity on app.kubernetes.io/name=flux. Inherited onto a shard pod it forbids scheduling on the flux-aio node, so on a single schedulable node every shard stays Pending and the tenant plane stalls (no ready shard -> assignments deferred -> HelmReleases stuck on key=tenants).
Drop the podAntiAffinity during sanitisation while keeping any benign nodeAffinity (e.g. os=linux). The fixture now carries flux-aio's affinity and TestBuildShardDeployment asserts the strip.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replacing the whole resources block dropped cloned values the override
did not name (e.g. the inherited cpu limit when only a memory limit is
configured), contradicting the documented inherit-when-empty semantics.
Also document the intentional provisioner-vs-placement sizing
asymmetry.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
ComputePlacement ignores pins to non-existent shards, but rebalance
treated any pinned tenant as sticky, so an invalid pin blocked expected
moves. Honor only pins placement itself honors.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
A failed patch no longer refreshes the rebalance cooldown or inflates
the moves counter, so retries are not delayed and not double-counted.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
The Update handler rebuilds the HelmRelease from the Application, which
re-stamped sharding.fluxcd.io/key with the ApplicationDefinition default
and bounced the HelmRelease off its operator-assigned shard on every
update. Carry the live label over from the current HelmRelease instead.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
--shard-count now accepts "auto" (the new default) or an explicit
positive integer. Auto mode drives the shard count from the tenant
HelmRelease count — K = clamp(ceil(H/100), 1, min(16, T)) — applied
with hysteresis anchored on the currently provisioned shards (scale up
eagerly past 120 HR/shard, scale down lazily under 60 HR/shard) so K
does not flap around sizing boundaries.
The per-shard target is 100 HelmReleases (down from the 150 in the
design doc) to leave headroom for existing tenants growing their
HelmRelease count without an immediate reshard.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
E2E caught the cloned shard crashlooping on Talos: the installer injects
the node-local KubePrism endpoint (KUBERNETES_SERVICE_HOST=localhost,
KUBERNETES_SERVICE_PORT=7445) into hostNetwork workloads, and the clone
carried it into a non-hostNetwork pod where localhost is dead. Sanitise
those env vars away so shards use the in-cluster defaults.
Defense in depth for the same failure mode: the placement controller now
defers reassignments whose target shard Deployment has no ready replica,
so a backfill can never hand tenants to a helm-controller that is not
actually running, and reacts to shard readiness changes directly.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Three leader-elected/served parts on metadata-only watches, so the
operator never decodes the helm-controller status-patch firehose:
- placement controller: singleton full-sync reconciler that rebuilds
the tenant index from HelmRelease labels (restart-safe), records the
assignment on the tenant namespace, self-heals HR labels and paces
reassignments so a backfill never floods a target shard;
- shard provisioner: reconciles helm-controller-shard<i> Deployments
cloned from the flux-aio helm-controller container and sanitised
(no host networking, no localhost cross-container wiring, per-shard
watch-label-selector), prunes drained extra shards and retires the
legacy flux-tenants Deployment once the tenants bucket is empty;
- CREATE-only mutating webhook: stamps the tenant's recorded shard on
newborn HelmReleases so each one is born on the correct shard; every
miss is permissive and falls back to the placement controller.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Tenant-granular placement for sharding the management-cluster tenant
helm-controller: tenant key extraction from HelmRelease metadata,
greedy least-loaded assignment weighted by HelmRelease count (with the
N-tenants-over-N-shards 1-per-shard guarantee), minimal-movement
rescale, threshold-gated rebalance with per-tenant cooldown hook,
pinned tenants, and the autosizing recommendation formula.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Aleksei Sviridkin