Native Go fuzzing over the flag parsers in config.go, asserting the
invariants that matter rather than just no-panic:
- ParseShardIndex round-trips with ShardName (accepted input is the canonical
name of the returned index; index is non-negative).
- ShardName round-trips back through ParseShardIndex for every valid index.
- ParseShardCount: a non-auto success yields count >= 1; auto yields 0.
- ParsePinnedTenants: every entry in a success has a non-empty key mapped to a
valid shard name.
Adds an OSS-Fuzz-detectable fuzzing signal for the OpenSSF Scorecard Fuzzing
check while covering parsers that sit on the sharding hot path.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BuildShardDeployment clones the flux-aio pod spec, which carries a required podAntiAffinity on app.kubernetes.io/name=flux. Inherited onto a shard pod it forbids scheduling on the flux-aio node, so on a single schedulable node every shard stays Pending and the tenant plane stalls (no ready shard -> assignments deferred -> HelmReleases stuck on key=tenants).
Drop the podAntiAffinity during sanitisation while keeping any benign nodeAffinity (e.g. os=linux). The fixture now carries flux-aio's affinity and TestBuildShardDeployment asserts the strip.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replacing the whole resources block dropped cloned values the override
did not name (e.g. the inherited cpu limit when only a memory limit is
configured), contradicting the documented inherit-when-empty semantics.
Also document the intentional provisioner-vs-placement sizing
asymmetry.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
ComputePlacement ignores pins to non-existent shards, but rebalance
treated any pinned tenant as sticky, so an invalid pin blocked expected
moves. Honor only pins placement itself honors.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
A failed patch no longer refreshes the rebalance cooldown or inflates
the moves counter, so retries are not delayed and not double-counted.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
The Update handler rebuilds the HelmRelease from the Application, which
re-stamped sharding.fluxcd.io/key with the ApplicationDefinition default
and bounced the HelmRelease off its operator-assigned shard on every
update. Carry the live label over from the current HelmRelease instead.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
--shard-count now accepts "auto" (the new default) or an explicit
positive integer. Auto mode drives the shard count from the tenant
HelmRelease count — K = clamp(ceil(H/100), 1, min(16, T)) — applied
with hysteresis anchored on the currently provisioned shards (scale up
eagerly past 120 HR/shard, scale down lazily under 60 HR/shard) so K
does not flap around sizing boundaries.
The per-shard target is 100 HelmReleases (down from the 150 in the
design doc) to leave headroom for existing tenants growing their
HelmRelease count without an immediate reshard.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
E2E caught the cloned shard crashlooping on Talos: the installer injects
the node-local KubePrism endpoint (KUBERNETES_SERVICE_HOST=localhost,
KUBERNETES_SERVICE_PORT=7445) into hostNetwork workloads, and the clone
carried it into a non-hostNetwork pod where localhost is dead. Sanitise
those env vars away so shards use the in-cluster defaults.
Defense in depth for the same failure mode: the placement controller now
defers reassignments whose target shard Deployment has no ready replica,
so a backfill can never hand tenants to a helm-controller that is not
actually running, and reacts to shard readiness changes directly.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Three leader-elected/served parts on metadata-only watches, so the
operator never decodes the helm-controller status-patch firehose:
- placement controller: singleton full-sync reconciler that rebuilds
the tenant index from HelmRelease labels (restart-safe), records the
assignment on the tenant namespace, self-heals HR labels and paces
reassignments so a backfill never floods a target shard;
- shard provisioner: reconciles helm-controller-shard<i> Deployments
cloned from the flux-aio helm-controller container and sanitised
(no host networking, no localhost cross-container wiring, per-shard
watch-label-selector), prunes drained extra shards and retires the
legacy flux-tenants Deployment once the tenants bucket is empty;
- CREATE-only mutating webhook: stamps the tenant's recorded shard on
newborn HelmReleases so each one is born on the correct shard; every
miss is permissive and falls back to the placement controller.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Tenant-granular placement for sharding the management-cluster tenant
helm-controller: tenant key extraction from HelmRelease metadata,
greedy least-loaded assignment weighted by HelmRelease count (with the
N-tenants-over-N-shards 1-per-shard guarantee), minimal-movement
rescale, threshold-gated rebalance with per-tenant cooldown hook,
pinned tenants, and the autosizing recommendation formula.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
