Extends the ZooKeeper->KRaft migration hook so that N Kafka clusters
sharing one namespace all reach KRaft automatically, with no manual
steps. This closes the last gap in the migration: previously only one
cluster per namespace could migrate in place (the broker pool must be
named exactly "kafka" to adopt the existing <cluster>-kafka-N brokers,
and that name is namespace-unique), so a second co-namespaced ZooKeeper
cluster hit the render guard and its HelmRelease never rendered again --
stuck on ZooKeeper with no way forward once a future Strimzi drops the
migration.
The clusters now serialize on the "kafka" name automatically. Whichever
upgrade wins the name (claimed atomically via kubectl create) migrates
in place; a cluster that loses fails its hook fast and its next reconcile
retries. Once a migrated cluster sees other ZooKeeper clusters still
waiting, it frees the name: it stands up its own release-hashed
"b-<hash>" broker pool (node IDs pinned clear of the existing broker and
controller ranges), enables Cruise Control, drains every partition off
the "kafka"-pool brokers with a remove-brokers KafkaRebalance, deletes
the drained pool, and removes Cruise Control again. The next waiting
cluster's retry adopts the freed name; the last one to migrate keeps
"kafka". Everything happens inside the one namespace -- the hook's Role
is namespaced and it never reads or touches another.
Robustness details learned from live runs:
- The drain rebalance sets skipHardGoalCheck: true. A mandatory broker
evacuation must not be blocked by Cruise Control's capacity hard goals;
under a small resources preset the capacity model can briefly
over-estimate CPU from noisy samples and reject the proposal with
CpuCapacityGoal even when real CPU is near idle.
- The drain wait refreshes the proposal (strimzi.io/rebalance=refresh) on
any NotReady and keeps polling, rather than treating it as terminal: a
just-enabled Cruise Control is unreachable for a minute or two, and its
model settles as metric windows age out. This also makes the drain
resumable -- a re-run refreshes a stale rebalance instead of reading its
old failure forever.
- Cruise Control is enabled with explicit resources (it is a JVM and a
tenant LimitRange would OOM it at 128Mi), transiently -- kafka.yaml
never sets spec.cruiseControl, so Helm's server-side apply does not
fight the hook patching it on and off.
Every phase is idempotent so a Job killed mid-drain (the drain scales
with data size and can run for hours) resumes on retry; the Job deadline
and the release's helm-install-timeout are raised accordingly.
Verified on a live cluster across five runs of two co-namespaced
ZooKeeper Kafkas migrating to KRaft: both reach KRaft automatically, the
first drains onto its b-<hash> pool and frees the name, the second adopts
"kafka" in place (broker node IDs and PVCs unchanged), and each cluster's
data is byte-for-byte identical before and after. helm-unittest 87/87 and
hack/kafka-migration-hook.bats 14/14 (the real hook script against a stub
kubectl) cover the freeing decision, the atomic name claim, the drain's
refresh/skip-hard-goal handling, and the "delete the pool only after the
rebalance is Ready" guard.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Alexey Artamonov
cd9f4d6a27