mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 05:25:14 +09:00
Alexey Artamonov
cd9f4d6a27
Extends the ZooKeeper->KRaft migration hook so that N Kafka clusters sharing one namespace all reach KRaft automatically, with no manual steps. This closes the last gap in the migration: previously only one cluster per namespace could migrate in place (the broker pool must be named exactly "kafka" to adopt the existing <cluster>-kafka-N brokers, and that name is namespace-unique), so a second co-namespaced ZooKeeper cluster hit the render guard and its HelmRelease never rendered again -- stuck on ZooKeeper with no way forward once a future Strimzi drops the migration. The clusters now serialize on the "kafka" name automatically. Whichever upgrade wins the name (claimed atomically via kubectl create) migrates in place; a cluster that loses fails its hook fast and its next reconcile retries. Once a migrated cluster sees other ZooKeeper clusters still waiting, it frees the name: it stands up its own release-hashed "b-<hash>" broker pool (node IDs pinned clear of the existing broker and controller ranges), enables Cruise Control, drains every partition off the "kafka"-pool brokers with a remove-brokers KafkaRebalance, deletes the drained pool, and removes Cruise Control again. The next waiting cluster's retry adopts the freed name; the last one to migrate keeps "kafka". Everything happens inside the one namespace -- the hook's Role is namespaced and it never reads or touches another. Robustness details learned from live runs: - The drain rebalance sets skipHardGoalCheck: true. A mandatory broker evacuation must not be blocked by Cruise Control's capacity hard goals; under a small resources preset the capacity model can briefly over-estimate CPU from noisy samples and reject the proposal with CpuCapacityGoal even when real CPU is near idle. - The drain wait refreshes the proposal (strimzi.io/rebalance=refresh) on any NotReady and keeps polling, rather than treating it as terminal: a just-enabled Cruise Control is unreachable for a minute or two, and its model settles as metric windows age out. This also makes the drain resumable -- a re-run refreshes a stale rebalance instead of reading its old failure forever. - Cruise Control is enabled with explicit resources (it is a JVM and a tenant LimitRange would OOM it at 128Mi), transiently -- kafka.yaml never sets spec.cruiseControl, so Helm's server-side apply does not fight the hook patching it on and off. Every phase is idempotent so a Job killed mid-drain (the drain scales with data size and can run for hours) resumes on retry; the Job deadline and the release's helm-install-timeout are raised accordingly. Verified on a live cluster across five runs of two co-namespaced ZooKeeper Kafkas migrating to KRaft: both reach KRaft automatically, the first drains onto its b-<hash> pool and frees the name, the second adopts "kafka" in place (broker node IDs and PVCs unchanged), and each cluster's data is byte-for-byte identical before and after. helm-unittest 87/87 and hack/kafka-migration-hook.bats 14/14 (the real hook script against a stub kubectl) cover the freeing decision, the atomic name claim, the drain's refresh/skip-hard-goal handling, and the "delete the pool only after the rebalance is Ready" guard. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>