Files
cozystack/packages
Alexey Artamonov cb7578bb31 fix(kafka): data-safe, Helm-clean ZooKeeper->KRaft migration
Two live-reproduced blockers in the in-place ZK->KRaft migration are
fixed, plus the concurrency guard behind them.

1. Data loss (node IDs). The migration created the broker and controller
   KafkaNodePools together without pinning node IDs. On Strimzi 0.45 the
   controller pool deterministically took the existing broker IDs
   [0..N-1], so the brokers were recreated empty at [N..] and their data
   PVCs were orphaned (reproduced 2/2 on dev9: 20000 messages -> 0,
   partitions leaderless, migration wedged in KRaftMigration). Naming the
   broker pool "kafka" is necessary but NOT sufficient: Strimzi adopts by
   node id, not pool name. The controller pool now pins
   strimzi.io/next-node-ids above the broker range, so the broker pool
   keeps [0..N-1] and adopts the existing <cluster>-kafka-N brokers and
   their data in place.

2. Release stuck "failed" (Helm lifecycle). The pre-upgrade hook creates
   the pools, and the chart also declared them as regular manifests, so
   Helm's main apply tried to reconcile a resource that exists in the
   cluster but was absent from the previous (ZooKeeper) release revision
   and aborted with "original object KafkaNodePool ... not found",
   leaving the HelmRelease failed even though the cluster migrated fine
   (reproduced 3/3 under server-side apply, the production
   helm-controller path). This is unavoidable while the pools are Helm
   main-manifest resources across the ZK->KRaft boundary, so the pools
   are now managed entirely by the migration Job (a pre-install,
   pre-upgrade hook) on every install and upgrade, and removed from the
   regular manifests. The Job ensures both pools (creating or updating
   them idempotently), then drives the migration state machine.
   templates/kafkanodepools.yaml and the deployed-version gate ConfigMap
   (templates/version.yaml) are removed; the broker pool name is resolved
   per path by kafka.brokerPoolName ("b-<hash>" fresh, "kafka"
   migrating), and the render-time name-length guard moves to the Kafka
   CR template.

3. Concurrency (one migration per namespace). The collision guard read
   the "kafka" pool owner and then applied, a non-atomic read-then-write
   two concurrent per-namespace migrations could both pass. When the pool
   is absent the hook now uses "kubectl create" as the atomic arbiter:
   exactly one concurrent creator wins, the loser re-reads the owner and
   aborts.

Verified live on dev9 under server-side apply: a fresh install comes up
KRaft with pools at [0,1,2]/[3,4,5] and the release is "deployed"; an
in-place migration of a 20000-message ZooKeeper cluster ends "deployed",
adopts the broker PVCs in place (same UIDs), and the KRaft data is
byte-for-byte identical to the pre-migration ZooKeeper data (matching
sha256 over every partition:offset:key:value). helm-unittest 86/86 and
hack/kafka-migration-hook.bats 12/12 (the real hook script against a stub
kubectl) cover the resolution branches, the fail-closed reads, the atomic
collision guard and the finalize guard.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
..