The pre-install/pre-upgrade hook applies the node pools and the drain
KafkaRebalance with kubectl, and its container has no resources of its
own, so in a tenant with resource quotas it runs under the LimitRange
default of 128Mi. When it applies an object that already exists,
kubectl downloads the cluster's whole OpenAPI v2 document only to
decide that a custom resource takes a JSON merge patch: with the 374
CRDs of an e2e cluster that peaks at about 300Mi, so every upgrade of a
Kafka in such a tenant OOM-kills the hook and fails with "pre-upgrade
hooks failed".
--openapi-patch=false skips that lookup and builds the same three-way
merge patch from the last-applied annotation, peaking at about 35Mi on
the same cluster. A server-side apply would be as small but changes
what gets removed: the "kafka" pool is created with "kubectl create",
whose manager keeps co-owning every field it set, so a later
server-side apply leaves those fields in place when the chart drops
them. A larger memory limit would charge every upgrade against the
tenant quota instead.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The shell translator rewrites test bodies and loses Bats reporting.
The unit lane needs complete TAP results and JUnit reports without
changing how live-cluster suites run.
Sourced production helpers still need a dash pass, and unit test bodies
need nounset without exporting shell options into subprocesses.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The "free the kafka pool" gate counted other ZooKeeper clusters through a
`kubectl ... | awk` pipeline under `set -eu` without `pipefail`, its
kubectl error dropped by `2>/dev/null`. A failed read then counted zero and
kept the shared pool, stranding any co-namespaced cluster still waiting for
the name. Separate the read status from the count, mirroring the drain read
below, and cover it with a bats case.
Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Extends the ZooKeeper->KRaft migration hook so that N Kafka clusters
sharing one namespace all reach KRaft automatically, with no manual
steps. This closes the last gap in the migration: previously only one
cluster per namespace could migrate in place (the broker pool must be
named exactly "kafka" to adopt the existing <cluster>-kafka-N brokers,
and that name is namespace-unique), so a second co-namespaced ZooKeeper
cluster hit the render guard and its HelmRelease never rendered again --
stuck on ZooKeeper with no way forward once a future Strimzi drops the
migration.
The clusters now serialize on the "kafka" name automatically. Whichever
upgrade wins the name (claimed atomically via kubectl create) migrates
in place; a cluster that loses fails its hook fast and its next reconcile
retries. Once a migrated cluster sees other ZooKeeper clusters still
waiting, it frees the name: it stands up its own release-hashed
"b-<hash>" broker pool (node IDs pinned clear of the existing broker and
controller ranges), enables Cruise Control, drains every partition off
the "kafka"-pool brokers with a remove-brokers KafkaRebalance, deletes
the drained pool, and removes Cruise Control again. The next waiting
cluster's retry adopts the freed name; the last one to migrate keeps
"kafka". Everything happens inside the one namespace -- the hook's Role
is namespaced and it never reads or touches another.
Robustness details learned from live runs:
- The drain rebalance sets skipHardGoalCheck: true. A mandatory broker
evacuation must not be blocked by Cruise Control's capacity hard goals;
under a small resources preset the capacity model can briefly
over-estimate CPU from noisy samples and reject the proposal with
CpuCapacityGoal even when real CPU is near idle.
- The drain wait refreshes the proposal (strimzi.io/rebalance=refresh) on
any NotReady and keeps polling, rather than treating it as terminal: a
just-enabled Cruise Control is unreachable for a minute or two, and its
model settles as metric windows age out. This also makes the drain
resumable -- a re-run refreshes a stale rebalance instead of reading its
old failure forever.
- Cruise Control is enabled with explicit resources (it is a JVM and a
tenant LimitRange would OOM it at 128Mi), transiently -- kafka.yaml
never sets spec.cruiseControl, so Helm's server-side apply does not
fight the hook patching it on and off.
Every phase is idempotent so a Job killed mid-drain (the drain scales
with data size and can run for hours) resumes on retry; the Job deadline
and the release's helm-install-timeout are raised accordingly.
Verified on a live cluster across five runs of two co-namespaced
ZooKeeper Kafkas migrating to KRaft: both reach KRaft automatically, the
first drains onto its b-<hash> pool and frees the name, the second adopts
"kafka" in place (broker node IDs and PVCs unchanged), and each cluster's
data is byte-for-byte identical before and after. helm-unittest 87/87 and
hack/kafka-migration-hook.bats 14/14 (the real hook script against a stub
kubectl) cover the freeing decision, the atomic name claim, the drain's
refresh/skip-hard-goal handling, and the "delete the pool only after the
rebalance is Ready" guard.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Two live-reproduced blockers in the in-place ZK->KRaft migration are
fixed, plus the concurrency guard behind them.
1. Data loss (node IDs). The migration created the broker and controller
KafkaNodePools together without pinning node IDs. On Strimzi 0.45 the
controller pool deterministically took the existing broker IDs
[0..N-1], so the brokers were recreated empty at [N..] and their data
PVCs were orphaned (reproduced 2/2 on dev9: 20000 messages -> 0,
partitions leaderless, migration wedged in KRaftMigration). Naming the
broker pool "kafka" is necessary but NOT sufficient: Strimzi adopts by
node id, not pool name. The controller pool now pins
strimzi.io/next-node-ids above the broker range, so the broker pool
keeps [0..N-1] and adopts the existing <cluster>-kafka-N brokers and
their data in place.
2. Release stuck "failed" (Helm lifecycle). The pre-upgrade hook creates
the pools, and the chart also declared them as regular manifests, so
Helm's main apply tried to reconcile a resource that exists in the
cluster but was absent from the previous (ZooKeeper) release revision
and aborted with "original object KafkaNodePool ... not found",
leaving the HelmRelease failed even though the cluster migrated fine
(reproduced 3/3 under server-side apply, the production
helm-controller path). This is unavoidable while the pools are Helm
main-manifest resources across the ZK->KRaft boundary, so the pools
are now managed entirely by the migration Job (a pre-install,
pre-upgrade hook) on every install and upgrade, and removed from the
regular manifests. The Job ensures both pools (creating or updating
them idempotently), then drives the migration state machine.
templates/kafkanodepools.yaml and the deployed-version gate ConfigMap
(templates/version.yaml) are removed; the broker pool name is resolved
per path by kafka.brokerPoolName ("b-<hash>" fresh, "kafka"
migrating), and the render-time name-length guard moves to the Kafka
CR template.
3. Concurrency (one migration per namespace). The collision guard read
the "kafka" pool owner and then applied, a non-atomic read-then-write
two concurrent per-namespace migrations could both pass. When the pool
is absent the hook now uses "kubectl create" as the atomic arbiter:
exactly one concurrent creator wins, the loser re-reads the owner and
aborts.
Verified live on dev9 under server-side apply: a fresh install comes up
KRaft with pools at [0,1,2]/[3,4,5] and the release is "deployed"; an
in-place migration of a 20000-message ZooKeeper cluster ends "deployed",
adopts the broker PVCs in place (same UIDs), and the KRaft data is
byte-for-byte identical to the pre-migration ZooKeeper data (matching
sha256 over every partition:offset:key:value). helm-unittest 86/86 and
hack/kafka-migration-hook.bats 12/12 (the real hook script against a stub
kubectl) cover the resolution branches, the fail-closed reads, the atomic
collision guard and the finalize guard.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Fresh KRaft clusters no longer need to be alone in a namespace. Their
node pools get short, release-hashed names — "b-<hash>" for brokers,
"c-<hash>" for controllers — instead of the fixed "kafka"/"controller".
Each release hashes differently, so the names are unique within the
namespace (KafkaNodePool object names are namespace-unique) and any
number of Kafka instances coexist. The hash keeps the pool name a fixed
length: Strimzi derives every pod/PVC name as <cluster>-<pool>-<id> and
the cluster name is already the release name, so embedding the release
name in the pool name too would double it into every pod hostname and
blow the 63-character DNS label limit for long release names. A
render-time guard fails with a clear message when even the hashed
controller node name would still exceed 63 characters.
The fixed "kafka" name is still required to adopt an existing ZooKeeper
cluster's <cluster>-kafka-N brokers and data in place during migration,
so kafka.brokerPoolName resolves to "kafka" for a migrating/migrated
cluster and "b-<hash>" for a fresh one. A second ZooKeeper->KRaft
migration into a namespace that already holds a "kafka" pool owned by
another cluster fails closed, both at render time and in the pre-upgrade
hook (whose owner probe uses --ignore-not-found + an explicit exit-code
check, so a transient kubectl error aborts rather than silently waiving
the collision guard). This replaces the previous blanket "one Kafka per
namespace" guard, which over-restricted the fresh case.
The pool names are resolved from a live Helm lookup; that is safe because
Helm propagates any non-NotFound read error and fails the whole render
(pkg/engine/lookup_func.go), so a transient apiserver/RBAC miss retries
rather than silently flipping a live cluster's pool name — an empty
lookup means genuine absence or a never-applied client-side render.
The migration hook and workload monitors follow the same naming. Tests:
helm-unittest fixtures (via kubernetesProvider) cover every resolution
branch and the fail-closed guards; hack/kafka-migration-hook.bats drives
the migration Job's real shell script against a stub kubectl across the
fresh / read-error / already-KRaft / owner-collision / happy-path /
stuck / retry cases. Verified on a live cluster: in-place ZooKeeper->KRaft
migration preserved all data and adopted the original broker PVCs, and
two fresh clusters coexist with distinct pools, pods and PVCs.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>