feat(platform): move Strimzi resources to the v1 API before Strimzi 1.x

Strimzi 1.0 removed v1beta2 from its CRDs, and the API server refuses
a CRD that drops a version its status still lists as stored. Every
stored object has to be rewritten as v1 before the new operator
arrives, which needs the CRDs of Strimzi 0.49 to 0.51 that serve both
versions and Strimzi's own conversion tool. The 0.45 operator is
stopped first: Strimzi supports the conversion only under an operator
that knows v1, and 0.45 keeps rewriting the Kafka status through
v1beta2. The conversion Job's ClusterRole can patch CRDs and write
ConfigMaps in every namespace, so it goes with its binding and
ServiceAccount once the Job has finished, successfully or not; a
failed Job stays for diagnosis, and a Job still running when the wait
times out keeps them, because the retry waits for that Job.

Two references to v1beta2 outlive the stored objects. Helm builds the
objects of a release before it upgrades or uninstalls it, so every
stored revision of every Kafka application is rewritten to v1. And the
API server refuses server-side apply to an object whose managedFields
still name a version the CRD no longer serves; when helm-controller
recovers by wiping them, a field a later chart drops is never removed.
Those entries move to v1 as well. A Helm action already running stores
the releases it read when it finishes, v1beta2 included, so the version
is not stamped while any Kafka release revision is still pending.

Strimzi 1.x also runs neither ZooKeeper nor Kafka 3.x. A ZooKeeper
cluster, or a shared broker pool still being drained, left by a
platform that moved past v1.7 before its ZooKeeper to KRaft migration
finished cannot be finished afterwards, so the migration stops
instead; Kafka applications on v3.8 or v3.9 move to v4.3. The
migration also stops, before it changes anything, while a Kafka
application topic or a KafkaTopic sets a config Kafka 4 removed or a
segment.bytes below the 1 MiB minimum of Kafka 4: the new kafka chart
refuses to render such a topic, and a Kafka 4 broker whose log holds
the small segment.bytes does not start. segment.bytes is read the way
Kafka reads it: trimmed, with an optional sign, leading zeros decimal.

Strimzi's conversion tool refuses some v1beta2 objects outright, and
convert-resource stops at the first one, with the operators already
stopped. While v1beta2 is served the migration lists every Strimzi
kind first and refuses, naming each object and the reason, before it
stops anything. The list follows the 0.51.0 tool the conversion Job
runs, and a test fails when that image changes.

The conversion deletes the Kafka's KRaft metadata state, and KRaft is
the state Strimzi never leaves. Before it stops the operators, the
migration records an annotation on every Kafka it accepts, so a retry
after the conversion still accepts a NotReady Kafka that passed on
its KRaft state.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
This commit is contained in:
Aleksei Sviridkin committed 2026-11-17 12:02:45 +03:00
1 parent 5a5d5cd3d8
commit af0db99c71
12 files changed
+2096 -1

No files matched your search

File diff suppressed because it is too large. Load diff
+4
View File
@@ -0,0 +1,4 @@
tenant-a one v3.9
tenant-b two v3.8
tenant-c three v4.2
tenant-d four
@@ -0,0 +1,2 @@
tenant-a kafka-one
tenant-b
@@ -0,0 +1,4 @@
{"items":[
{"metadata":{"namespace":"tenant-a","name":"kafka-one","annotations":{"strimzi.io/kraft":"enabled"}},"spec":{"kafka":{"version":"3.9.0"}},"status":{"kafkaMetadataState":"KRaft","kafkaMetadataVersion":"3.9-IV0","conditions":[{"type":"Ready","status":"True"}]}},
{"metadata":{"namespace":"tenant-b","name":"kafka-two"},"spec":{"kafka":{"version":"3.8.1"}},"status":{"kafkaMetadataState":"KRaft","kafkaMetadataVersion":"3.9-IV0","conditions":[{"type":"Ready","status":"True"}]}}
]}
+5
View File
@@ -0,0 +1,5 @@
{"items":[
{"metadata":{"namespace":"tenant-a","name":"kafka-one"},"spec":{"kafka":{"version":"3.9.0"}},"status":{"kafkaMetadataState":"KRaft","kafkaMetadataVersion":"3.9-IV0","conditions":[{"type":"Ready","status":"True"}]}},
{"metadata":{"namespace":"tenant-c","name":"kafka-zk"},"spec":{"kafka":{"version":"3.9.0"},"zookeeper":{"replicas":3}},"status":{"kafkaMetadataState":"ZooKeeper"}},
{"metadata":{"namespace":"tenant-d","name":"kafka-mid"},"spec":{"kafka":{"version":"3.9.0"}},"status":{"kafkaMetadataState":"KRaftPostMigration"}}
]}
+342
View File
@@ -0,0 +1,342 @@
#!/bin/sh
# Fake kubectl for hack/migration-61-strimzi-v1-conversion.bats. It serves only
# the calls migration 61 and lib/cozystack-version.sh make, and logs each one
# to $FAKE_CMDLOG.
#
# FAKE_CMDLOG file to append a one-line record of each call to
# FAKE_STATE directory; its "stored" file holds the storedVersions
# every Strimzi CRD reports, and the conversion Job
# rewrites it to ["v1"] when it succeeds
# FAKE_NO_STRIMZI non-empty => the Kafka CRD is absent
# FAKE_KAFKAS fixture served for the Strimzi Kafka list; annotations
# written to it are kept in $FAKE_STATE
# FAKE_JOB succeed (default) | fail | noop (succeeds, converts nothing)
# | hang (neither succeeds nor fails)
# | nopod (Failed condition, no failed pod: the pod was
# never created)
# FAKE_PULL non-empty => the conversion pod cannot pull its image
# FAKE_PATCH_FAIL non-empty => every application patch exits 1
# FAKE_CRD_READ_FAIL, FAKE_APIRES_FAIL, FAKE_SECRET_LIST_FAIL: that read
# exits 1
# FAKE_APPS_ABSENT non-empty => the apps.cozystack.io API is not installed
# FAKE_DISCOVERY_PARTIAL non-empty => another API group fails discovery, so
# api-resources exits 1 after listing what it found
# FAKE_RELEASE_REWRITTEN_BACK: release patches do not stick
# FAKE_RELEASE_PENDING: revision v4 of kafka-one is pending-upgrade, a
# Helm action that has not finished
# FAKE_REBALANCES comma-separated ns/name served for the KafkaRebalance list
# FAKE_HELMRELEASES fixture served for the Kafka application HelmRelease
# list; none by default, and a missing file fails the read
# FAKE_KAFKATOPICS fixture served for the KafkaTopic list, the same way
# FAKE_STRIMZI_OBJECTS directory; <plural>.json in it is served for the JSON
# list of that Strimzi kind other than Kafka and
# KafkaTopic, an empty list when it is absent
# FAKE_STRIMZI_LIST_FAIL plural of one such kind whose JSON list read exits 1
# The deployed Helm release of kafka-one is release-kafka-one.json, encoded
# the way Helm stores it; a patch of its Secret is written to
# $FAKE_STATE/release, so a second run reads what the first one wrote.
set -u
log() { [ -n "${FAKE_CMDLOG:-}" ] && printf '%s\n' "$*" >> "$FAKE_CMDLOG"; }
log "KUBECTL $*"
args="$*"
# The Kafka list as it stands: converted, annotated, or the fixture.
kafkas_src() {
for f in "$FAKE_STATE/kafkas-converted.json" "$FAKE_STATE/kafkas-live.json"; do
[ -f "$f" ] && { printf '%s' "$f"; return; }
done
printf '%s' "$FAKE_KAFKAS"
}
arg_after() { k="$1"; shift; p=""; for a in "$@"; do [ "$p" = "$k" ] && { printf '%s' "$a"; return; }; p="$a"; done; }
case "$args" in
*"apply --filename -"*)
manifest=$(cat)
v=$(printf '%s\n' "$manifest" | sed -n 's/^ version: "\{0,1\}\([0-9]\{1,\}\)"\{0,1\}$/\1/p' | head -1)
log "STAMP ${v:-unknown}"
exit 0
;;
*"apply -f -"*)
manifest=$(cat)
case "$manifest" in
*"kind: Job"*)
order=$(printf '%s\n' "$manifest" | sed -n 's/.*v1-api-conversion.sh", "\([a-z-]*\)".*/\1/p' | tr '\n' ' ')
log "APPLY-CONVERSION ${order% }"
log "CONVERSION-DEADLINE $(printf '%s\n' "$manifest" | sed -n 's/^ activeDeadlineSeconds: \([0-9]*\)$/\1/p')"
exit 0
;;
esac
log "APPLY-UNKNOWN"
exit 1
;;
*"apply --server-side"*)
log "APPLY-CRDS $(arg_after -f "$@")"
exit 0
;;
*"get crd kafkas.kafka.strimzi.io --ignore-not-found -o name"*)
[ -n "${FAKE_NO_STRIMZI:-}" ] || echo "customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io"
exit 0
;;
*"get crd kafkas.kafka.strimzi.io -o jsonpath={.spec.versions[*].name}"*)
printf '%s' "${FAKE_SERVED:-v1 v1beta2}"
exit 0
;;
*"get crd "*"storedVersions"*)
if [ -n "${FAKE_CRD_READ_FAIL:-}" ]; then echo "Error from server: etcdserver: request timed out" >&2; exit 1; fi
cat "$FAKE_STATE/stored"
exit 0
;;
*" annotate kafkas.v1beta2.kafka.strimzi.io "*|*" annotate kafkas.v1.kafka.strimzi.io "*)
ns=$(arg_after -n "$@"); kind=$(arg_after annotate "$@"); name=$(arg_after "$kind" "$@")
kv=""; for a in "$@"; do case "$a" in cozystack.io/*=*) kv="$a" ;; esac; done
log "ANNOTATE $ns/$name $kv"
src=$(kafkas_src); dst="$FAKE_STATE/kafkas-live.json"
[ "$src" = "$FAKE_STATE/kafkas-converted.json" ] && dst="$src"
jq --arg ns "$ns" --arg name "$name" --arg k "${kv%%=*}" --arg v "${kv#*=}" \
'.items |= map(if .metadata.namespace == $ns and .metadata.name == $name then .metadata.annotations[$k] = $v else . end)' \
"$src" > "$dst.new" && mv "$dst.new" "$dst"
exit 0
;;
*"get "*".kafka.strimzi.io --all-namespaces -o json"|*"get strimzipodsets."*".core.strimzi.io --all-namespaces -o json")
resource=$(arg_after get "$@")
case "${resource%%.*}" in
kafkas|kafkatopics) ;;
*)
if [ "${resource%%.*}" = kafkarebalances ] && [ -n "${FAKE_REBALANCE_FAIL:-}" ]; then echo "Error from server: etcdserver: request timed out" >&2; exit 1; fi
if [ "${resource%%.*}" = "${FAKE_STRIMZI_LIST_FAIL:-}" ]; then echo "Error from server: etcdserver: request timed out" >&2; exit 1; fi
f="${FAKE_STRIMZI_OBJECTS:-/nonexistent}/${resource%%.*}.json"
if [ -f "$f" ]; then cat "$f"; else echo '{"items":[]}'; fi
exit 0
;;
esac
;;
esac
case "$args" in
*"get kafkarebalances.v1beta2.kafka.strimzi.io"*|*"get kafkarebalances.v1.kafka.strimzi.io"*)
if [ -n "${FAKE_REBALANCE_FAIL:-}" ]; then echo "Error from server: etcdserver: request timed out" >&2; exit 1; fi
[ -z "${FAKE_REBALANCES:-}" ] || printf '%s\n' "$FAKE_REBALANCES" | tr ',' '\n'
exit 0
;;
*"get helmreleases.helm.toolkit.fluxcd.io --all-namespaces -l apps.cozystack.io/application.kind=Kafka -o json"*)
if [ -n "${FAKE_HELMRELEASES:-}" ]; then cat "$FAKE_HELMRELEASES" || exit 1; else echo '{"items":[]}'; fi
exit 0
;;
*"get kafkatopics.v1beta2.kafka.strimzi.io --all-namespaces -o json"*|*"get kafkatopics.v1.kafka.strimzi.io --all-namespaces -o json"*)
if [ -n "${FAKE_KAFKATOPICS:-}" ]; then cat "$FAKE_KAFKATOPICS" || exit 1; else echo '{"items":[]}'; fi
exit 0
;;
*"get kafkas.v1beta2.kafka.strimzi.io"*)
if [ -n "${FAKE_KAFKA_LIST_FAIL:-}" ]; then echo "Error from server: Kafka list failed" >&2; exit 1; fi
case "${FAKE_SERVED:-v1 v1beta2}" in v1) echo "v1beta2 is not served" >&2; exit 1 ;; esac
cat "$(kafkas_src)"
exit 0
;;
*"get kafkas.v1.kafka.strimzi.io --all-namespaces -o json"|*"get kafkas.v1.kafka.strimzi.io --all-namespaces --output json")
if [ -n "${FAKE_KAFKA_LIST_FAIL:-}" ]; then echo "Error from server: Kafka list failed" >&2; exit 1; fi
cat "$(kafkas_src)"
exit 0
;;
*"get kafkas.v1.kafka.strimzi.io"*)
cat "$FAKE_RELEASES"
exit 0
;;
*"get secret -l owner=helm,name=kafka-one -o"*)
if [ -n "${FAKE_SECRET_LIST_FAIL:-}" ]; then echo "Error from server (Forbidden): secrets is forbidden" >&2; exit 1; fi
# v4 stands for a failed upgrade: still the latest revision, so uninstall
# builds it.
echo "sh.helm.release.v1.kafka-one.v3 sh.helm.release.v1.kafka-one.v4"
exit 0
;;
*"get secret -l owner=helm,name=kafka-failed-install -o"*)
echo "sh.helm.release.v1.kafka-failed-install.v1"
exit 0
;;
*"get secret sh.helm.release.v1.kafka-failed-install.v1 -o jsonpath={.data.release}"*)
if [ ! -f "$FAKE_STATE/release-failed" ]; then
gzip -c "$(dirname "$0")/release-kafka-one.json" | base64 | tr -d '\n' | base64 | tr -d '\n' > "$FAKE_STATE/release-failed"
fi
cat "$FAKE_STATE/release-failed"
exit 0
;;
*"patch secret sh.helm.release.v1.kafka-failed-install.v1 --type merge --patch-file "*)
jq -r '.data.release' "$(arg_after --patch-file "$@")" > "$FAKE_STATE/release-failed"
log "PATCH-RELEASE tenant-f/sh.helm.release.v1.kafka-failed-install.v1"
exit 0
;;
*"get deployment --all-namespaces -l app.kubernetes.io/name=entity-operator"*)
[ -n "${FAKE_NO_OPERATOR:-}" ] || echo "tenant-a kafka-one-entity-operator"
exit 0
;;
*"scale deployment kafka-one-entity-operator --replicas=0"*)
log "SCALE-ENTITY tenant-a/kafka-one-entity-operator"
exit 0
;;
*"get deployment kafka-one-entity-operator -o json"*)
echo '{"spec":{"selector":{"matchLabels":{"strimzi.io/cluster":"kafka-one","strimzi.io/name":"kafka-one-entity-operator"}}}}'
exit 0
;;
*"wait --for=delete pod -l strimzi.io/"*)
log "WAIT-ENTITY $(arg_after -n "$@")/$(arg_after -l "$@")"
exit 0
;;
*"get secret -l owner=helm,name="*)
exit 0
;;
*"get secret sh.helm.release.v1.kafka-one.v"*" -o jsonpath={.data.release}"*)
rev=$(arg_after secret "$@"); rev=${rev##*.}
if [ ! -f "$FAKE_STATE/release-$rev" ]; then
status=deployed
[ "$rev" != v4 ] || [ -z "${FAKE_RELEASE_PENDING:-}" ] || status=pending-upgrade
jq -c --arg s "$status" '.info.status = $s' "$(dirname "$0")/release-kafka-one.json" |
gzip -c | base64 | tr -d '\n' | base64 | tr -d '\n' > "$FAKE_STATE/release-$rev"
fi
cat "$FAKE_STATE/release-$rev"
exit 0
;;
*"patch secret sh.helm.release.v1.kafka-one.v"*" --type merge --patch-file "*)
rev=$(arg_after secret "$@"); rev=${rev##*.}
# FAKE_RELEASE_REWRITTEN_BACK: a Helm upgrade that was already running
# stores its v1beta2 revision over the rewrite.
[ -n "${FAKE_RELEASE_REWRITTEN_BACK:-}" ] || jq -r '.data.release' "$(arg_after --patch-file "$@")" > "$FAKE_STATE/release-$rev"
log "PATCH-RELEASE tenant-a/sh.helm.release.v1.kafka-one.$rev"
exit 0
;;
*"get deployment strimzi-cluster-operator --ignore-not-found -o name"*)
if [ -n "${FAKE_OPERATOR_READ_FAIL:-}" ]; then echo "Error from server: etcdserver: request timed out" >&2; exit 1; fi
[ -n "${FAKE_NO_OPERATOR:-}" ] || echo "deployment.apps/strimzi-cluster-operator"
exit 0
;;
*"scale deployment strimzi-cluster-operator --replicas=0"*)
log "SCALE-OPERATOR 0"
exit 0
;;
*"wait --for=delete pod -l name=strimzi-cluster-operator"*)
exit 0
;;
*"get kafkas.kafka.strimzi.io --all-namespaces --show-managed-fields -o json"*)
if [ -f "$FAKE_STATE/managed" ]; then cat "$FAKE_STATE/managed"; else cat "$(dirname "$0")/managed-kafkas.json"; fi
exit 0
;;
*"--all-namespaces --show-managed-fields -o json"*)
echo '{"items":[]}'
exit 0
;;
*"get kafkas.kafka.strimzi.io "*" --show-managed-fields -o json"*)
ns=$(arg_after -n "$@"); name=$(arg_after kafkas.kafka.strimzi.io "$@")
src="$FAKE_STATE/managed"; [ -f "$src" ] || src="$(dirname "$0")/managed-kafkas.json"
jq --arg ns "$ns" --arg n "$name" '.items[] | select(.metadata.namespace == $ns and .metadata.name == $n)' "$src"
exit 0
;;
*"patch kafkas.kafka.strimzi.io "*"--patch-file "*)
ns=$(arg_after -n "$@"); name=$(arg_after kafkas.kafka.strimzi.io "$@")
patch=$(arg_after --patch-file "$@")
src="$FAKE_STATE/managed"; [ -f "$src" ] || src="$(dirname "$0")/managed-kafkas.json"
rv=$(jq -r '.metadata.resourceVersion' "$patch")
log "PATCH-MANAGED $ns/$name rv=$rv $(jq -r '[.metadata.managedFields[] | "\(.manager)=\(.apiVersion)"] | join(",")' "$patch")"
# FAKE_MANAGED_CONFLICT: a concurrent write bumps the object before the
# first patch lands, so that patch carries a stale resourceVersion.
if [ -n "${FAKE_MANAGED_CONFLICT:-}" ] && [ ! -f "$FAKE_STATE/bumped" ]; then
: > "$FAKE_STATE/bumped"
jq --arg ns "$ns" --arg n "$name" '(.items[] | select(.metadata.namespace == $ns and .metadata.name == $n) | .metadata.resourceVersion) |= (tonumber + 1 | tostring)' "$src" > "$FAKE_STATE/managed.new" && mv "$FAKE_STATE/managed.new" "$FAKE_STATE/managed"
fi
if [ -n "${FAKE_MANAGED_PATCH_FAIL:-}" ]; then echo "Error from server (Forbidden): forbidden" >&2; exit 1; fi
src="$FAKE_STATE/managed"; [ -f "$src" ] || src="$(dirname "$0")/managed-kafkas.json"
cur=$(jq -r --arg ns "$ns" --arg n "$name" '.items[] | select(.metadata.namespace == $ns and .metadata.name == $n) | .metadata.resourceVersion' "$src")
if [ "$cur" != "$rv" ]; then
echo "Error from server (Conflict): Operation cannot be fulfilled on kafkas.kafka.strimzi.io \"$name\": the object has been modified; please apply your changes to the latest version and try again" >&2
exit 1
fi
jq --arg ns "$ns" --arg n "$name" --slurpfile p "$patch" \
'(.items[] | select(.metadata.namespace == $ns and .metadata.name == $n) | .metadata) |= (.managedFields = $p[0].metadata.managedFields | .resourceVersion = (.resourceVersion | tonumber + 1 | tostring))' \
"$src" > "$FAKE_STATE/managed.new" && mv "$FAKE_STATE/managed.new" "$FAKE_STATE/managed"
exit 0
;;
*"wait --for=condition=Established"*)
exit 0
;;
*"get job strimzi-v1-api-conversion --ignore-not-found -o jsonpath={.metadata.name} {.status.succeeded}{.status.failed}"*)
# FAKE_OLD_JOB: a Job an earlier attempt left behind, running, succeeded
# or failed.
case "${FAKE_OLD_JOB:-}" in
running) printf 'strimzi-v1-api-conversion ' ;;
succeeded|failed) printf 'strimzi-v1-api-conversion 1' ;;
esac
exit 0
;;
*"get job strimzi-v1-api-conversion -o jsonpath={.status.succeeded}"*)
case "${FAKE_JOB:-succeed}" in
succeed)
if [ -n "${FAKE_CONVERSION_CHANGES_SPEC:-}" ] && [ ! -f "$FAKE_STATE/kafkas-converted.json" ]; then
jq '.items |= map(
.apiVersion = "kafka.strimzi.io/v1"
| .metadata.generation += 1
| del(.spec.kafka.replicas, .spec.kafka.storage)
| del(.status.kafkaMetadataState)
| if .spec.entityOperator.topicOperator.reconciliationIntervalSeconds != null then
.spec.entityOperator.topicOperator |=
(.reconciliationIntervalMs = (.reconciliationIntervalSeconds * 1000)
| del(.reconciliationIntervalSeconds))
else . end)' "$(kafkas_src)" > "$FAKE_STATE/kafkas-converted.json"
fi
printf '["v1"]' > "$FAKE_STATE/stored"
echo 1
;;
noop) echo 1 ;;
esac
exit 0
;;
*"get job strimzi-v1-api-conversion -o jsonpath={.status.failed}"*)
case "${FAKE_JOB:-succeed}" in
fail) printf '1' ;;
esac
case "$args" in
*'conditions[?(@.type=="Failed")]'*)
case "${FAKE_JOB:-succeed}" in fail|nopod) printf ' True' ;; esac
;;
esac
exit 0
;;
*"get pod -l batch.kubernetes.io/job-name=strimzi-v1-api-conversion -o jsonpath="*)
[ -z "${FAKE_PULL:-}" ] || printf ' ImagePullBackOff'
exit 0
;;
*" logs "*)
echo "fake conversion log"
exit 0
;;
"delete "*|*" delete "*)
cascade=""
case "$args" in *"--cascade=foreground"*) cascade=" foreground" ;; esac
log "DELETE $(arg_after delete "$@") $(arg_after "$(arg_after delete "$@")" "$@")$cascade"
exit 0
;;
*"api-resources --api-group=apps.cozystack.io"*)
echo "kafkas.apps.cozystack.io"
if [ -n "${FAKE_DISCOVERY_PARTIAL:-}" ]; then echo "error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" >&2; exit 1; fi
exit 0
;;
*"get --raw /apis/apps.cozystack.io/v1alpha1"*)
if [ -n "${FAKE_APIRES_FAIL:-}" ]; then echo "Error from server (ServiceUnavailable): the server is currently unable to handle the request" >&2; exit 1; fi
if [ -n "${FAKE_APPS_ABSENT:-}" ]; then echo "Error from server (NotFound): the server could not find the requested resource" >&2; exit 1; fi
echo '{"kind":"APIResourceList","groupVersion":"apps.cozystack.io/v1alpha1","resources":[{"name":"kafkas","kind":"Kafka"},{"name":"kafkas/status","kind":"Kafka"}]}'
exit 0
;;
*"get kafkas.apps.cozystack.io"*" kafka-"*)
# Release names of the applications; failed-install has no Kafka CR.
printf '%s\n' 'tenant-a kafka-one' 'tenant-f kafka-failed-install'
exit 0
;;
*"get kafkas.apps.cozystack.io"*)
cat "$FAKE_APPS"
exit 0
;;
*"patch kafkas.apps.cozystack.io"*)
if [ -n "${FAKE_PATCH_FAIL:-}" ]; then echo "Error from server (Conflict)" >&2; exit 1; fi
log "PATCH $(arg_after -n "$@")/$(arg_after kafkas.apps.cozystack.io "$@") $(arg_after -p "$@")"
exit 0
;;
esac
log "UNMODELLED"
printf 'fake kubectl: unmodelled call: %s\n' "$args" >&2
exit 1
@@ -0,0 +1,54 @@
{
"items": [
{
"metadata": {
"namespace": "tenant-a",
"name": "kafka-one",
"managedFields": [
{
"manager": "helm-controller",
"operation": "Apply",
"apiVersion": "kafka.strimzi.io/v1beta2",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:spec": {
"f:kafka": {
"f:version": {}
}
}
}
},
{
"manager": "strimzi-cluster-operator",
"operation": "Update",
"apiVersion": "kafka.strimzi.io/v1beta2",
"subresource": "status",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {}
}
}
],
"resourceVersion": "100"
}
},
{
"metadata": {
"namespace": "tenant-b",
"name": "kafka-two",
"managedFields": [
{
"manager": "helm-controller",
"operation": "Apply",
"apiVersion": "kafka.strimzi.io/v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:spec": {}
}
}
],
"resourceVersion": "101"
}
}
]
}
@@ -0,0 +1 @@
{"name":"kafka-one","version":3,"info":{"status":"deployed"},"manifest":"---\n# Source: kafka/templates/kafka.yaml\napiVersion: kafka.strimzi.io/v1beta2\nkind: Kafka\nmetadata:\n name: kafka-one\n---\n# Source: kafka/templates/topics.yaml\napiVersion: kafka.strimzi.io/v1beta2\nkind: KafkaTopic\nmetadata:\n name: kafka-one-orders\n---\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: kafka-one-metrics\n","hooks":[{"name":"kafka-one-kafka-migration","manifest":"apiVersion: batch/v1\nkind: Job\n# script applies apiVersion: kafka.strimzi.io/v1beta2\n"}]}
@@ -0,0 +1 @@
# Stand-in for strimzi-crds-0.51.0.yaml; the fake kubectl never reads it.
@@ -29,6 +29,14 @@ RUN set -eux; \
echo "${sha256} /usr/bin/etcd-migrate" | sha256sum -c -; \
chmod +x /usr/bin/etcd-migrate
# CRDs of Strimzi 0.51.0, the last release serving both v1beta2 and v1.
# Migration 61 applies them so Strimzi's conversion tool can move every stored
# object to v1 before the kafka-operator release installs the v1-only CRDs.
RUN mkdir /strimzi-crds && \
curl -fsSL -o /strimzi-crds/strimzi-crds-0.51.0.yaml \
https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.51.0/strimzi-crds-0.51.0.yaml && \
echo "a11257e67985a946a6facfb17d729aac5670b89675d410d13162fe455a5ac7a8 /strimzi-crds/strimzi-crds-0.51.0.yaml" | sha256sum -c -
COPY migrations /migrations
# v1alpha2 etcd CRDs, vendored into the build context by `make image-migrations`.
# migration 50 applies these before adoption so it does not deadlock waiting for
+661
View File
@@ -0,0 +1,661 @@
#!/bin/bash
# Migration 61 --> 62
# Move the Strimzi custom resources to the v1 API before Strimzi 1.x arrives.
#
# Strimzi 1.0 removed the v1beta2 API from its CRDs. The kafka-operator
# release replaces the CRDs on upgrade (upgradeCRDs: CreateReplace), and the
# API server refuses a CRD whose spec.versions no longer lists a version that
# status.storedVersions still records. So every stored object has to be
# rewritten as v1, and storedVersions narrowed to [v1], before the new
# operator arrives. That needs CRDs serving both versions, which only Strimzi
# 0.49 to 0.51 ship: the 0.51.0 CRDs are baked into this image (see the
# Dockerfile). The conversion itself is Strimzi's v1-api-conversion tool, run
# as a Job from the 0.51.0 operator image: convert-resource rewrites the
# objects, crd-upgrade switches the storage version and narrows
# storedVersions.
#
# Strimzi 0.46 also removed ZooKeeper. A ZooKeeper-based Kafka that did not
# reach KRaft on the release carrying the chart's ZooKeeper to KRaft
# migration cannot be run, converted or migrated by anything after this
# point, so this migration refuses to proceed while one is left.
#
# Strimzi 1.2 runs neither Kafka 3.8 nor 3.9, so every Kafka application on
# v3.8 or v3.9 is moved to v4.3. Until the new kafka chart arrives, the
# release renders with the old one, which rejects v4.3, and the HelmRelease
# reports a values error; no Helm action runs in that window.
#
# Ordering: this runs as the platform chart's pre-upgrade hook, before the
# upgrade re-points cozystack-packages, so on a variant that runs migrations
# neither the new operator nor the new kafka chart is rendered before it. The
# default variant renders no migration hook, and there these steps are manual.
#
# Re-runnable: every step tolerates having run before.
set -euo pipefail
# shellcheck source=lib/cozystack-version.sh
. "$(dirname "$0")/lib/cozystack-version.sh"
STRIMZI_CRD_FILE="${STRIMZI_CRD_FILE:-/strimzi-crds/strimzi-crds-0.51.0.yaml}"
CONVERSION_IMAGE="${CONVERSION_IMAGE:-quay.io/strimzi/operator:0.51.0@sha256:ef3e406aa09449dd0241d53cdc81e15e58bf59b51f2425fb64384a179d676b1d}"
CONVERSION_NS="${CONVERSION_NS:-cozy-kafka-operator}"
CONVERSION_NAME=strimzi-v1-api-conversion
CONVERSION_TIMEOUT="${CONVERSION_TIMEOUT:-1800}"
# The run that starts the Job waits up to CONVERSION_TIMEOUT for it. As the
# platform hook, a run is cut off sooner by the Helm upgrade timeout
# (cozystack-operator --helmrelease-upgrade-timeout, 10m by default), and the
# retry, the next pod of the hook Job or a new hook Job once helm-controller
# retries the upgrade, waits up to CONVERSION_TIMEOUT again for the same Job.
# The Job is not owned by the hook Job and nothing else bounds it, so the
# deadline is the only limit: a conversion gets two waits, and a Job whose pod
# never starts then fails and the next run starts the conversion over.
CONVERSION_DEADLINE=$((CONVERSION_TIMEOUT * 2))
POLL_INTERVAL="${POLL_INTERVAL:-5}"
KAFKA_TARGET_VERSION=v4.3
# The CRDs the conversion tool rewrites. kafkamirrormakers.kafka.strimzi.io
# is not among them: MirrorMaker 1 left in Strimzi 0.46 and its CRD is left
# behind untouched.
STRIMZI_CRDS="kafkas.kafka.strimzi.io kafkanodepools.kafka.strimzi.io
kafkaconnects.kafka.strimzi.io kafkaconnectors.kafka.strimzi.io
kafkabridges.kafka.strimzi.io kafkamirrormaker2s.kafka.strimzi.io
kafkarebalances.kafka.strimzi.io kafkatopics.kafka.strimzi.io
kafkausers.kafka.strimzi.io strimzipodsets.core.strimzi.io"
# --ignore-not-found separates a cluster without Strimzi (no output, rc 0)
# from an API error, which aborts under set -e.
crd=$(kubectl get crd kafkas.kafka.strimzi.io --ignore-not-found -o name)
if [ -z "$crd" ]; then
echo "Strimzi is not installed; nothing to convert"
stamp_cozystack_version 62
exit 0
fi
# The running 0.45 operator must not write to the resources from here on.
# Strimzi supports the conversion only under an operator that knows v1
# (0.49 or later), and 0.45 rewrites the Kafka status through v1beta2 on every
# reconcile, because the v1 schema prunes its kafkaMetadataState field. Each
# such write records a v1beta2 managedFields entry again after the rewrite
# below. The kafka-operator release brings the operator back at Strimzi 1.2:
# its chart sets the replica count, so the upgrade restores it. Kafka pods keep
# running while no operator reconciles them.
stop_old_operator() {
operator=$(kubectl -n "$CONVERSION_NS" get deployment strimzi-cluster-operator --ignore-not-found -o name)
if [ -n "$operator" ]; then
kubectl -n "$CONVERSION_NS" scale deployment strimzi-cluster-operator --replicas=0
kubectl -n "$CONVERSION_NS" wait --for=delete pod -l name=strimzi-cluster-operator --timeout=300s
fi
# The topic and user operators of each cluster write KafkaTopic and
# KafkaUser status through v1beta2 the same way, also while they shut down.
# Only now that the cluster operator is gone does a scale-down stick; the
# 1.2 operator scales them back up when it rolls them onto its own image.
kubectl get deployment --all-namespaces -l app.kubernetes.io/name=entity-operator,app.kubernetes.io/managed-by=strimzi-cluster-operator \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' |
while read -r ns name; do
kubectl -n "$ns" scale deployment "$name" --replicas=0
# An empty selector would wait on every pod in the namespace.
selector=$(kubectl -n "$ns" get deployment "$name" -o json |
jq -er '.spec.selector.matchLabels // {} | to_entries | map("\(.key)=\(.value)") | join(",") | select(length > 0)')
kubectl -n "$ns" wait --for=delete pod -l "$selector" --timeout=300s
done
}
# Called as an if condition, where set -e does not apply: a failed read exits
# here, or it would pass for an absent CRD.
stored_v1_only() {
for c in $STRIMZI_CRDS; do
if ! v=$(kubectl get crd "$c" --ignore-not-found -o jsonpath='{.status.storedVersions}'); then
echo "ERROR: reading the stored versions of $c failed" >&2
exit 1
fi
if [ -n "$v" ] && [ "$v" != '["v1"]' ]; then
return 1
fi
done
return 0
}
# A Job whose pod the API server refused fails on its deadline with no failed
# pod to count, so its Failed condition is read as well.
job_failed() {
f=$(kubectl -n "$CONVERSION_NS" get job "$CONVERSION_NAME" -o jsonpath='{.status.failed} {.status.conditions[?(@.type=="Failed")].status}')
case "$f" in
[1-9]*|*True*) return 0 ;;
esac
return 1
}
# wait_for_conversion: 0 once the conversion Job has succeeded; on a failure
# or a timeout, print its logs and return 1.
wait_for_conversion() {
echo "Waiting for the conversion Job"
waited=0
pull_reported=
while :; do
succeeded=$(kubectl -n "$CONVERSION_NS" get job "$CONVERSION_NAME" -o jsonpath='{.status.succeeded}')
# A pod that cannot pull its image leaves no logs to print, and its Job
# fails only at the deadline.
if [ -z "$pull_reported" ]; then
waiting=$(kubectl -n "$CONVERSION_NS" get pod -l "batch.kubernetes.io/job-name=$CONVERSION_NAME" \
-o jsonpath='{.items[*].status.initContainerStatuses[*].state.waiting.reason} {.items[*].status.containerStatuses[*].state.waiting.reason}' || true)
case "$waiting" in
*ErrImagePull*|*ImagePullBackOff*)
echo "ERROR: the conversion pod cannot pull $CONVERSION_IMAGE. A cluster that cannot reach quay.io needs that image mirrored under the same name and digest, for example through a registry mirror for quay.io in the container runtime of the nodes. The Job fails ${CONVERSION_DEADLINE}s after it started, and the next run of the migration starts it again." >&2
pull_reported=1
;;
esac
fi
if [ "${succeeded:-0}" -ge 1 ]; then
kubectl -n "$CONVERSION_NS" logs "job/$CONVERSION_NAME" --all-containers --prefix || true
return 0
fi
if job_failed || [ "$waited" -ge "$CONVERSION_TIMEOUT" ]; then
echo "ERROR: the Strimzi v1 API conversion Job did not succeed; its logs:" >&2
kubectl -n "$CONVERSION_NS" logs "job/$CONVERSION_NAME" --all-containers --prefix >&2 || true
return 1
fi
sleep "$POLL_INTERVAL"
waited=$((waited + POLL_INTERVAL))
done
}
# The Job's ClusterRole can patch CRDs and write ConfigMaps everywhere, so
# none of it outlives a conversion that has finished, whether it succeeded or
# failed. A failed Job itself stays for diagnosis. A Job still running when
# the wait times out keeps its rights: the retry waits for that same Job.
remove_conversion_rbac() {
kubectl delete clusterrolebinding "$CONVERSION_NAME" --ignore-not-found
kubectl delete clusterrole "$CONVERSION_NAME" --ignore-not-found
kubectl -n "$CONVERSION_NS" delete serviceaccount "$CONVERSION_NAME" --ignore-not-found
}
remove_conversion_job() {
kubectl -n "$CONVERSION_NS" delete job "$CONVERSION_NAME" --ignore-not-found --wait=true
remove_conversion_rbac
}
# The platform hook can time out while the Job converts, and the retry kills
# the hook at whatever point it reached. A Job still running then has to
# finish before anything here touches the CRDs, or the CRD apply below would
# switch the storage version back under it. Its outcome is read from the CRDs
# afterwards, not from the Job.
job_state=$(kubectl -n "$CONVERSION_NS" get job "$CONVERSION_NAME" --ignore-not-found -o jsonpath='{.metadata.name} {.status.succeeded}{.status.failed}{.status.conditions[?(@.type=="Failed")].status}')
case "$job_state" in
"$CONVERSION_NAME ")
echo "A conversion Job from an earlier attempt is still running"
# backoffLimit is 0, so a Job reporting no failure has not finished.
if ! wait_for_conversion; then
if ! job_failed; then
echo "ERROR: the conversion Job of an earlier attempt is still running; retry once it finishes" >&2
exit 1
fi
fi
;;
esac
# A previous conversion can still be changing these objects, so the gate runs
# after its wait but before a new operator stop or any CRD mutation.
served=$(kubectl get crd kafkas.kafka.strimzi.io -o jsonpath='{.spec.versions[*].name}')
case " $served " in
*" v1beta2 "*) kafka_api=v1beta2 ;;
*" v1 "*) kafka_api=v1 ;;
*) echo "ERROR: no supported Strimzi Kafka API is served" >&2; exit 1 ;;
esac
# KRaft is the last Strimzi metadata state, so the state with a metadata
# version proves KRaft. The conversion deletes kafkaMetadataState, so without
# it the proof is Ready=True, or the annotation a previous run recorded below.
# Ready because the assembly operator retains kafkaMetadataVersion on failure:
# https://github.com/strimzi/strimzi-kafka-operator/blob/0.51.0/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/KafkaAssemblyOperator.java#L189-L192
KRAFT_ANNOTATION=cozystack.io/kraft-verified
kafkas=$(kubectl get "kafkas.$kafka_api.kafka.strimzi.io" --all-namespaces -o json)
unproven=$(printf '%s' "$kafkas" | jq --slurp --exit-status --raw-output --arg verified "$KRAFT_ANNOTATION" '
def nonempty_string: type == "string" and length > 0;
def unproven_reason:
if .spec.zookeeper != null then "ZooKeeper section in spec"
elif (.status | type) != "object" then "no status"
elif .status.kafkaMetadataState != null and .status.kafkaMetadataState != "KRaft"
then "metadata state \(.status.kafkaMetadataState)"
elif (.status.kafkaMetadataVersion | nonempty_string | not) then "no kafkaMetadataVersion"
elif .status.kafkaMetadataState == "KRaft" or
any(.status.conditions[]?; .type == "Ready" and .status == "True") or
(.metadata.annotations[$verified] | nonempty_string)
then null
else "not Ready" end;
if length != 1 or (.[0] | type) != "object" or (.[0].items | type) != "array"
then error("invalid Strimzi Kafka list") else .[0].items end
| map(
if type != "object" or (.metadata.namespace | nonempty_string | not) or
(.metadata.name | nonempty_string | not) or (.spec | type) != "object"
then error("invalid Strimzi Kafka object") else . end
| unproven_reason as $reason
| select($reason != null)
| "\(.metadata.namespace)/\(.metadata.name)" +
(if (.metadata.annotations["meta.helm.sh/release-name"] | nonempty_string)
then " (Helm release: \(.metadata.namespace)/\(.metadata.annotations["meta.helm.sh/release-name"]))"
else "" end) + ": \($reason)")
| join("\n")')
if [ -n "$unproven" ]; then
echo "ERROR: KRaft is not confirmed for these Kafka clusters:" >&2
printf '%s\n' "$unproven" | sed 's/^/ /' >&2
echo "Strimzi 1.x cannot run ZooKeeper or an unfinished migration. Upgrade through Cozystack v1.7 and let each Kafka finish its KRaft migration and report Ready before upgrading past it. A Kafka created after this migration stopped the Strimzi operators has no status until the upgrade finishes: delete that application and create it again afterwards." >&2
exit 1
fi
# The v1.7 chart removes its drain rebalance only when the shared pool is free.
rebalances=$(kubectl get "kafkarebalances.$kafka_api.kafka.strimzi.io" --all-namespaces \
-o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}')
drains=$(printf '%s\n' "$rebalances" | grep -- '-drain-kafka$' || true)
if [ -n "$drains" ]; then
echo "ERROR: a ZooKeeper to KRaft migration of Cozystack v1.7 is still draining a shared broker pool:" >&2
printf '%s\n' "$drains" | sed 's/^/ /' >&2
echo "Let the kafka application upgrade on v1.7 finish, which removes these KafkaRebalances, before upgrading past it." >&2
exit 1
fi
# Kafka 4.2/4.3 reject these topic configs that Kafka 3.x accepted, and the
# new kafka chart refuses to render them, so such an application would stay on
# its pre-upgrade spec under an operator that cannot run it; a broker whose log
# holds a segment.bytes below the floor does not start. The same limits are in
# packages/apps/kafka/templates/topics.yaml,
# packages/core/platform/templates/_helpers.tpl and
# packages/system/backupstrategy-controller/files/kafka-backup.sh.
# Application values are read from the HelmReleases, the same source as the
# platform chart's guard; KafkaTopics cover topics created outside the chart.
SEGMENT_BYTES_MIN=1048576
# shellcheck disable=SC2016 # a jq program, expanded by jq
rejected_topic_configs='
(("message.downconversion.enable", "message.format.version", "message.timestamp.difference.max.ms")
as $key | select($config | has($key)) | "\($owner): topic \($topic) sets \($key)"),
($config["segment.bytes"]
# Kafka trims characters up to U+0020 and reads the rest with
# Integer.parseInt, which allows a sign and leading zeros.
| (if type == "number" then .
elif type == "string" then
capture("^[\\x00-\\x20]*(?:\\+|(?<sign>-))?0*(?<digits>[0-9]+)[\\x00-\\x20]*$")
| "\(.sign // "")\(.digits)" | tonumber
else empty end)
| select(. < $min)
| "\($owner): topic \($topic) sets segment.bytes=\(.), below the Kafka 4 minimum of \($min)")'
releases=$(kubectl get helmreleases.helm.toolkit.fluxcd.io --all-namespaces -l apps.cozystack.io/application.kind=Kafka -o json)
topics=$(kubectl get "kafkatopics.$kafka_api.kafka.strimzi.io" --all-namespaces -o json)
rejected=$(printf '%s' "$releases" | jq --raw-output --argjson min "$SEGMENT_BYTES_MIN" '
.items[]
| "\(.metadata.namespace)/\(.metadata.labels["apps.cozystack.io/application.name"] // .metadata.name)" as $owner
| (.spec.values.topics // []) | if type == "array" then .[] else empty end
| select(type == "object") | .name as $topic
| (.config | if type == "object" then . else {} end) as $config
| '"$rejected_topic_configs")
rejected_crs=$(printf '%s' "$topics" | jq --raw-output --argjson min "$SEGMENT_BYTES_MIN" '
.items[]
| "\(.metadata.namespace)/kafkatopic \(.metadata.name)" as $owner
| (.spec.topicName // .metadata.name) as $topic
| (.spec.config | if type == "object" then . else {} end) as $config
| '"$rejected_topic_configs")
rejected=$(printf '%s\n%s' "$rejected" "$rejected_crs" | sed '/^$/d')
if [ -n "$rejected" ]; then
echo "ERROR: Kafka 4 rejects topic configs these Kafka applications and KafkaTopics still set:" >&2
printf '%s\n' "$rejected" | sed 's/^/ /' >&2
echo "Remove the configs Kafka 4 removed and raise segment.bytes to the minimum before running this migration." >&2
exit 1
fi
# Strimzi's conversion tool refuses some v1beta2 objects outright, and
# convert-resource stops at the first one, with the operators stopped and the
# objects it reached before already converted. These are the refusals of the
# tool in CONVERSION_IMAGE, read from the strimzi-kafka-operator 0.51.0 source
# (v1-api-conversion/src/main/java/io/strimzi/kafka/api/conversion/v1/converter/):
# a missing .spec on every kind, and what KafkaConversions,
# SharedConversions, MirrorMaker2Conversions and the Conversion.move of
# KafkaConnectConverter reject. Re-read them when CONVERSION_IMAGE moves off
# 0.51.0; a Bats case fails on that change. The tool cannot do a dry run here
# instead: it is Java, and this image has no JVM. Once the CRDs serve v1
# alone, every object is stored as v1 and nothing is converted.
# shellcheck disable=SC2016 # a jq program, expanded by jq
conversion_refusals='
def oauth: .authentication.type? == "oauth";
.spec as $s
| if $s == null then "no .spec"
elif $crd == "kafkas.kafka.strimzi.io" then
(select($s.kafka.resources != null) | ".spec.kafka.resources (use a KafkaNodePool)"),
(select(any($s.kafka.listeners[]?; oauth)) | "OAuth authentication on a listener"),
(select(any($s.kafka.listeners[]?; .authentication.type? == "custom" and .authentication.secrets != null))
| "secrets in a custom listener authentication"),
(select($s.kafka.authorization.type? == "keycloak") | "Keycloak authorization"),
(select($s.kafka.authorization.type? == "opa") | "Open Policy Agent authorization")
elif $crd == "kafkabridges.kafka.strimzi.io" then
select($s | oauth) | "OAuth client authentication"
elif $crd == "kafkaconnects.kafka.strimzi.io" then
(select($s | oauth) | "OAuth client authentication"),
(select($s.externalConfiguration != null) | ".spec.externalConfiguration"),
(select($s.build.output.additionalKanikoOptions != null and $s.build.output.additionalBuildOptions != null)
| "both .spec.build.output.additionalKanikoOptions and additionalBuildOptions")
elif $crd == "kafkamirrormaker2s.kafka.strimzi.io" then
[$s.clusters[]?.alias] as $aliases
| (if $s.target != null then $s.target.alias else $s.connectCluster end) as $target
| (select($s.externalConfiguration != null) | ".spec.externalConfiguration"),
(select(any($s.clusters[]?, $s.target, $s.mirrors[]?.source; oauth)) | "OAuth client authentication"),
(select($s.target == null and ($aliases == [] or $s.connectCluster == null))
| "no target cluster in .spec.target, or in .spec.connectCluster and .spec.clusters"),
(select($s.target == null and $aliases != [] and $s.connectCluster != null and
($aliases | index([$s.connectCluster])) == null)
| ".spec.connectCluster is not an alias in .spec.clusters"),
($s.mirrors[]? |
(select(.targetCluster != null and .targetCluster != $target)
| "a mirror whose targetCluster is not the target cluster"),
(select(.source == null and ($aliases == [] or .sourceCluster == null))
| "a mirror without .source or .sourceCluster"),
(select(.source == null and $aliases != [] and .sourceCluster != null and
(.sourceCluster as $source | $aliases | index([$source])) == null)
| "a mirror whose sourceCluster is not an alias in .spec.clusters"))
else empty end'
case " $served " in
*" v1beta2 "*)
refused=
for c in $STRIMZI_CRDS; do
list=$(kubectl get "${c%%.*}.$kafka_api.${c#*.}" --all-namespaces -o json)
found=$(printf '%s' "$list" | jq --raw-output --arg crd "$c" '.items[]
| "\($crd) \(.metadata.namespace)/\(.metadata.name)" as $object
| [ '"$conversion_refusals"' ] | unique | .[] | "\($object): \(.)"')
refused=$(printf '%s\n%s' "$refused" "$found" | sed '/^$/d')
done
if [ -n "$refused" ]; then
echo "ERROR: the Strimzi v1 API conversion refuses these objects:" >&2
printf '%s\n' "$refused" | sed 's/^/ /' >&2
echo "Change each one as the Strimzi v1 API requires and run this migration again." >&2
exit 1
fi
;;
esac
# Record the proof before anything converts, so a retry after the conversion
# accepts a Kafka that passed only on its KRaft state.
printf '%s' "$kafkas" | jq --raw-output --arg verified "$KRAFT_ANNOTATION" '.items[]
| select(.metadata.annotations[$verified] | type == "string" and length > 0 | not)
| "\(.metadata.namespace) \(.metadata.name) \(.status.kafkaMetadataVersion)"' |
while read -r ns name version; do
kubectl -n "$ns" annotate "kafkas.$kafka_api.kafka.strimzi.io" "$name" --overwrite "$KRAFT_ANNOTATION=$version"
done
# A Job failed on its deadline can still have a pod terminating, and
# convert-resource writes until it is gone. A foreground deletion returns
# only once the Job's pods are.
if [ -n "$job_state" ]; then
kubectl -n "$CONVERSION_NS" delete job "$CONVERSION_NAME" --ignore-not-found --cascade=foreground --wait=true
fi
if stored_v1_only; then
echo "Every Strimzi CRD already stores only v1; skipping the conversion"
# The 0.45 operator can only still be running while the CRDs serve
# v1beta2. Once they serve v1 alone the operator is 1.x, and a run by hand
# must not stop it: no Helm upgrade would bring it back.
case " $served " in
*" v1beta2 "*) stop_old_operator ;;
esac
remove_conversion_job
else
if [ ! -r "$STRIMZI_CRD_FILE" ]; then
echo "ERROR: $STRIMZI_CRD_FILE is not readable; set STRIMZI_CRD_FILE to strimzi-crds-0.51.0.yaml from the Strimzi 0.51.0 release" >&2
exit 1
fi
stop_old_operator
# These CRDs mark v1beta2 as the storage version, so a re-run after a
# partial crd-upgrade switches storage back and converts from the start.
echo "Applying the Strimzi 0.51.0 CRDs, which serve both v1beta2 and v1"
kubectl apply --server-side --force-conflicts --field-manager=cozystack-migration-61 -f "$STRIMZI_CRD_FILE"
for c in $STRIMZI_CRDS; do
kubectl wait --for=condition=Established --timeout=120s "crd/$c"
done
# Whatever an earlier attempt left has finished by now, and its result was
# undone by the CRD apply above, so the conversion starts over.
kubectl -n "$CONVERSION_NS" delete job "$CONVERSION_NAME" --ignore-not-found --wait=true
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: $CONVERSION_NAME
namespace: $CONVERSION_NS
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: $CONVERSION_NAME
rules:
- apiGroups: ["kafka.strimzi.io"]
resources: ["kafkas", "kafkanodepools", "kafkaconnects", "kafkaconnectors", "kafkabridges", "kafkamirrormaker2s", "kafkarebalances", "kafkatopics", "kafkausers"]
verbs: ["get", "list", "patch", "update"]
- apiGroups: ["core.strimzi.io"]
resources: ["strimzipodsets"]
verbs: ["get", "list", "patch", "update"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions", "customresourcedefinitions/status"]
verbs: ["get", "list", "patch", "update"]
# The tool moves a KafkaBridge metrics configuration into a ConfigMap.
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "create", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: $CONVERSION_NAME
subjects:
- kind: ServiceAccount
name: $CONVERSION_NAME
namespace: $CONVERSION_NS
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: $CONVERSION_NAME
---
apiVersion: batch/v1
kind: Job
metadata:
name: $CONVERSION_NAME
namespace: $CONVERSION_NS
spec:
backoffLimit: 0
activeDeadlineSeconds: $CONVERSION_DEADLINE
template:
metadata:
labels:
policy.cozystack.io/allow-to-apiserver: "true"
spec:
serviceAccountName: $CONVERSION_NAME
restartPolicy: Never
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
# crd-upgrade must not start before every object is converted.
initContainers:
- name: convert-resource
image: $CONVERSION_IMAGE
command: ["/opt/v1-api-conversion/bin/v1-api-conversion.sh", "convert-resource", "--all-namespaces", "--log-level", "INFO"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
containers:
- name: crd-upgrade
image: $CONVERSION_IMAGE
command: ["/opt/v1-api-conversion/bin/v1-api-conversion.sh", "crd-upgrade"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
EOF
if ! wait_for_conversion; then
if job_failed; then
remove_conversion_rbac
fi
exit 1
fi
if ! stored_v1_only; then
echo "ERROR: the conversion Job succeeded but a Strimzi CRD still records a stored version other than v1" >&2
for c in $STRIMZI_CRDS; do
echo " $c: $(kubectl get crd "$c" --ignore-not-found -o jsonpath='{.status.storedVersions}')" >&2
done
remove_conversion_rbac
exit 1
fi
remove_conversion_job
fi
# Moving the applications off Kafka 3.x first makes the old chart reject
# their values, so no Helm action can store a new v1beta2 revision behind the
# rewrite below.
# kubectl api-resources fails when any API group fails discovery, so only
# this group is read: NotFound means it is not installed, any other error
# aborts.
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
if app_group=$(kubectl get --raw /apis/apps.cozystack.io/v1alpha1 2>"$work/app-group.err"); then
kafka_apps=$(printf '%s' "$app_group" | jq --raw-output 'any(.resources[]?; .name == "kafkas")')
elif grep -qF '(NotFound)' "$work/app-group.err"; then
kafka_apps=false
else
echo "ERROR: reading the apps.cozystack.io API failed: $(cat "$work/app-group.err")" >&2
exit 1
fi
has_kafka_apps() { [ "$kafka_apps" = true ]; }
if has_kafka_apps; then
kubectl get kafkas.apps.cozystack.io --all-namespaces \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.spec.version}{"\n"}{end}' |
while read -r ns name version; do
case "$version" in
v3.8|v3.9)
echo "Moving Kafka $ns/$name from $version to $KAFKA_TARGET_VERSION"
kubectl -n "$ns" patch kafkas.apps.cozystack.io "$name" --type merge \
-p "{\"spec\":{\"version\":\"$KAFKA_TARGET_VERSION\"}}"
;;
esac
done
fi
# Helm builds the objects of a stored release before it upgrades,
# uninstalls or rolls back to it, and fails on an API version the server no
# longer serves, so every Kafka release would be stuck once the v1-only CRDs
# land. Uninstall reads the latest revision whatever its status, and upgrade
# falls back to the last failed one, so every revision of each release is
# rewritten, the way helm-mapkubeapis does: the Secret holds base64 of
# gzipped JSON, inside the base64 of the Secret data. A release whose install
# failed has no Kafka CR, so the releases are listed from the applications as
# well. The chart's hooks carry no Strimzi kinds, so only the manifest needs
# the rewrite.
list_releases() {
kubectl get kafkas.v1.kafka.strimzi.io --all-namespaces \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.annotations.meta\.helm\.sh/release-name}{"\n"}{end}' > "$work/releases.all"
if has_kafka_apps; then
kubectl get kafkas.apps.cozystack.io --all-namespaces \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" kafka-"}{.metadata.name}{"\n"}{end}' >> "$work/releases.all"
fi
sort -u "$work/releases.all" > "$work/releases"
}
# scan_releases rewrite|check: rewrite every revision that still names
# v1beta2, or list the ones that do and the ones still pending.
scan_releases() {
list_releases
while read -r ns release; do
[ -n "$release" ] || continue
secrets=$(kubectl -n "$ns" get secret -l "owner=helm,name=$release" -o jsonpath='{.items[*].metadata.name}')
for secret in $secrets; do
kubectl -n "$ns" get secret "$secret" -o jsonpath='{.data.release}' | base64 -d | base64 -d | gzip -dc > "$work/release.json"
if [ "$1" = check ]; then
status=$(jq -r '.info.status' "$work/release.json")
case "$status" in
pending-*)
echo " Helm release $ns/$secret is $status"
continue
;;
esac
fi
if ! jq -e '.manifest | contains("apiVersion: kafka.strimzi.io/v1beta2")' "$work/release.json" > /dev/null; then
continue
fi
if [ "$1" = check ]; then
echo " Helm release $ns/$secret"
continue
fi
echo "Rewriting Helm release $ns/$secret to kafka.strimzi.io/v1"
jq -c '.manifest |= gsub("apiVersion: kafka.strimzi.io/v1beta2\n"; "apiVersion: kafka.strimzi.io/v1\n")' "$work/release.json" |
gzip -c | base64 | tr -d '\n' | base64 | tr -d '\n' > "$work/release.b64"
jq -n --rawfile r "$work/release.b64" '{data: {release: $r}}' > "$work/patch.json"
kubectl -n "$ns" patch secret "$secret" --type merge --patch-file "$work/patch.json"
done
done < "$work/releases"
}
# The objects keep managedFields entries recorded under the old API versions,
# and once the CRDs serve only v1 the API server refuses every server-side
# apply to such an object ("request to convert CR to an invalid group/version",
# kubernetes/kubernetes#111937). helm-controller recovers by wiping the
# entries, and the fields Helm owned pass to "before-first-apply", so a field a
# later chart drops is never removed. Rewrite the entries to v1 while both
# versions are still served; the conversion is None, so field paths are
# spelled the same under either, and an entry for a field v1 dropped only
# names a field that no longer exists.
#
# The patch replaces the whole managedFields list, so a patch built from a
# stale read could drop an entry a concurrent writer added in between. The
# patch carries the resourceVersion it was built from: the API server then
# answers a concurrent change with a conflict, and the object is read again.
stale_fields='[.metadata.managedFields[]? | select(.apiVersion | test("/v1(alpha1|beta1|beta2)$"))] | length > 0'
# list_stale_managed <crd>: the objects of that kind with entries to rewrite.
list_stale_managed() {
kubectl get "$1" --all-namespaces --show-managed-fields -o json |
jq -r ".items[] | select($stale_fields) | \"\\(.metadata.namespace) \\(.metadata.name)\"" > "$work/managed"
}
rewrite_managed() {
for c in $STRIMZI_CRDS; do
list_stale_managed "$c"
while read -r ns name; do
echo "Moving the managedFields of $c $ns/$name to v1"
attempt=0
while :; do
kubectl -n "$ns" get "$c" "$name" --show-managed-fields -o json > "$work/object.json"
if ! jq -e "$stale_fields" "$work/object.json" > /dev/null; then
break
fi
jq '{metadata: {resourceVersion: .metadata.resourceVersion, managedFields: (.metadata.managedFields | map(.apiVersion |= sub("/v1(alpha1|beta1|beta2)$"; "/v1")))}}' \
"$work/object.json" > "$work/managed-patch.json"
if kubectl -n "$ns" patch "$c" "$name" --type merge --patch-file "$work/managed-patch.json" 2> "$work/patch.err"; then
break
fi
attempt=$((attempt + 1))
if ! grep -q 'the object has been modified' "$work/patch.err" || [ "$attempt" -ge 5 ]; then
cat "$work/patch.err" >&2
exit 1
fi
done
done < "$work/managed"
done
}
scan_releases rewrite
rewrite_managed
# A Helm action already running when the releases were rewritten keeps the
# releases it read in memory and stores them when it finishes: its own
# revision, and the previous one as superseded, both naming v1beta2 again.
# Scan once more and refuse to stamp the version while a revision is still
# pending or names v1beta2: the hook then fails and runs the migration again.
scan_releases check > "$work/leftover"
for c in $STRIMZI_CRDS; do
list_stale_managed "$c"
sed "s|^| managedFields of $c |" "$work/managed" >> "$work/leftover"
done
if [ -s "$work/leftover" ]; then
echo "ERROR: these still name a removed Strimzi API version, or a Helm action on them has not finished:" >&2
cat "$work/leftover" >&2
echo "A pending release clears once helm-controller finishes or unlocks that HelmRelease action." >&2
exit 1
fi
stamp_cozystack_version 62
+1 -1
View File
@@ -14,7 +14,7 @@ migrations:
# refuses to advance past a migration file missing from the image (exit 1), so
# a stale pin fails loudly rather than silently skipping the etcd adoption.
image: ghcr.io/cozystack/cozystack/platform-migrations:v1.6.0@sha256:6777042e3e9c6bad76e7d96fb64baa115061975f39061cb3ede84e21d0e2213f
targetVersion: 61
targetVersion: 62
# Adopt legacy etcd clusters onto the v1alpha2 operator WITHOUT the mandatory
# pre-adoption safety snapshot (migration 50). Leave false unless migration 50
# has actually refused to proceed and you accept the risk: adoption rewires