Files
cozystack/examples/backups/kafka/02-create-backupclass.sh
T
Andrey Kolkov 5bd029c46a fix(kafka): restore with the captured replication factor and a complete listing
Record each topic's ReplicationFactor in the backup manifest, off the
same --describe line the partition count already comes from, and
recreate the topic with it on restore. The value used to be a
BackupClass parameter defaulting to 1, and the Cozystack Kafka chart
sets the broker-level min.insync.replicas to 2 as soon as kafka.replicas
reaches 2: kafka-console-producer writes with acks=all, the broker
rejects every write into a topic with fewer replicas than that, and the
producer logs the rejection and exits 0. A restore onto any multi-broker
cluster therefore landed zero records and failed the post-replay count
with a message naming the wrong cause; in place, the documented delete
had already taken the live topic with it. replicationFactor stays as an
override for restoring into a smaller cluster, and a value the target
cannot host now fails at --create in Kafka's own words.

Hold the pre-replay emptiness check to the topic's partition count, as
the backup side already does. kafka-get-offsets skips a partition it
could not read and still exits 0, so a leader election in flight dropped
the populated partitions from the listing, the surviving empty ones
summed to zero, and the replay appended a full copy into a live topic -
the exact outcome the check was moved ahead of the replay to prevent.
Both restore-side counts now go through one helper that refuses a short
listing. The data-file-versus-manifest check runs both ways before
anything is created, and the replay walks the manifest topic by topic,
announcing each one, so a retried Pod's log says which topics landed.

The demo now runs two brokers with two-replica topics, which is the
shape the chart puts min.insync.replicas=2 on; the restore steps assert
the restored topic's replication factor, and step 06 submits a second
RestoreJob against the populated topic and requires it to be refused
with the topic untouched.

Smaller findings from the same round: the topics parameter is held to
Kafka's name charset (a literal "*" no longer path-expands) and refused
on a repeated name, which produced a Succeeded backup no restore could
balance; the strategy verifies S3 TLS against the seaweedfs CA projected
into the <app>-backup-s3 Secret instead of curl -k, following the
rabbitmq example, so the e2e endpoint moves to the .svc name the serving
cert covers; the image lookup keeps digest-pinned references; the
default namespace is tenant-root, the only one the demo ever ran in;
step 05 caches the resolved Backup name for the restore steps; the decoy
count subtracts begin offsets like the topic count; the bats test name
carries its count again; and the scenario walk-throughs the sibling
examples ship are added.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-By: LLM
2026-09-23 17:44:17 +04:00

47 lines
1.9 KiB
Bash
Executable File

#!/bin/bash
# Step 02: Map the Kafka application kind to the Job strategy from step 01. The
# strategy parameters (which topics to back up, an optional replication-factor
# override for the topics restore recreates) travel through the BackupClass and
# are exposed to the strategy template as `.Parameters`. They are also
# snapshotted onto the resulting Backup's driverMetadata, so a later restore
# re-renders with the same values.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/00-helpers.sh"
print_header "Step 02: Create BackupClass '${BACKUPCLASS_NAME}'"
log_command "kubectl apply -f - (BackupClass: $BACKUPCLASS_NAME -> $STRATEGY_NAME)"
kubectl apply -f - <<EOF
apiVersion: backups.cozystack.io/v1alpha1
kind: BackupClass
metadata:
name: ${BACKUPCLASS_NAME}
spec:
strategies:
- application:
apiGroup: apps.cozystack.io
kind: Kafka
strategyRef:
apiGroup: strategy.backups.cozystack.io
kind: Job
name: ${STRATEGY_NAME}
parameters:
# Empty "topics" backs up every non-internal topic - the wider net, not
# the safer one: it can pull in a compacted or transactional topic,
# which this logical strategy fails closed on (see the drain guard in
# 01-create-strategy.sh and README "Scope and limitations"). Set a
# comma/space-separated list to restrict the backup to topics known to
# be plain.
topics: "${TOPIC}"
# Restore recreates each topic with the replication factor the backup
# captured from the source. Set replicationFactor to override it - the
# way to restore into a cluster with fewer brokers than the source.
# Like every parameter it is snapshotted at backup time, so it has to
# be in place before the BackupJob whose Backup will be restored.
EOF
log_success "BackupClass '${BACKUPCLASS_NAME}' created."
echo -e "\n${GREEN}${BOLD}Next:${NC} ./03-create-bucket.sh"