mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
Record each topic's ReplicationFactor in the backup manifest, off the same --describe line the partition count already comes from, and recreate the topic with it on restore. The value used to be a BackupClass parameter defaulting to 1, and the Cozystack Kafka chart sets the broker-level min.insync.replicas to 2 as soon as kafka.replicas reaches 2: kafka-console-producer writes with acks=all, the broker rejects every write into a topic with fewer replicas than that, and the producer logs the rejection and exits 0. A restore onto any multi-broker cluster therefore landed zero records and failed the post-replay count with a message naming the wrong cause; in place, the documented delete had already taken the live topic with it. replicationFactor stays as an override for restoring into a smaller cluster, and a value the target cannot host now fails at --create in Kafka's own words. Hold the pre-replay emptiness check to the topic's partition count, as the backup side already does. kafka-get-offsets skips a partition it could not read and still exits 0, so a leader election in flight dropped the populated partitions from the listing, the surviving empty ones summed to zero, and the replay appended a full copy into a live topic - the exact outcome the check was moved ahead of the replay to prevent. Both restore-side counts now go through one helper that refuses a short listing. The data-file-versus-manifest check runs both ways before anything is created, and the replay walks the manifest topic by topic, announcing each one, so a retried Pod's log says which topics landed. The demo now runs two brokers with two-replica topics, which is the shape the chart puts min.insync.replicas=2 on; the restore steps assert the restored topic's replication factor, and step 06 submits a second RestoreJob against the populated topic and requires it to be refused with the topic untouched. Smaller findings from the same round: the topics parameter is held to Kafka's name charset (a literal "*" no longer path-expands) and refused on a repeated name, which produced a Succeeded backup no restore could balance; the strategy verifies S3 TLS against the seaweedfs CA projected into the <app>-backup-s3 Secret instead of curl -k, following the rabbitmq example, so the e2e endpoint moves to the .svc name the serving cert covers; the image lookup keeps digest-pinned references; the default namespace is tenant-root, the only one the demo ever ran in; step 05 caches the resolved Backup name for the restore steps; the decoy count subtracts begin offsets like the topic count; the bats test name carries its count again; and the scenario walk-throughs the sibling examples ship are added. Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io> Assisted-By: LLM
47 lines
1.9 KiB
Bash
Executable File
47 lines
1.9 KiB
Bash
Executable File
#!/bin/bash
|
|
# Step 02: Map the Kafka application kind to the Job strategy from step 01. The
|
|
# strategy parameters (which topics to back up, an optional replication-factor
|
|
# override for the topics restore recreates) travel through the BackupClass and
|
|
# are exposed to the strategy template as `.Parameters`. They are also
|
|
# snapshotted onto the resulting Backup's driverMetadata, so a later restore
|
|
# re-renders with the same values.
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
source "$SCRIPT_DIR/00-helpers.sh"
|
|
|
|
print_header "Step 02: Create BackupClass '${BACKUPCLASS_NAME}'"
|
|
log_command "kubectl apply -f - (BackupClass: $BACKUPCLASS_NAME -> $STRATEGY_NAME)"
|
|
|
|
kubectl apply -f - <<EOF
|
|
apiVersion: backups.cozystack.io/v1alpha1
|
|
kind: BackupClass
|
|
metadata:
|
|
name: ${BACKUPCLASS_NAME}
|
|
spec:
|
|
strategies:
|
|
- application:
|
|
apiGroup: apps.cozystack.io
|
|
kind: Kafka
|
|
strategyRef:
|
|
apiGroup: strategy.backups.cozystack.io
|
|
kind: Job
|
|
name: ${STRATEGY_NAME}
|
|
parameters:
|
|
# Empty "topics" backs up every non-internal topic - the wider net, not
|
|
# the safer one: it can pull in a compacted or transactional topic,
|
|
# which this logical strategy fails closed on (see the drain guard in
|
|
# 01-create-strategy.sh and README "Scope and limitations"). Set a
|
|
# comma/space-separated list to restrict the backup to topics known to
|
|
# be plain.
|
|
topics: "${TOPIC}"
|
|
# Restore recreates each topic with the replication factor the backup
|
|
# captured from the source. Set replicationFactor to override it - the
|
|
# way to restore into a cluster with fewer brokers than the source.
|
|
# Like every parameter it is snapshotted at backup time, so it has to
|
|
# be in place before the BackupJob whose Backup will be restored.
|
|
EOF
|
|
|
|
log_success "BackupClass '${BACKUPCLASS_NAME}' created."
|
|
echo -e "\n${GREEN}${BOLD}Next:${NC} ./03-create-bucket.sh"
|