cozystack/examples/backups/kafka

Kafka topic-data backup/restore example

This directory shows how to back up and restore the topic data of a Cozystack-managed Kafka application using the generic Job backup strategy. Unlike the app-specific drivers (Altinity for ClickHouse, CNPG for Postgres, ...), the Job strategy has no built-in knowledge of the application: it runs a Pod the operator supplies. Here that Pod is a stock Strimzi Kafka image (which carries the kafka-*.sh CLI plus bash, curl and tar) running one shell script:

• backup: freeze each partition's end offset, drain [begin, end) with kafka-console-consumer → tar → PUT the tarball to S3.
• restore: GET the tarball from S3 → tar -x → recreate the topics with their original partition count and replication factor → replay every partition file with kafka-console-producer.

The same PodTemplateSpec serves both directions. The strategy engine templates each string field independently — it cannot add or remove containers per mode — so the single image branches at runtime on {{ .Mode }} (rendered into the MODE env var). Backup pushes to a key scoped by the source app ({{ .Release.Name }}); restore reads the key scoped by the backup's source ({{ .Backup.ApplicationRef.Name }}), so a to-copy restore lands the source's data into a differently-named target. The upload uses curl --aws-sigv4, so no purpose-built backup image is needed — a stock client image plus a shell script is the whole driver.

Consistency model

The backup takes a frozen-end-offset cut. For each topic the driver reads every partition's end offset once (kafka-get-offsets --time -1) and its begin offset (--time -2), then drains only up to the frozen end. Anything produced after that read (offset ≥ the frozen end) is excluded, so a topic's partitions are captured as of the same instant — a coherent point-in-time view of that topic. The reads happen per topic, so the cut is atomic within a topic but not across topics (Kafka offers no cross-topic guarantee regardless). This needs no dependency on record timestamps: it is a cut by log position, not by wall-clock time.

Scope and limitations

This is a logical DATA backup, deliberately narrow — it captures topic records (key + value) and nothing else:

• Not captured: topic configs (retention, compaction, min.insync.replicas), ACLs, SCRAM users, and consumer-group offsets. Restore recreates each topic with its original partition count and its original replication factor: the backup manifest records the ReplicationFactor that kafka-topics --describe reports for every topic, and restore recreates the topic with it. That is not cosmetic. The Cozystack Kafka chart raises the broker-level min.insync.replicas to 2 as soon as kafka.replicas reaches 2, the console producer writes with acks=all, and a broker rejects every such write into a topic whose replication factor is below its min.insync.replicas — while the producer logs the rejection and exits 0. A topic recreated with too few replicas would therefore be restored empty and fail the post-replay count with a misleading message. The replicationFactor BackupClass parameter overrides the captured value, which is the way to restore into a cluster with fewer brokers than the source; a value the target cannot host fails at --create in Kafka's own words, before any replay. The converse is checked too: after the create, the driver reads the topic's effective min.insync.replicas with kafka-configs --describe --all and refuses, before replaying, a replication factor below it, naming the parameter to raise, since the alternative is an empty replay reported as a count shortfall. Like every parameter it is snapshotted onto the Backup at backup time, so it has to be set on the BackupClass before the BackupJob whose Backup will be restored.
• Offsets change on restore: restore re-produces records, so the target assigns fresh offsets. A consumer group's committed position from the source is therefore meaningless on the restored cluster. Preserving consumer offsets requires either offset translation or a byte-identical volume-snapshot restore (Velero/CSI) — neither is in scope here.
• Topic names are matched literally: kafka-topics and kafka-get-offsets treat --topic as a Java regex, so a name carrying a metacharacter would otherwise select its siblings (audit.events also matches audit-events) — deleting, measuring or backing up the wrong topic. Every filtering call here pins the name with \Q...\E; --create and the console producer/consumer take a literal name and are left unquoted. The demo is set up so this cannot silently regress: the topic is orders.v1 and a decoy orders-v1 sits beside it, matching the first as a regex but not as a literal. The decoy is never named in the BackupClass, so a correct run leaves it untouched, and removing a pin from the offset listing, from the in-place --delete or from either record count reddens the suite: the partition-set guard counts the decoy's partitions, the --delete takes the decoy with it, and the pre- and post-replay counts pick up its records. Two pins are not covered by it: kafka-topics --describe returns topics in the admin client's result-map order rather than sorted, so which topic the two PartitionCount reads see through head -1 is not something a fixture can pin down. The begin-offset lookups need no pin at all — they match on the whole topic:partition key, so a line belonging to another topic cannot supply this partition's begin. The topics parameter itself is held to Kafka's [a-zA-Z0-9._-] name charset and refused on a repeated name: a duplicate would run the per-topic block twice, overwriting its own data files under a manifest that then lists every partition twice, and no restore of that backup could ever balance its count.
• Fidelity: records are round-tripped as key<TAB>value text via the console tools, so keys and values must be newline- and tab-free. Message headers and record timestamps are not preserved. A production driver would use a binary-safe client (e.g. a purpose-built image) for arbitrary payloads.
• Compacted and transactional topics are out of scope: the drain reads exactly end - begin records per partition, which equals the consumable count only on a plain topic. On a compacted topic (sparse offsets) or one written by transactional producers (control markers occupy offsets but are never delivered), fewer records are consumable. kafka-console-consumer does not report this — it idles to its --timeout-ms and still exits 0 — so the strategy compares the drained line count against end - begin and fails the backup on any shortfall rather than uploading a partial tarball. The partition set is checked the same way, because kafka-get-offsets lists only the partitions whose lookup succeeded and still exits 0: the number of partitions it returned is compared against the topic's own PartitionCount, so one skipped by a leader election mid-backup fails the run instead of silently vanishing from it. Note the interaction with the parameter default: an empty topics backs up every non-internal topic, so the all-topics setting is the riskiest, not the safest — one compacted or transactional topic in the namespace aborts the whole run. Restrict topics to topics you know are plain, or accept that the default may fail closed. Conversely, if topics is empty and the namespace holds only internal topics, the loop backs up nothing: an empty tarball is uploaded and the Backup reports Succeeded. That is correct for a genuinely empty cluster, but confirm the source actually held user topics before trusting such a run — the restore side refuses such an artifact outright (manifest is empty), so an empty backup is not a restorable one.
• Partition placement: keyed records re-produce into the same partition because the topic is recreated with the same partition count and the default (murmur2) partitioner is deterministic. Null-key records fare worse: the dump renders a null key as the literal string null (the console formatter's default) and the replay reads it back as a four-byte null key — so the record not only lands in an unpredictable partition but has its key changed from null to "null", which shifts its compaction identity downstream.
• Restore appends, it does not replace: the driver creates the topic only if absent (--create --if-not-exists) and then replays every record, so restoring into a topic that still holds data adds to it rather than overwriting. Restore is safe only against an absent or empty topic — the in-place step below deletes the topic first for exactly this reason. An empty topic is not enough on its own: --if-not-exists leaves a pre-existing topic's partition count alone, which would silently re-place every keyed record, so the driver also compares the target's PartitionCount against the backup and refuses a differently shaped topic. The driver enforces the emptiness too: it reads the target's record count before replaying and refuses a topic that already holds data, so the strategy Job's backoffLimit: 2 — which retries a Pod that died mid-replay — cannot stack a second copy on top of a partial one. That pre-replay count is held to the topic's own partition count the same way the backup side is, because a listing shortened by a leader election in flight would otherwise sum the surviving empty partitions to zero and let the replay through. The in-place step drives this path on purpose: after the restore it submits a second RestoreJob against the populated topic and requires it to settle Failed with the topic unchanged. A retry after a Pod died mid-replay stops at that check on the first topic the dead Pod had already replayed and settles Failed too; the restore Pod's log names each topic before replaying it and again once it has landed, so the log says which topics still hold a partial copy. The remedy is to empty or delete every topic the log shows as replayed or started — not only the one the refusal names — and re-run. The post-replay count check then still verifies what landed.
• Each backup overwrites the previous one: the S3 object key is <namespace>/<app>/kafka-topics.tar with no run identifier, so every Backup of the same application resolves to the same tarball and restoring an older Backup restores the latest data. The generic Job driver exposes no unique per-run handle at backup time — the Backup artifact is created only after the Job completes — so a per-run key would need a driver change.
• Local disk usage is unbounded: the drain stages the whole topic under /tmp and then again as a tarball on the Pod's writable layer, and the strategy Pod requests no ephemeral-storage. A large topic can fill the node's ephemeral storage and evict other Pods; set an ephemeral-storage request and limit sized to your data before backing up anything substantial.
• TLS is verified against the endpoint's CA: the in-cluster seaweedfs presents a self-signed certificate, so 03-create-bucket.sh caches its CA (the seaweedfs-ca-cert Secret in tenant-root, or the cert-manager CA Certificate it discovers when that name is absent) and create_s3_secret adds it as ca.crt to each <app>-backup-s3 Secret. The strategy Pod projects that one key as a file and points curl --cacert at it; a Secret without the key leaves the file absent and curl falls back to the image's trust store, which is the right behaviour for a publicly-trusted endpoint — set S3_CA_SECRET="" to skip the copy there, since --cacert replaces the system bundle rather than extending it. The rabbitmq example mounts the same CA for the same reason.
• Pod Security: the strategy Pod and the demo's CLI Pod set runAsNonRoot, allowPrivilegeEscalation: false, drop all capabilities and a RuntimeDefault seccomp profile, so they satisfy the restricted Pod Security Standard on the stock Strimzi image, which runs as non-root UID 1001. Pinning an image that runs as root would need a runAsUser suited to that image instead.

For a faithful, offset-preserving Kafka backup, prefer a volume-snapshot strategy over this logical one; this example exists to show the generic Job path end to end.

Step order

File	Role	Triggered by
00-helpers.sh	Shared bash helpers, env defaults, and the kafka-CLI / S3-secret helpers; sourced by every step.	n/a
01-create-strategy.sh	Creates the cluster-scoped Job strategy (the Kafka backup/restore PodTemplateSpec).	admin
02-create-backupclass.sh	Maps apps.cozystack.io/Kafka to that strategy, with the topics parameter (and the optional replicationFactor override).	admin
03-create-bucket.sh	Provisions a Bucket and caches its S3 coordinates plus the endpoint's CA into .bucket-info.env (chmod 600; raw access keys). cleanup.sh removes this file.	admin. A tenant admin can create the Bucket itself, but the script also waits on its HelmRelease and COSI objects, reads the bucket-<name>-backup Secret, and copies the CA from a Secret in tenant-root (see the TLS bullet above); no tenant role grants access to helmreleases or objectstorage.k8s.io, or get on those Secrets
04-create-kafka.sh	Provisions a two-broker Kafka, creates the <app>-backup-s3 Secret, seeds a topic (replication factor 2) with sentinel messages.	admin. A tenant admin can create the Kafka itself, but the script also waits on its HelmRelease and Strimzi Kafka, creates the Secret, and seeds the topic through a CLI Pod it execs into; no tenant role grants access to helmreleases or kafkas.kafka.strimzi.io, create on secrets or pods, or pods/exec
05-create-backupjob.sh	Submits a BackupJob, waits for Succeeded and caches the resolved Backup name into .backup-name.env.	tenant
06-restore-in-place.sh	Deletes the topic and restores it into the same instance via RestoreJob, then submits a second RestoreJob that must be refused because the topic is no longer empty.	admin. A tenant admin can submit the two RestoreJobs, but the script deletes and checks the topic through the CLI Pod and reads the refusal from the restore Pod's log; no tenant role grants create on pods or pods/exec, or get on pods/log
07-restore-to-copy.sh	Provisions a second two-broker Kafka and restores into it via RestoreJob.spec.targetApplicationRef.	admin, for the same reasons as step 04. The tenant admin's part is the second Kafka and the RestoreJob
cleanup.sh	Removes everything created by the demo.	admin: it deletes Secrets, the Backup, and the cluster-scoped BackupClass and strategy, and no tenant role grants delete on any of them
run-all.sh	Convenience runner that executes 01..07 in order.	demo

The scenario walk-throughs 90-scenario-admin-prepare.md, 91-scenario-user-backup.md and 92-scenario-user-restore.md describe the same steps from the admin's and the tenant's side, and say which parts of the tenant's side need admin access.

Running

# Uses NAMESPACE=tenant-root by default; override any knob via the environment.
./run-all.sh
# ... or step by step, in numeric order.
./cleanup.sh

The demo runs in tenant-root because that is where the in-cluster seaweedfs lives: a tenant created isolated has a Cilium egress allowlist that blocks its Pods from reaching it. Point NAMESPACE at another tenant only if its Pods can reach the S3 endpoint the Bucket advertises.

Both Kafka instances run KAFKA_REPLICAS (default 2) brokers and the demo topics are created with TOPIC_REPLICAS (default 2) replicas. That is deliberate: with two brokers the chart sets min.insync.replicas to 2, so a restore that recreated the topic with the wrong replication factor could not land a single record, and the restore steps additionally assert that the restored topic carries the captured replication factor. A single-broker run (KAFKA_REPLICAS=1 TOPIC_REPLICAS=1) works too, but proves less.

By default the scripts resolve KAFKA_IMAGE from the running Strimzi operator — the newest image in its STRIMZI_KAFKA_IMAGES map, tag- or digest-pinned — so the backup Pod reuses an image already cached on the nodes with no extra pull or version skew. Set KAFKA_IMAGE explicitly to pin a different one, for example on an air-gapped clone where the operator cannot be read (the backup Pod only needs the kafka-*.sh CLI plus curl and tar). To see what the operator ships:

kubectl -n cozy-kafka-operator get deploy strimzi-cluster-operator \
  -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' | grep KAFKA_IMAGES