Commit Graph
15 Commits
Author SHA1 Message Date
Aleksei Sviridkin 874e1e44b7 fix(backups): run the kafka-metadata demo CLI Pod as restricted
The kafka-metadata walkthrough started its long-lived CLI Pod with a
bare kubectl run, so the Pod carried no securityContext and a namespace
enforcing the restricted Pod Security level would reject it. The kafka
topic-data walkthrough already layers the restricted context onto its
CLI Pod; the metadata demo now applies the same JSON patch. A running
CLI Pod left by an earlier run without that context is replaced rather
than reused, since a Pod's securityContext cannot change in place.

The new bats cases read the overrides kubectl run receives and fail
unless both the Pod-level and the container-level fields are present
and the patch is sent as --override-type=json, and check that an owned
Pod without the restricted context is recreated.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:02 +03:00
Aleksei Sviridkin 873d9f4c8e fix(backups): stop the nats and kafka-metadata demos using a Pod they refused
Both demos read their CLI output through command substitutions, where
bash does not apply errexit. When the ownership check refused a
same-named Pod another party owns, the refusal was ignored and the exec
went into that very Pod. A failed read of the Pod's phase or owner was
likewise read as "no Pod", so the helper went on to delete and replace
a Pod whose owner it had not seen.

Every step of the Pod setup now returns on failure, and the phase and
owner reads tell a missing Pod (--ignore-not-found) from a failed read.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 02:21:01 +03:00
Aleksei Sviridkin ead7048b35 refactor(backups): define the walkthrough wait helpers once
Every backup walkthrough carried its own wait_for_field, and most their
own wait_hr_ready and wait_deleted: twelve, eight and two copies, which
had already drifted into five, three and two bodies. A fix made in one
copy did not reach the others, and a diagnostics change carried by hand
between two of them once went into the wrong function and was caught
only by reading the diff.

The helpers now live in examples/backups/_lib/wait-helpers.sh, sourced
by each walkthrough's 00-helpers.sh. Where copies differed, the shared
body keeps the diagnostics any of them had: wait_for_field leaves
kubectl's stderr attached, and wait_hr_ready prints the conditions and
history dump on Stalled or timeout. Call sites and budgets are unchanged.

A new unit test fails when any walkthrough defines one of these helpers
again, which replaces the byte-identity pin between copies.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 02:21:01 +03:00
IvanHunters 9cea7dcd90 fix(backups): exec into a CLI Pod in the kafka-metadata demo
kafka_run seeded and verified topics through a throwaway `kubectl run -i` Pod per call. That Pod's stdout returns over an attach that only carries what the container writes after the attach registers, so a CLI that finishes first comes back empty with exit 0 -- the empty-read defect fixed for the redis demo. Run one long-lived CLI Pod and `kubectl exec` into it: an exec'd process owns its pipes, so its output cannot be lost that way, and stderr stays attached (the topic_meta / topic_partitions reads keep it) so a failed read says why.

The Pod carries cozystack.io/backup-demo=kafka-metadata: kafka_cli_pod reuses only a Ready Pod the demo owns, replaces a leftover in a terminal phase, and refuses a same-named Pod it does not own; cleanup.sh reaps by that label with --grace-period=1. Unit tests under hack/ pin the exec path -- a throwaway run Pod or a re-added stderr redirect reddens them.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 00:07:20 +03:00
IvanHunters 7b94287970 fix(backups): wait for the platform strategy in rabbitmq and kafka demos
The rabbitmq and kafka-metadata backup demos read the platform's cozy-default-<engine> strategy once, right after install, where the Helm upgrade that renders it is gated on dependsOn and can leave it absent for minutes; the demo then failed immediately instead of waiting. Replace the single read with kubectl wait --for=create, so it polls through NotFound and fails only on a real error.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 00:07:08 +03:00
Andrey Kolkov caae13a60a docs(backups): drop the inaccurate NAMESPACE-override note from the demo runner
run-all.sh said NAMESPACE could be overridden through the environment, but
the numbered manifests pin namespace: tenant-root while only the waits read
$NAMESPACE, so an override would create resources in one namespace and wait
in another. The demo is tenant-root-specific anyway: the system bucket
credentials and the derived strategy live there. Say that instead.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-11 16:05:23 +04:00
Andrey Kolkov bd7d4abe84 fix(backups): size the Kafka metadata demo brokers at the chart default
The demo's source and to-copy Kafka fixtures requested resourcesPreset
"small", the deprecated legacy alias worth 1 CPU / 512Mi — half the Kafka
chart's own c1.small default (1 CPU / 1Gi). 512Mi is too little for the
broker: it crashlooped before the HelmRelease went Ready, so the demo
never reached the backup. Use c1.small on the kafka and zookeeper blocks
of both fixtures, matching the chart default and the in-tree kafka e2e
fixture that comes up cleanly.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-11 16:05:23 +04:00
Andrey Kolkov c5a8a39bcd docs(backups): name the demo BackupClass in the Kafka flow headers
The run-all.sh flow header and the chainsaw harness header both said the
ad-hoc BackupJob runs against the platform cozy-default BackupClass. It
routes to this demo's kafka-metadata BackupClass instead, which maps Kafka
to the kafka-strategy-default strategy that run-all.sh derives from the
shipped cozy-default-kafka at run time. The numbered manifests already
described this; only the two summary headers overstated the one piece of
wiring the demo exists to exercise. Reword them to match.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:53 +04:00
Andrey Kolkov 2cb2b459fa fix(backups): reject a non-numeric recorded partition count on Kafka restore
The restore path validated only the shape parsed from the live cluster,
not the partition count and replication factor read from the stored
object. A non-numeric recorded partition count made `[ "${lp}" -lt
"${parts}" ]` and its `elif` exit 2; an `if` condition is exempt from
errexit, so both read as false, the topic was skipped, and the loop still
printed its per-topic success line - so the RestoreJob reached Succeeded
without ever touching the topic. A well-formed but wrong-content object,
which curl -f cannot catch, must fail loudly like the unrecognised-record
guard.

Validate the recorded parts and rf are numeric before the comparisons and
--create, with a bats case that reddens when the guard is dropped.

Also correct a stale to-copy example comment: the strategy reads the exact
object the source backup recorded in status.artifact.uri (injected as
ARTIFACT_URI), not a key derived from .Backup.ApplicationRef.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:53 +04:00
Andrey Kolkov 4925a84c26 fix(backups): pin the restore-side guards and bound a running Kafka Job
- delete_topic waited with `grep -qx` (BRE), so deleting `audit.events`
  matched the live `audit-events` and spun to its timeout — the e2e
  failed at step 25 on every run that reached it. Use `grep -qxF`.
- The two -k/--connect-to guards were written as `! grep`, which cannot
  fail under hack/cozytest.sh (it wraps each @test in `set -e` + a
  trailing `return 0`, and bash exempts a `!`-negated command from
  errexit). Rewrite them as positive `if grep ...; then return 1; fi`;
  verified the -k reinstatement now reddens the suite under cozytest.
- Cover the restore-side literal-topic guard that was unpinned at every
  level: a bats case runs the existing-topic branch with a colliding
  sibling and asserts the `--alter` argv carries `\Q…\E`, and run-all.sh
  now leaves `audit.events` live so the e2e exercises that branch instead
  of only `--create`. Add bats cases for the partition-shrink refusal, the
  `-f` upload guard, and `--exclude-internal`.
- Bound a running Job: reconcileKafka / reconcileKafkaRestore now fail the
  BackupJob/RestoreJob once kafkaDefaultBackupDeadline elapses in the
  Job-running branch (buildJobStrategyBatchJob sets no
  activeDeadlineSeconds), so an unschedulable pod no longer pins it in
  Running forever. Pinned by TestReconcileKafka_FailsWhenJobNeverCompletes.
- Fail closed on an unknown record kind and an unknown MODE, and validate
  the live partition/RF parsed on restore (an empty value silently skipped
  reconciliation). Correct the stale "image pinned in the strategy" comment
  in the example helpers and the chainsaw test (the controller now resolves
  the target broker's image as .ClientImage).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:52 +04:00
Andrey Kolkov ec26f1171d fix(backups): make Kafka metadata e2e reach an in-cluster S3 endpoint
The e2e drove run-all.sh against the shipped cozy-default-kafka strategy,
whose endpoint is the platform's EXTERNAL S3 ingress. In CI that host
(s3.example.org) is an unroutable placeholder with no issued cert, so the
backup Pod died on `curl: (6) Could not resolve host` and the BackupJob
went Failed — the chainsaw round-trip failed there, not in the driver.

Mirror the rabbitmq e2e model, but without duplicating the 150-line
driver script: run-all.sh now derives a demo strategy from the shipped
cozy-default-kafka at run time (jq-patch: override S3_ENDPOINT, and when
an endpoint override is given, mount the copied self-signed seaweedfs CA
and point CURL_CA_BUNDLE at it), applies a demo BackupClass, and routes
the BackupJob/Plan to it. The chainsaw run sets
S3_ENDPOINT=https://seaweedfs-s3.tenant-root.svc:8333 (the .svc FQDN the
seaweedfs cert SAN covers). No -k; TLS is verified against the CA. With
no override (a real cluster whose external endpoint resolves and uses a
public CA) the shipped endpoint is reused and no CA is mounted, since
CURL_CA_BUNDLE replaces the system bundle rather than extending it.

The shipped cozy-default-kafka strategy is untouched. jq derivation
verified on a sample in both CA modes; example scripts pass bash -n.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:34 +04:00
Andrey Kolkov 0270650237 fix(backups): harden Kafka metadata S3 handling and own the artifact
- Drop `-k`: mirror the RabbitMQ S3 layer (ARTIFACT_URI as the single
  source of truth, forcePathStyle honoured, whole base URL kept) so
  curl --aws-sigv4 verifies TLS against the ACME endpoint the helper
  returns. The derived endpoint is default-port, so the curl 7.76.1
  --connect-to port workaround is gone too; this also fixes the IPv6
  and path-prefix mangling in the old host-splitting.
- Own the artifact: a cleanup mode in the script plus cleanupKafkaBackup
  wired into BackupReconciler.cleanupOnDelete delete a deleted Backup's
  object (best-effort, waited on) instead of orphaning it, matching the
  RabbitMQ precedent. Kafka has the identical own-your-object property.
- Restore probes existence with --list (fail closed), so an unreachable
  broker aborts instead of being misread as "topic absent" and created.
- Hold the fixes with tests at the right level: a colliding topic pair
  (audit.events / audit-events, distinct partition counts) in the e2e so
  a revert of the \Q...\E literal-match breaks the round-trip, and Go
  tests pinning the scope stamp and the cleanup Job.
- Correct the docs/comments: endpoint-shape row, the "cached on nodes"
  claim, and the narrower curl 7.76.1 S3 envelope (empty-body payload
  hash) that makes a small modern-curl client image the right follow-up.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:06 +04:00
Andrey Kolkov 537aadf389 fix(backups): fix Kafka backup S3 signing and literal topic names
Reproduced end-to-end against a real Kafka 3.9.1 broker:

- [CRITICAL] The Strimzi image ships curl 7.76.1, whose --aws-sigv4
  omits a non-default port from the SigV4 canonical Host while sending it
  in the Host header, so every S3 PUT/GET 403s and the driver never
  uploads or downloads. Sign/send the bare host and redirect the
  connection to the real port with --connect-to (the same fix the
  topic-data example already carries), and accept the in-cluster
  self-signed cert with -k.
- [CRITICAL] kafka-topics --topic is a Java regex, not a literal: a
  backup entry for "audit.events" also matched a live "audit-events", so
  restore repartitioned an unrelated topic irreversibly and backup
  recorded the wrong partition/replication values. Wrap the name in
  \Q...\E on every filter call (--describe on both paths, --alter on
  restore) so it matches exactly one topic; --create and
  kafka-configs --entity-name are already literal. Applied in the example
  helpers too.
- [MAJOR] cozy-default's Kafka route looked like a data backup at
  runtime. Stamp scope=topic-metadata on the produced Backup's
  DriverMetadata and say "topic definitions and configs only; no message
  data" in the Ready condition, so kubectl describe backup is honest.
- [MAJOR] Add reconcile tests for the two Go behaviours a mutation could
  silently drop: the Strimzi-Ready precondition on backup (BackupJob
  stays out of Succeeded and creates no Job) and the applicationRef Kind
  gate on the restore target.
- [MINOR] Fail closed on the config-capture grep (exit != 1 aborts) and
  clarify the image pin has no automated producer.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:21:25 +04:00
Andrey Kolkov 405eb25095 fix(backups): harden Kafka metadata backup
- Fail closed on topic listing: set -o pipefail, capture kafka-topics
  --list --exclude-internal exit separately, and abort on failure so an
  unreachable broker can no longer produce an empty backup reported as
  Succeeded. An empty list after a successful call stays legitimate.
- Check each kafka-topics --describe exit and validate the parsed
  partition count / replication factor before writing a record, so a
  timed-out describe cannot write a corrupt orders\t\t\t line.
- Store topic configs one per line with key and value in separate
  tab-delimited fields, sourced from kafka-configs --describe (one config
  per line), so a value containing a comma (cleanup.policy=compact,delete)
  round-trips; on restore each config is applied with the value bracketed
  only when it contains a comma.
- Detect topic divergence on restore instead of a silent --if-not-exists
  no-op: grow partitions with --alter when the backup asks for more, and
  fail loudly (naming the topic and both values) when it asks for fewer
  partitions or a different replication factor. A restore that cannot
  reach the recorded state no longer reports Succeeded.
- Add KafkaStrategyKind to the RestoreJob cleanup-dispatch no-op group so
  deleting a Kafka RestoreJob does not run a pointless Velero cleanup; add
  a regression test alongside the siblings, plus a reconcile Kind-gate
  test and a rendering test that actually renders the strategy template.
- Wire the kafka-metadata Chainsaw suite into both selectors
  (select-install suite_to_source, select-e2e src_to_suites) so editing
  the kafka app or the suite selects it and the bats unit tests pass.
- Pin the client image to the operator's default broker version
  (kafka 3.9.1) with a digest, and align the example default; fix the
  values comment. Point the RabbitMQ comment references (which shipped in
  the CRD description) at the Job strategy actually mirrored, and trim the
  in-pod comments that rendered into the manifest.
- Document the Strimzi Topic Operator caveat: for a topic owned by a
  KafkaTopic CR the operator reconciles its config from the CR, so an
  in-place config restore is durable only for out-of-band topics.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:21:24 +04:00
Andrey Kolkov a410a3f48b feat(backups): Kafka topic-metadata backup via a dedicated strategy driver
Add a strategy.backups.cozystack.io/Kafka driver that backs up the topic
metadata of a Cozystack-managed Kafka application — partition counts,
replication factors and non-default topic configs — through the Kafka
Admin API, and restores it with kafka-topics --create + kafka-configs
--alter. Modelled on the Job/Rabbitmq strategies: the driver renders a
one-shot batch Job, gates on the Strimzi Kafka cluster reporting Ready,
and keys each backup by .BackupName so runs do not overwrite each other.

Scope is deliberately narrow: message payloads, consumer-group offsets,
ACLs, quotas and KafkaUsers are out of scope, and there is no PITR.
Restored topics are created directly (unmanaged on the target) — the
accepted tradeoff of an Admin-API-only driver. Both restore flows are
implemented: in-place recreates dropped topics into the source, to-copy
applies the source's topics onto a freshly-bootstrapped empty cluster.

Wire it into cozy-default (strategy-kafka-default.yaml gated on the
bucket name, a BackupClass binding, read RBAC on kafka.strimzi.io/kafkas
for the readiness gate, and the Strimzi Kafka client image), document it
in backup-classes.md, and add an honest round-trip example plus Chainsaw
e2e (examples/backups/kafka-metadata, kafka-3-metadata-roundtrip) that
proves a per-run retention.ms sentinel survives both restores.

Unit tests cover the driver's pure helpers (application-ref validation,
readiness verdict, parameter round-trip, per-run artifact-URI keying).

Not yet verified on a live cluster.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:20:27 +04:00