admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-08 21:14:56 +09:00
Code Issues Packages Projects Releases Wiki Activity
6,691 Commits 569 Branches 294 Tags
b981ea7f268dca179512e76f0da8191da369fb43
Commit Graph
84 Commits
Author SHA1 Message Date
IvanHunters c355de63ed fix(backups): exec into a CLI Pod in the nats demo 
nats_cli seeded and verified JetStream streams through a throwaway `kubectl run -i` Pod per call, with the same attach-race empty-read the redis demo fixed. NATS ships no long-lived Pod carrying the `nats` CLI, so run one nats-box Pod and `kubectl exec` into it; stream_message_count keeps the exec's stderr so a failed read says why. The password travels in NATS_PASSWORD on stdin rather than the --server URL the exec request records in the apiserver audit log.

The Pod carries cozystack.io/backup-demo=nats: nats_cli_pod reuses only a Ready Pod the demo owns, replaces a leftover in a terminal phase, and refuses a same-named Pod it does not own; cleanup.sh reaps by that label with --grace-period=1. Unit tests under hack/ pin the exec path, the preserved stderr, and the password staying out of argv.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-09-25 00:07:30 +03:00
IvanHunters 9cea7dcd90 fix(backups): exec into a CLI Pod in the kafka-metadata demo 
kafka_run seeded and verified topics through a throwaway `kubectl run -i` Pod per call. That Pod's stdout returns over an attach that only carries what the container writes after the attach registers, so a CLI that finishes first comes back empty with exit 0 -- the empty-read defect fixed for the redis demo. Run one long-lived CLI Pod and `kubectl exec` into it: an exec'd process owns its pipes, so its output cannot be lost that way, and stderr stays attached (the topic_meta / topic_partitions reads keep it) so a failed read says why.

The Pod carries cozystack.io/backup-demo=kafka-metadata: kafka_cli_pod reuses only a Ready Pod the demo owns, replaces a leftover in a terminal phase, and refuses a same-named Pod it does not own; cleanup.sh reaps by that label with --grace-period=1. Unit tests under hack/ pin the exec path -- a throwaway run Pod or a re-added stderr redirect reddens them.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-09-25 00:07:20 +03:00
IvanHunters 7b94287970 fix(backups): wait for the platform strategy in rabbitmq and kafka demos 
The rabbitmq and kafka-metadata backup demos read the platform's cozy-default-<engine> strategy once, right after install, where the Helm upgrade that renders it is gated on dependsOn and can leave it absent for minutes; the demo then failed immediately instead of waiting. Replace the single read with kubectl wait --for=create, so it polls through NotFound and fails only on a real error.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-09-25 00:07:08 +03:00
IvanHunters 9e3ed1f2cb fix(backups): stop the redis backup demo from reading empty values (#4399) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The redis backup round-trip suite went red on PRs that do not touch
redis, each time on an empty read (#4395). The report snapshots of those
runs point to two separate causes, and I fix both here.

The empty marker reads came from the demo helper that ran redis-cli in a
throwaway `kubectl run -i --rm` Pod. kubectl gets that Pod's stdout over
an attach, and containerd's attach only carries output written after the
attach is registered. kubectl falls back to the Pod log only when the
attach fails. A redis-cli that finishes in that gap comes back as an
empty string with exit 0. In the failed runs, the master's log shows the
key present, and sentinel logged no failover, yet the GET read ''.
`redis_cmd` now runs redis-cli by `kubectl exec` in a sentinel Pod,
which already has it. An exec'd process gets its own pipes, so its
output can't be lost that way, and its exit code comes back. The
password now goes over stdin rather than a Pod env var.

The missing-strategy failure was real: `cozy-default-redis` did not
exist yet. It gets rendered behind a lookup of the platform bucket, so
it appears only after a forced Helm upgrade. helm-controller holds that
upgrade while `cozystack-basics` is not Ready, and the e2e install's
late platform patches keep it not Ready for several minutes. The demo
read the object once. It now waits for the object with `kubectl wait
--for=create` and a 15 minute bound. That wait fails right away on
anything other than NotFound.

The demo's reads also stop sending stderr to `/dev/null`, so the next
failure names its cause. `hack/redis-backup-demo-helpers.bats` covers
`redis_cmd` with stubbed kubectl and redis-cli: it reads through exec,
keeps the password out of argv, sends no AUTH when there is no Secret,
and fails loudly when the exec fails or sentinel has no master.

Closes #4395

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI changes.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `examples/backups/redis/` and a new unit test
under `hack/`. No entry in the trigger map covers either.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): the Redis backup demo reads its keys through `kubectl exec`, so a short-lived Pod can no longer drop its output, and it waits for the platform's `cozy-default-redis` strategy to appear instead of failing on its first read.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Redis backup and restore commands now report Kubernetes and Secret
read errors instead of hiding them, while still treating a missing
password Secret as empty.
* Redis commands pass authentication securely and fail clearly when
Sentinel cannot identify a master.
* Backup demos wait for the Redis backup strategy to appear and provide
clearer error messages.
* **Tests**
* Added coverage for Redis authentication handling, command failures,
and missing Sentinel masters.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-24 22:22:25 +03:00
Myasnikov Daniil cadd72a4be fix(kafka): use the shared wait_hr_ready body in the topic-data example 
The topic-data example was branched before the backup examples moved to
one wait_hr_ready body and merged after they did, so it carried the old
copy. hack/mariadb-first-boot-waits.bats finds every copy in the tree and
runs its failure branches under errexit, and the old one does not report
a kubectl that cannot answer. That turned the unit job red on main and on
every pull request tested against it.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-09-24 14:20:13 +05:00
Timofei Larkin 95ece888ba feat(kafka): add topic-data backup/restore example and e2e roundtrip (#3580) 
## What this PR does

Adds a worked backup/restore example for a Cozystack-managed `Kafka`
application's **topic data**, built on the generic `Job` backup strategy
— the same pattern as `examples/backups/nats`, with no purpose-built
backup image. A stock Strimzi Kafka image (the `kafka-*.sh` CLI plus
`bash`, `curl` and `tar`) runs one shell script that branches on `{{
.Mode }}`: on backup it freezes each partition's end offset and drains
`[begin, end)` to a tarball it `PUT`s to S3; on restore it `GET`s the
tarball, recreates the topics with their original partition count and
replication factor, and replays every partition file. S3 access is `curl
--aws-sigv4`, verified against the endpoint's CA projected from the
`<app>-backup-s3` Secret, so no extra client image is needed.

The consistency model is a frozen-end-offset cut: `kafka-get-offsets
--time -1` is read once per partition at backup start and the consumer
drains only up to that offset, so anything produced during the run is
excluded and every partition of a topic is captured as of the same
instant — a cut by log position, independent of record timestamps.

This is deliberately a DATA-only backup. Topic configs, ACLs, SCRAM
users and **consumer-group offsets** are not captured, and re-produced
records receive fresh offsets — the README spells out the scope and
points at a volume-snapshot strategy for a faithful, offset-preserving
backup. Keyed records reproduce into their original partition because
the topic is recreated with the same partition count and the default
(murmur2) partitioner is deterministic.

The script fails closed rather than reporting a partial backup or a
lossy restore as success: it refuses an offset listing short of the
topic's partition count, a drain short of `end - begin`, a `topics`
value with a duplicate or a name outside Kafka's charset, a manifest
that disagrees with the data files, a target topic that already holds
records (checked before the replay, so a retried Pod cannot append a
second copy), a pre-existing topic with a different partition count or
replication factor, a replication factor below the target's effective
`min.insync.replicas`, and a post-replay count that does not match the
manifest. Each topic's replication factor is recorded in the manifest
and restored as captured; the `replicationFactor` BackupClass parameter
overrides it for restoring into a smaller cluster.

Also adds a `kafka-2-backup-roundtrip` Chainsaw test to
`hack/e2e-chainsaw/kafka`, which drives the example `run-all.sh` as its
harness (Bucket → two-broker source Kafka with a seeded 3-partition,
2-replica topic plus a decoy topic that the demo topic matches as a
regex but not as a literal → BackupJob → in-place restore with count,
content and replication-factor assertions → a second RestoreJob into the
populated topic that must be refused with the topic untouched →
two-broker target Kafka + to-copy restore + verify), mirroring
`mariadb-2-backup-roundtrip`. Two brokers, not one, because that is the
shape on which the chart sets `min.insync.replicas: 2` and a restore
with the wrong replication factor cannot land a record. It resolves the
Kafka image from the running operator so the backup Pod reuses the image
already cached on the nodes.

The strategy script itself is pinned by
`hack/kafka-topic-backup-strategy.bats`, on the shape of
`hack/kafka-backup-script.bats`: it renders the strategy through a stub
`kubectl` exactly as step 01 applies it, lifts the container body out,
and executes it against stubbed `kafka-*.sh`, a stubbed `curl` and a
directory standing in for S3, with fault injection for the broker
behaviours the guards exist for. Fourteen cases, one per guard; removing
the partition-listing guard reddens one of them.

Verified: `chainsaw/kafka-2-backup-roundtrip` passed in this PR's CI on
the two-broker shape (record count, byte-identical content dump and
replication factor 2 asserted after both restores, the refused
RestoreJob settled Failed with the topic untouched);
`hack/kafka-topic-backup-strategy.bats` passes under both
`hack/cozytest.sh` and bats.

Files:

- `examples/backups/kafka/` — 14 files (`00-helpers.sh`, `01`..`07`,
`run-all.sh`, `cleanup.sh`, `README.md`, `90`/`91`/`92` scenario
walk-throughs).
- `hack/kafka-topic-backup-strategy.bats` — hermetic test of the
strategy script.
- `hack/e2e-chainsaw/kafka/chainsaw-test.yaml` — new
`kafka-2-backup-roundtrip` Test appended.
- `hack/backup-access-preflight.bats` — the kafka example and Test join
the S3-preflight roster.
- `hack/select-e2e.sh`, `hack/e2e-chainsaw/.chainsaw.yaml` —
comment-only: their enumerations of preflight-gated suites are replaced
by a pointer.
- `.gitignore` — the example's `.source-dump.txt` snapshot.

No package sources (`values.yaml`, `values.schema.json`, `Chart.yaml`,
package `README.md`) are touched, so `make generate` produces no diff.

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` file-by-file
against the diff. The change adds example scripts under
`examples/backups/kafka`, one unit test under `hack/`, and appends one
Chainsaw Test to an existing file under `hack/e2e-chainsaw/kafka` — it
adds no package under `packages/apps` or `packages/extra`, changes no
CRD, no `ApplicationDefinition`, no `values.schema.json`, no
platform/installer values, no `hack/*.mk` anchor or make-target
behaviour, and no node prerequisites in `hack/e2e-prepare-cluster.bats`.
None of the downstream triggers match.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(kafka): add a topic-data backup/restore example and e2e round-trip on the generic Job backup strategy (data-only: topic records with a frozen-end-offset consistency cut and partition-preserving replay; consumer offsets, topic configs, ACLs and users are out of scope)
```
 2026-09-24 13:00:15 +04:00
Andrey Kolkov 7ce85a2f7f test(kafka): pin the strategy script's guards with a hermetic bats suite 
The in-Pod script had grown eight fail-closed guards across as many
rounds, and the only thing exercising any of them was a 40-minute
cluster test that drives the happy path plus one refusal. The partition
listing guard in particular - the whole of the previous round's second
fix - could be deleted with nothing in the tree going red. Add
hack/kafka-topic-backup-strategy.bats on the shape of
hack/kafka-backup-script.bats: render the strategy through a stub
kubectl exactly as step 01 applies it, lift the container body out of
it, and execute it against stubbed kafka-*.sh, a stubbed curl and a
directory standing in for S3, with fault injection for the broker
behaviours the guards exist for. Deleting that listing guard now fails
one case; the script gains BIN/SCRATCH/CA_FILE overrides so it can run
outside the Pod, each defaulting to the Pod's value.

The same rounds' smaller findings: the manifest's topic name is held to
Kafka's charset like the topics parameter, since it reaches unquoted
expansions with globbing on; the shape check after --create compares
the replication factor as well as the partition count, and refuses a
replication factor below the topic's effective min.insync.replicas
before replaying, because kafka-console-producer only logs a rejected
acks=all write and the run would otherwise end at the count check with
the cause unnamed; the manifest's replication factor must agree across
a topic's lines; topic_records carries its own || return 1 on each
command, as a function called as the operand of || runs with errexit
off; the CA lookup in step 03 distinguishes a denied read from an
absent Secret, and the README's persona column says the copy needs the
admin; the e2e comment no longer claims the 2-label endpoint fails the
serving cert's SANs, which include '*.tenant-root'.

Rebased onto main, which moved the chart to KRaft: the demo's Kafka CRs
drop the zookeeper block the chart no longer has.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-By: LLM
 2026-09-23 18:00:15 +04:00
Andrey Kolkov 5bd029c46a fix(kafka): restore with the captured replication factor and a complete listing 
Record each topic's ReplicationFactor in the backup manifest, off the
same --describe line the partition count already comes from, and
recreate the topic with it on restore. The value used to be a
BackupClass parameter defaulting to 1, and the Cozystack Kafka chart
sets the broker-level min.insync.replicas to 2 as soon as kafka.replicas
reaches 2: kafka-console-producer writes with acks=all, the broker
rejects every write into a topic with fewer replicas than that, and the
producer logs the rejection and exits 0. A restore onto any multi-broker
cluster therefore landed zero records and failed the post-replay count
with a message naming the wrong cause; in place, the documented delete
had already taken the live topic with it. replicationFactor stays as an
override for restoring into a smaller cluster, and a value the target
cannot host now fails at --create in Kafka's own words.

Hold the pre-replay emptiness check to the topic's partition count, as
the backup side already does. kafka-get-offsets skips a partition it
could not read and still exits 0, so a leader election in flight dropped
the populated partitions from the listing, the surviving empty ones
summed to zero, and the replay appended a full copy into a live topic -
the exact outcome the check was moved ahead of the replay to prevent.
Both restore-side counts now go through one helper that refuses a short
listing. The data-file-versus-manifest check runs both ways before
anything is created, and the replay walks the manifest topic by topic,
announcing each one, so a retried Pod's log says which topics landed.

The demo now runs two brokers with two-replica topics, which is the
shape the chart puts min.insync.replicas=2 on; the restore steps assert
the restored topic's replication factor, and step 06 submits a second
RestoreJob against the populated topic and requires it to be refused
with the topic untouched.

Smaller findings from the same round: the topics parameter is held to
Kafka's name charset (a literal "*" no longer path-expands) and refused
on a repeated name, which produced a Succeeded backup no restore could
balance; the strategy verifies S3 TLS against the seaweedfs CA projected
into the <app>-backup-s3 Secret instead of curl -k, following the
rabbitmq example, so the e2e endpoint moves to the .svc name the serving
cert covers; the image lookup keeps digest-pinned references; the
default namespace is tenant-root, the only one the demo ever ran in;
step 05 caches the resolved Backup name for the restore steps; the decoy
count subtracts begin offsets like the topic count; the bats test name
carries its count again; and the scenario walk-throughs the sibling
examples ship are added.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-By: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov b141166cde fix(kafka): key begin-offset lookups on topic:partition 
The begin-offset lookups matched on the partition index alone, so a line
belonging to another topic could supply this partition's begin. The cut
then shifts silently and the drain check cannot see it, because n is
derived from that same wrong begin. Match on the whole topic:partition
key in all four lookups, which holds whatever order the listing arrives
in and no longer depends on the topic filter being pinned.

Hold the tarball's file names to the manifest as well. The manifest is
validated as untrusted input, but the replay derived its topic from the
data file names, so a file naming a topic the manifest never listed was
replayed into a topic that skipped the create, shape, emptiness and
post-replay checks while the restore still reported success.

Correct what the decoy is documented to prove. It reliably covers the
offset listing, the in-place delete and the two record counts; it does
not cover the two PartitionCount reads, because kafka-topics --describe
yields topics in the admin client's result-map order rather than sorted,
so which one their head -1 sees is not something a fixture can pin down.
The previous text claimed every pin reddens the run, and explained the
decoy's name by a sort order that does not govern that output.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov 17475f1ee5 fix(kafka): refuse a non-empty restore target before replaying 
The replay appends and the driver retries a Pod that died mid-replay, so
checking the record count only afterwards meant every attempt had already
written another full copy into a live topic and the check merely reported
the damage. Read the target's count before replaying and refuse a topic
that already holds records, leaving it as the failed attempt left it; the
post-replay check still verifies what landed.

Name the decoy so it sorts before the demo topic. Both PartitionCount
reads pipe --describe through head -1, so an unpinned describe reads
whichever matching topic comes first: with the decoy sorting after the
topic those two pins were never exercised and could be removed with the
suite staying green. "-" precedes ".", so orders-v1 now sorts first and
an unpinned describe reads its partition count instead.

Also state in the README that an empty backup is refused on restore
rather than merely uploaded, so the empty-tarball note does not read as
if such an artifact were usable.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov 2c9b3dfeea fix(kafka): validate the restore manifest and pin topic matching in the suite 
The manifest arrives inside the downloaded tarball but was consumed as
trusted input. A non-numeric partition field slipped through the numeric
test that computes a topic's partition count: an if condition is exempt
from errexit, so test exits 2, the term reads false, the count comes out
short, the topic is created smaller, and the shape check then compares
that short value against itself and passes. The per-topic total still
added up, so the restore reported Succeeded with every keyed record
re-placed. Validate each manifest field on read, refuse an empty manifest
whose verification loops would otherwise run zero times and report a
successful restore of nothing, and guard the restore-side PartitionCount
read the way the backup side already guards its own.

The delete-wait still matched the topic name as a regex; use grep -qxF.
The suite now pins that class of bug instead of relying on review: the
demo topic is orders.v1 and a decoy ordersXv1 sits beside it, matching the
first as a regex but not as a literal. The decoy is never named in the
BackupClass, so dropping any \Q...\E pin fails the run on its own - the
partition-set guard counts the decoy's partitions, or the in-place delete
takes the decoy with it.

Also: the host-side record count skipped a partition missing from the
begin-offset listing instead of defaulting it to 0, undercounting the
topic; the two preflight invariants now cover the kafka flow they already
describe; and the README names the disabled TLS verification.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov 9cb9c70083 fix(kafka): match topic names literally, not as regexes 
kafka-topics and kafka-get-offsets compile --topic into a Java regex and
full-match it, so any name carrying a metacharacter also selects its
siblings - "audit.events" matches "audit-events". Two consequences here:

- The in-place step's `--delete --topic "$TOPIC"` deleted every topic the
  name happened to match, which for a demo meant destroying a bystander
  topic to "simulate data loss".
- The partition-set guard read another topic's partitions into `ends` and
  another topic's PartitionCount into `pc`, so it either refused a legal
  backup or, when the counts coincided, wrote duplicate manifest lines and
  overwrote a data file.

Pin every filtering call with \Q...\E, the same fix the kafka-metadata
example carries. --create and the console producer/consumer take a literal
name and stay unquoted.

Also: verify the restore target's partition count matches the backup, since
--if-not-exists leaves an existing topic's shape alone and the post-replay
total cannot see re-placed records; stop wrapping the whole kafka_run in
2>/dev/null, which hid pod and API failures rather than in-pod noise; and
point the warned image fallback at the newest entry of the operator's map,
the one the resolver itself picks, instead of the oldest.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov d405a36e3c fix(kafka): validate the backup's partition set against the topic 
The drain guard only covers a short read within a partition it was told
about. kafka-get-offsets lists just the partitions whose lookup
succeeded - GetOffsetShell logs a per-partition KafkaException to stderr,
skips it and still exits 0 - so a partition lost to a leader election got
no manifest line, no data file and no drain check, and the tarball was
uploaded with the Backup reporting Succeeded. The restore-side count
check could not catch it either, since it sums the same short manifest,
and a missing highest-numbered partition made restore recreate the topic
with fewer partitions, re-placing every keyed record.

Read the topic's own PartitionCount before the drain and fail closed when
the offset listing is short. A short "begins" needs no separate check:
its per-partition default of 0 can only overstate n, which the existing
drain check then catches.

Also run the shared BucketAccess preflight in the e2e - accessGranted
does not prove the S3 server reloaded the new identity, and a fresh grant
answering the SigV4 PUT with 403 reds the run as a backup defect - and
state the .chainsaw.yaml namespace override as the rule it is rather than
as a claim about which Tests match a name pattern.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov 7d8dee66a5 fix(kafka): verify restore landed and namespace-scope the S3 key 
- Restore verification: kafka-console-producer, like the consumer on
  backup, exits 0 without every record being accepted, so a replay that
  dropped records still reported Succeeded. After the replay, compare
  each restored topic's record count (sum of end-begin over partitions)
  against the manifest and fail on any shortfall or overshoot - the
  latter being a restore into a non-empty topic, which this appending
  DATA restore does not support.
- Namespace-scope the S3 object key (<namespace>/<app>/kafka-topics.tar)
  so two same-named apps in different namespaces sharing one bucket do
  not collide. The namespace is .Release.Namespace in both modes and
  always names the source, since a to-copy target shares the source's
  namespace (a TypedLocalObjectReference).
- Warn instead of failing open when the Kafka image cannot be resolved
  from the operator: the pinned public literal is unreachable on an
  air-gapped cluster, so tell the caller to set KAFKA_IMAGE.
- Word the .chainsaw.yaml namespace note as "most" (etcd-2-backup-
  roundtrip sets no namespace) and document the empty all-topics backup.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov 9673af6aca fix(kafka): guard short backup drains and settle example teardown 
- Short-drain guard: kafka-console-consumer exits 0 after a timed-out
  short read, so a truncated drain was tarred and reported Succeeded.
  Compare the drained line count against end-begin per partition and
  fail the backup on any shortfall, making the README's "fails closed"
  promise real for compacted/transactional topics.
- Settling teardown: cleanup.sh returned before the Kafka apps and
  Bucket had uninstalled and could not report failure, letting the
  neighbouring chainsaw/kafka Test contend on PVC reclaim. Add a
  wait_deleted helper and a teardown_rc gate that waits for the heavy
  objects and exits non-zero when the teardown does not settle.
- Fully qualify the ambiguous delete kinds (backup/bucket/backupclass/
  restorejob/backupjob) so a bare shortcut cannot target a foreign group
  and silently leave the Cozystack artifact behind.
- Default KAFKA_IMAGE from the running operator (newest STRIMZI_KAFKA_
  IMAGES entry), falling back to the literal only when the operator
  cannot be read; drop the duplicate resolution note.
- Make the strategy and throwaway CLI Pods restricted-clean
  (runAsNonRoot + JSON-patch securityContext on the CLI Pods).
- gitignore .source-dump.txt; replace the deprecated "small" preset with
  c1.small; capture the to-copy cluster in the e2e catch; document the
  all-topics risk; de-enumerate the decaying suite-list comments.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov 21f5b2db2d fix(kafka): correct internal-topic filter and harden the delete-wait 
Two accuracy fixes in the all-topics path and the in-place delete-wait.
The all-topics default filtered `^_`, which also drops single-underscore
user topics such as `_schemas`, silently excluding user data from a
backup that still reports success; filter `^__` (Kafka's internal topics
are double-underscore). The in-place delete-wait ran `--list | grep` in a
pod snippet under `set -eu` without pipefail, so a transient `--list`
failure read as empty output and false-positived "deleted"; capture
`--list` on its own and check its exit before trusting it.

Sharpen the docs: the frozen-end-offset cut is per-topic, not
cross-topic; restore uses the configured replication factor (default 1),
not the source's; compacted and transactional topics are out of scope;
and the endpoint parse is demo-scope (no IPv6 / path-carrying forms).

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov 34a04e6b37 chore(kafka): document copy-safety caveats and harden all-topics loop 
Document the caveats that make the example safe to copy - a null key
round-trips as the literal string "null" (changing key and compaction
identity), the drain stages the topic twice under /tmp with no
ephemeral-storage bound, every backup of an app overwrites the same S3
key, and the Pods target warn-only Pod Security. Harden the all-topics
partition-count loop: the final `[ ... ] && parts=...` returned non-zero
under `set -e` when the last test was false, so use `if ... then ... fi`.
Align the default KAFKA_IMAGE with the vendored operator (0.45.1-rc1) and
list kafka-2- in the .chainsaw.yaml tenant-root header.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov cc8c56849d test(kafka): assert restore content and exact record count 
The backup roundtrip only summed offset counts, so a restore that
replayed garbage, scrambled keys, or ran twice (the strategy Job's
backoffLimit retry re-appends from the start) still passed. Both
verifications also compared with `< MESSAGE_COUNT`, so an inflated
count slipped through.

Snapshot the source topic in step 04 as a per-partition, offset-ordered
partition/key/value dump, and diff the restored topic against it in the
in-place and to-copy steps: byte-equality proves keys, values, partition
placement and per-partition ordering survived, not merely the count.
Make both count checks exact.

Document that restore appends into a non-empty topic and is safe only
against an absent or empty one. Raise the helper Pods'
--pod-running-timeout to 5m so a cold image pull on a first run outside
CI does not abort before seeding.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov d38b538d98 test(kafka): fail fast on terminal backup phases and capture strategy Job logs 
Two CI-quality fixes, both diagnostics/wall-clock only (neither can turn a
run falsely green):

- The chainsaw comment claimed run-all.sh fails fast on terminal
  BackupJob/RestoreJob phases and Stalled HelmReleases, but the kafka helpers
  never implemented it (carried over from mariadb). A BackupJob that settled
  phase=Failed was polled the full 600s and then mislabelled 'Timeout',
  burning ~10 of the 40 chainsaw minutes per already-red run. Port the mariadb
  variants: wait_for_field gains a terminal fail_value arg (call sites pass
  Failed), and a new wait_hr_ready fails fast on Stalled=True with an existence
  backstop. HR waits in steps 03/04/07 now use it; the Strimzi Kafka CR waits
  keep their genuine Ready condition. run-all.sh runs under set -e, so a
  non-zero step aborts the flow immediately, matching the comment.

- The catch block only captured the Strimzi broker Pods
  (app.kubernetes.io/instance=kafka-kafka-test); the generic Job strategy runs
  the drain/replay in controller-created Pods labelled
  job.strategy.backups.cozystack.io/mode=backup|restore (never that instance
  label) and not TTL-reaped, so the actual S3/SigV4 error left no log in the
  report. Add a second podLogs selector keyed on that label.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov b099f8dde4 fix(kafka): fail closed when listing topics for an all-topics backup 
In the empty-TOPICS (back up all non-internal topics) branch, the topic
list was resolved with a single pipeline under set -eu without pipefail:

  TLIST=$(kafka-topics.sh --list | grep -v '^_' || true)

The pipeline's exit status is grep's alone, so a real broker error
(unreachable / auth) that emits no stdout makes grep exit 1, || true swallows
it, and TLIST is empty - indistinguishable from 'no non-internal topics'. The
backup then drains nothing, tars an empty dir, PUTs it and reports success: a
silent empty backup a later restore would treat as authoritative.

Split the --list into its own command substitution so a broker error aborts
the Pod under set -e; filter internal topics only afterwards, so an empty
TLIST means genuinely no user topics.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov c84f77ac16 fix(kafka): sign S3 host without non-default port in backup strategy 
The Strimzi image ships curl 7.76.1, whose --aws-sigv4 omits a
non-default port from the SigV4 canonical Host while still sending it in the
Host header (a bug fixed in curl 7.87.0). seaweedfs recomputes the signature
from the received host:port, so every PUT/GET to the :8333 in-cluster S3
Service the e2e harness targets was rejected with HTTP 403, failing
kafka-2-backup-roundtrip at the backup upload. It passed on dev7 only because
that run used a :443 ingress, where the port is absent from the Host anyway.

Strip the port from the signed and sent URL and redirect the connection to the
real port with --connect-to, so the signed, sent and server-side Host all
agree on the bare host. Verified against the image's curl 7.76.1: the emitted
SigV4 signature matches a portless canonical Host, and the request still
reaches the :8333 listener. Correct on curl >= 7.87 too, which drops the
default port from the canonical Host as well.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Andrey Kolkov 7842b88676 feat(kafka): add topic-data backup/restore example and e2e roundtrip 
Add examples/backups/kafka, a worked backup/restore of a Cozystack-managed
Kafka application's topic data via the generic Job backup strategy - the
same pattern as examples/backups/nats, with no purpose-built backup image:
a stock Strimzi Kafka image (kafka-*.sh CLI + bash/curl/tar) runs one
shell script that branches on .Mode.

Consistency is a frozen-end-offset cut: kafka-get-offsets --time -1 is read
once per partition at backup start and the consumer drains only up to that
offset, so anything produced during the run is excluded and every partition
is captured as of the same instant. Restore recreates each topic with its
original partition count and replays every partition file; keyed records
reproduce into their original partition via the default partitioner.

This is deliberately a DATA-only backup: topic configs, ACLs, SCRAM users
and consumer-group offsets are not captured, and re-produced records get
fresh offsets. The README spells out the scope and points at a
volume-snapshot strategy for faithful, offset-preserving backups.

Add kafka-2-backup-roundtrip to hack/e2e-chainsaw/kafka, driving the
example run-all.sh as the harness (Bucket -> source Kafka + seeded topic
-> BackupJob -> in-place restore -> to-copy restore + verify) so the test
and the documented flow cannot drift, mirroring mariadb-2-backup-roundtrip.
The image is resolved from the running operator so the backup Pod reuses
the cached Kafka image.

Verified end to end on a live cluster: freeze cut orders:0 [0,8),
orders:1 [0,11), orders:2 [0,11) = 30 records; in-place and to-copy
restores both return 30 with the original 8/11/11 partition distribution.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
 2026-09-23 17:44:17 +04:00
Aleksei Sviridkin f629280944 fix(tests): raise the mongodb, rabbitmq and clickhouse HR waits 
Those walkthroughs gave their database HelmReleases 300s, under the
600s install timeout the generated releases carry, so a failed install
surfaced as a bare timeout rather than its own reason. None of these
charts sets a startup probe, so 660s clears both floors, as it does
for mariadb.

Each Chainsaw op that runs one of the scripts grows by exactly what
the waits it reaches gained, so it keeps the margin it had. The
clickhouse op still sits above the sum of every inner ceiling. The
rabbitmq one covers a failed install at the source wait, while the
ceilings up to its target wait add up to more than the op, as they
did before. The mongodb round-trip runs with SKIP_RESTORE, so only its
source wait counts there.

postgres keeps its 360s: whether a suite pays the extra minutes on
every red run is a per-suite call, and this one is not made here.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-23 11:56:11 +03:00
Aleksei Sviridkin 289acb2ed6 fix(tests): let the mariadb backup example cover a first boot 
The walkthrough gave each MariaDB HelmRelease 300s, less than the
release can legitimately take, so a slow first boot failed the harness
and a failed install surfaced as a bare timeout with no reason.

A wait here has two floors and must beat the larger. The chart grants a
310s startup budget, and the install can span it: helm-controller
defaults to the poller strategy, and kstatus reads the operator CR's
Ready=False as InProgress. The generated release also carries a 600s
install timeout, and a wait equal to it expires first, because its
clock starts at the apply and the install's only once the chart is
fetched. The install timeout is the binding floor, hence 660s.

On timeout and on Stalled, wait_hr_ready now dumps every condition and
the release history: a scheduled retry can overwrite the Ready message
before anyone reads it. It also stops folding different answers into one
"(none recorded)" line. An absent release, a release deleted during the
wait, a lookup kubectl could not answer, an empty history and a failed
history read each say so, with kubectl's own stderr kept, because that
message is usually the cause. A release seen earlier still gets its dump
when only the last lookup failed.

Every backup example whose wait_hr_ready carries this body takes the
same change, so the copies stay byte-identical.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-09-23 11:56:10 +03:00
Aleksei Sviridkin 01feb368a1 fix(backups): wait for the platform redis strategy in the backup demo 
The redis backup demo read cozy-default-redis once and gave up if it was
absent. That strategy is rendered behind a lookup of the platform
bucket, so it only exists after a Helm upgrade that runs once the
bucket is provisioned. The backupstrategy-controller forces that
upgrade, but helm-controller will not run it while a release in the
chart's dependsOn is not Ready.

A platform change shortly before the suite starts is enough to hold it
back. The e2e install patches the platform package late (exposed
services, then OIDC), the operator rewrites the cozystack-basics
release, which then waits on its own dependencies, and
backupstrategy-controller waits on cozystack-basics. On one run the
strategy was still missing eight minutes after those patches, and the
demo's single read failed the suite as if the backups stack were not
installed.

Wait for the object with a bound instead. 15 minutes covers the nine
minutes seen between those patches and the strategy's creation on
another run, and still leaves the demo inside its 30 minute step.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-23 05:33:08 +03:00
Aleksei Sviridkin 294b9699c3 fix(backups): read the redis demo's keys through exec, not kubectl run 
The redis backup demo read and wrote its marker key from a throwaway
`kubectl run -i --rm` Pod. kubectl returns that Pod's stdout over an
attach, the container runtime delivers only what the container writes
after the attach is registered, and kubectl falls back to the Pod log
only when the attach fails. A redis-cli that finishes in between comes
back as an empty reply with exit 0, which is how a GET of a key the
master held read as '' and failed the round-trip on PRs that do not
touch redis.

Run redis-cli by `kubectl exec` in a sentinel Pod instead, whose image
already carries it. An exec'd process writes into pipes created with
it, so nothing it prints is lost, and its exit status comes back too.

The demo's reads also stop discarding stderr. The same empty string used
to stand for a missing object, an API error and a lost stream alike, so
a red run could not say which one it was.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-23 05:32:36 +03:00
Andrey Kolkov caae13a60a docs(backups): drop the inaccurate NAMESPACE-override note from the demo runner 
run-all.sh said NAMESPACE could be overridden through the environment, but
the numbered manifests pin namespace: tenant-root while only the waits read
$NAMESPACE, so an override would create resources in one namespace and wait
in another. The demo is tenant-root-specific anyway: the system bucket
credentials and the derived strategy live there. Say that instead.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-11 16:05:23 +04:00
Andrey Kolkov bd7d4abe84 fix(backups): size the Kafka metadata demo brokers at the chart default 
The demo's source and to-copy Kafka fixtures requested resourcesPreset
"small", the deprecated legacy alias worth 1 CPU / 512Mi — half the Kafka
chart's own c1.small default (1 CPU / 1Gi). 512Mi is too little for the
broker: it crashlooped before the HelmRelease went Ready, so the demo
never reached the backup. Use c1.small on the kafka and zookeeper blocks
of both fixtures, matching the chart default and the in-tree kafka e2e
fixture that comes up cleanly.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-11 16:05:23 +04:00
Andrey Kolkov c5a8a39bcd docs(backups): name the demo BackupClass in the Kafka flow headers 
The run-all.sh flow header and the chainsaw harness header both said the
ad-hoc BackupJob runs against the platform cozy-default BackupClass. It
routes to this demo's kafka-metadata BackupClass instead, which maps Kafka
to the kafka-strategy-default strategy that run-all.sh derives from the
shipped cozy-default-kafka at run time. The numbered manifests already
described this; only the two summary headers overstated the one piece of
wiring the demo exists to exercise. Reword them to match.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-10 22:23:53 +04:00
Andrey Kolkov 2cb2b459fa fix(backups): reject a non-numeric recorded partition count on Kafka restore 
The restore path validated only the shape parsed from the live cluster,
not the partition count and replication factor read from the stored
object. A non-numeric recorded partition count made `[ "${lp}" -lt
"${parts}" ]` and its `elif` exit 2; an `if` condition is exempt from
errexit, so both read as false, the topic was skipped, and the loop still
printed its per-topic success line - so the RestoreJob reached Succeeded
without ever touching the topic. A well-formed but wrong-content object,
which curl -f cannot catch, must fail loudly like the unrecognised-record
guard.

Validate the recorded parts and rf are numeric before the comparisons and
--create, with a bats case that reddens when the guard is dropped.

Also correct a stale to-copy example comment: the strategy reads the exact
object the source backup recorded in status.artifact.uri (injected as
ARTIFACT_URI), not a key derived from .Backup.ApplicationRef.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-10 22:23:53 +04:00
Andrey Kolkov 4925a84c26 fix(backups): pin the restore-side guards and bound a running Kafka Job 
- delete_topic waited with `grep -qx` (BRE), so deleting `audit.events`
  matched the live `audit-events` and spun to its timeout — the e2e
  failed at step 25 on every run that reached it. Use `grep -qxF`.
- The two -k/--connect-to guards were written as `! grep`, which cannot
  fail under hack/cozytest.sh (it wraps each @test in `set -e` + a
  trailing `return 0`, and bash exempts a `!`-negated command from
  errexit). Rewrite them as positive `if grep ...; then return 1; fi`;
  verified the -k reinstatement now reddens the suite under cozytest.
- Cover the restore-side literal-topic guard that was unpinned at every
  level: a bats case runs the existing-topic branch with a colliding
  sibling and asserts the `--alter` argv carries `\Q…\E`, and run-all.sh
  now leaves `audit.events` live so the e2e exercises that branch instead
  of only `--create`. Add bats cases for the partition-shrink refusal, the
  `-f` upload guard, and `--exclude-internal`.
- Bound a running Job: reconcileKafka / reconcileKafkaRestore now fail the
  BackupJob/RestoreJob once kafkaDefaultBackupDeadline elapses in the
  Job-running branch (buildJobStrategyBatchJob sets no
  activeDeadlineSeconds), so an unschedulable pod no longer pins it in
  Running forever. Pinned by TestReconcileKafka_FailsWhenJobNeverCompletes.
- Fail closed on an unknown record kind and an unknown MODE, and validate
  the live partition/RF parsed on restore (an empty value silently skipped
  reconciliation). Correct the stale "image pinned in the strategy" comment
  in the example helpers and the chainsaw test (the controller now resolves
  the target broker's image as .ClientImage).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-10 22:23:52 +04:00
Andrey Kolkov ec26f1171d fix(backups): make Kafka metadata e2e reach an in-cluster S3 endpoint 
The e2e drove run-all.sh against the shipped cozy-default-kafka strategy,
whose endpoint is the platform's EXTERNAL S3 ingress. In CI that host
(s3.example.org) is an unroutable placeholder with no issued cert, so the
backup Pod died on `curl: (6) Could not resolve host` and the BackupJob
went Failed — the chainsaw round-trip failed there, not in the driver.

Mirror the rabbitmq e2e model, but without duplicating the 150-line
driver script: run-all.sh now derives a demo strategy from the shipped
cozy-default-kafka at run time (jq-patch: override S3_ENDPOINT, and when
an endpoint override is given, mount the copied self-signed seaweedfs CA
and point CURL_CA_BUNDLE at it), applies a demo BackupClass, and routes
the BackupJob/Plan to it. The chainsaw run sets
S3_ENDPOINT=https://seaweedfs-s3.tenant-root.svc:8333 (the .svc FQDN the
seaweedfs cert SAN covers). No -k; TLS is verified against the CA. With
no override (a real cluster whose external endpoint resolves and uses a
public CA) the shipped endpoint is reused and no CA is mounted, since
CURL_CA_BUNDLE replaces the system bundle rather than extending it.

The shipped cozy-default-kafka strategy is untouched. jq derivation
verified on a sample in both CA modes; example scripts pass bash -n.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-10 22:22:34 +04:00
Andrey Kolkov 0270650237 fix(backups): harden Kafka metadata S3 handling and own the artifact 
- Drop `-k`: mirror the RabbitMQ S3 layer (ARTIFACT_URI as the single
  source of truth, forcePathStyle honoured, whole base URL kept) so
  curl --aws-sigv4 verifies TLS against the ACME endpoint the helper
  returns. The derived endpoint is default-port, so the curl 7.76.1
  --connect-to port workaround is gone too; this also fixes the IPv6
  and path-prefix mangling in the old host-splitting.
- Own the artifact: a cleanup mode in the script plus cleanupKafkaBackup
  wired into BackupReconciler.cleanupOnDelete delete a deleted Backup's
  object (best-effort, waited on) instead of orphaning it, matching the
  RabbitMQ precedent. Kafka has the identical own-your-object property.
- Restore probes existence with --list (fail closed), so an unreachable
  broker aborts instead of being misread as "topic absent" and created.
- Hold the fixes with tests at the right level: a colliding topic pair
  (audit.events / audit-events, distinct partition counts) in the e2e so
  a revert of the \Q...\E literal-match breaks the round-trip, and Go
  tests pinning the scope stamp and the cleanup Job.
- Correct the docs/comments: endpoint-shape row, the "cached on nodes"
  claim, and the narrower curl 7.76.1 S3 envelope (empty-body payload
  hash) that makes a small modern-curl client image the right follow-up.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-10 22:22:06 +04:00
Andrey Kolkov 537aadf389 fix(backups): fix Kafka backup S3 signing and literal topic names 
Reproduced end-to-end against a real Kafka 3.9.1 broker:

- [CRITICAL] The Strimzi image ships curl 7.76.1, whose --aws-sigv4
  omits a non-default port from the SigV4 canonical Host while sending it
  in the Host header, so every S3 PUT/GET 403s and the driver never
  uploads or downloads. Sign/send the bare host and redirect the
  connection to the real port with --connect-to (the same fix the
  topic-data example already carries), and accept the in-cluster
  self-signed cert with -k.
- [CRITICAL] kafka-topics --topic is a Java regex, not a literal: a
  backup entry for "audit.events" also matched a live "audit-events", so
  restore repartitioned an unrelated topic irreversibly and backup
  recorded the wrong partition/replication values. Wrap the name in
  \Q...\E on every filter call (--describe on both paths, --alter on
  restore) so it matches exactly one topic; --create and
  kafka-configs --entity-name are already literal. Applied in the example
  helpers too.
- [MAJOR] cozy-default's Kafka route looked like a data backup at
  runtime. Stamp scope=topic-metadata on the produced Backup's
  DriverMetadata and say "topic definitions and configs only; no message
  data" in the Ready condition, so kubectl describe backup is honest.
- [MAJOR] Add reconcile tests for the two Go behaviours a mutation could
  silently drop: the Strimzi-Ready precondition on backup (BackupJob
  stays out of Succeeded and creates no Job) and the applicationRef Kind
  gate on the restore target.
- [MINOR] Fail closed on the config-capture grep (exit != 1 aborts) and
  clarify the image pin has no automated producer.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-10 22:21:25 +04:00
Andrey Kolkov 405eb25095 fix(backups): harden Kafka metadata backup 
- Fail closed on topic listing: set -o pipefail, capture kafka-topics
  --list --exclude-internal exit separately, and abort on failure so an
  unreachable broker can no longer produce an empty backup reported as
  Succeeded. An empty list after a successful call stays legitimate.
- Check each kafka-topics --describe exit and validate the parsed
  partition count / replication factor before writing a record, so a
  timed-out describe cannot write a corrupt orders\t\t\t line.
- Store topic configs one per line with key and value in separate
  tab-delimited fields, sourced from kafka-configs --describe (one config
  per line), so a value containing a comma (cleanup.policy=compact,delete)
  round-trips; on restore each config is applied with the value bracketed
  only when it contains a comma.
- Detect topic divergence on restore instead of a silent --if-not-exists
  no-op: grow partitions with --alter when the backup asks for more, and
  fail loudly (naming the topic and both values) when it asks for fewer
  partitions or a different replication factor. A restore that cannot
  reach the recorded state no longer reports Succeeded.
- Add KafkaStrategyKind to the RestoreJob cleanup-dispatch no-op group so
  deleting a Kafka RestoreJob does not run a pointless Velero cleanup; add
  a regression test alongside the siblings, plus a reconcile Kind-gate
  test and a rendering test that actually renders the strategy template.
- Wire the kafka-metadata Chainsaw suite into both selectors
  (select-install suite_to_source, select-e2e src_to_suites) so editing
  the kafka app or the suite selects it and the bats unit tests pass.
- Pin the client image to the operator's default broker version
  (kafka 3.9.1) with a digest, and align the example default; fix the
  values comment. Point the RabbitMQ comment references (which shipped in
  the CRD description) at the Job strategy actually mirrored, and trim the
  in-pod comments that rendered into the manifest.
- Document the Strimzi Topic Operator caveat: for a topic owned by a
  KafkaTopic CR the operator reconciles its config from the CR, so an
  in-place config restore is durable only for out-of-band topics.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-10 22:21:24 +04:00
Andrey Kolkov a410a3f48b feat(backups): Kafka topic-metadata backup via a dedicated strategy driver 
Add a strategy.backups.cozystack.io/Kafka driver that backs up the topic
metadata of a Cozystack-managed Kafka application — partition counts,
replication factors and non-default topic configs — through the Kafka
Admin API, and restores it with kafka-topics --create + kafka-configs
--alter. Modelled on the Job/Rabbitmq strategies: the driver renders a
one-shot batch Job, gates on the Strimzi Kafka cluster reporting Ready,
and keys each backup by .BackupName so runs do not overwrite each other.

Scope is deliberately narrow: message payloads, consumer-group offsets,
ACLs, quotas and KafkaUsers are out of scope, and there is no PITR.
Restored topics are created directly (unmanaged on the target) — the
accepted tradeoff of an Admin-API-only driver. Both restore flows are
implemented: in-place recreates dropped topics into the source, to-copy
applies the source's topics onto a freshly-bootstrapped empty cluster.

Wire it into cozy-default (strategy-kafka-default.yaml gated on the
bucket name, a BackupClass binding, read RBAC on kafka.strimzi.io/kafkas
for the readiness gate, and the Strimzi Kafka client image), document it
in backup-classes.md, and add an honest round-trip example plus Chainsaw
e2e (examples/backups/kafka-metadata, kafka-3-metadata-roundtrip) that
proves a per-run retention.ms sentinel survives both restores.

Unit tests cover the driver's pure helpers (application-ref validation,
readiness verdict, parameter round-trip, per-run artifact-URI keying).

Not yet verified on a live cluster.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-10 22:20:27 +04:00
Andrey Kolkov 1289f522fa fix(backups): unbreak install, kill destructive cleanup, reach the resource knob 
Store the Redis strategy Template schemaless, as the Rabbitmq strategy
already does: without the markers controller-gen emitted the full
PodTemplateSpec schema and the CRD reached ~602 KB, pushing the chart's
Helm-release Secret past the 1 MiB cap so backupstrategy-controller failed
to install at all.

Give the example strategy a cleanup branch, and both scripts an explicit
`elif restore` / `else exit 1`, so a deleted Backup rendered with
Mode=cleanup deletes its object instead of falling into the restore path
and FLUSHALLing the live master.

Forward a backupStrategyController block through the platform bundle
alongside backupStorage, so redisBackupResources — the restore-Pod sizing
knob the loader-not-ready message points at — is reachable via
spec.components.platform.values.backupStrategyController, not merely
settable on a Helm-managed Strategy CR that reverts on the next upgrade.

Correct the endpointCASecretName doc row to match the now-optional mount,
and `int` the activeDeadlineSeconds interpolation so a large values-file
override renders as an integer, not exponent form.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-09 16:53:50 +04:00
Andrey Kolkov 651a6e3023 fix(backups): harden the redis restore AOF wait and image build 
Wait for the throwaway loader's AOF rewrite to finish - no rewrite in
flight, last one ok, base AOF on disk - instead of a bare
aof_rewrite_in_progress:0 check, which right after `appendonly yes` can
fall through before the scheduled rewrite starts and replay an absent or
half-written base AOF. Same fix in the example strategy.

Declare the package image targets PHONY and serialize image-redis-backup
after image-backupstrategy-controller so their `yq -i values.yaml` digest
stamps cannot race under `make -j`. Run the demo cleanup under `set -e`
(every delete is already --ignore-not-found) so a real API failure aborts
instead of printing "Cleanup complete" over a half-torn-down demo.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-09 16:23:12 +04:00
Andrey Kolkov b4d621f4f7 fix(backups): delete Redis backup objects on cleanup and size the restore Pod 
Delete the backup object when its Backup is removed. The Redis driver
owns its S3 object (one per BackupJob, no engine retention prunes it), so
it now spawns a cleanup-mode Job that deletes the object and holds the
Backup's finalizer until it completes, mirroring the Rabbitmq driver.
Without this a retention-pruned Plan grew the bucket unboundedly and the
key was lost with the CR.

Size the strategy Pod from backupStrategyController.redisBackupResources
so an admin can raise the restore ceiling (the throwaway loader holds the
whole dataset in memory) via chart values, which a direct edit of the
cluster-scoped, Helm-managed Strategy CR could not survive. The
loader-not-ready diagnostic now names the memory limit, the likeliest
cause, instead of only the image version.

Make the endpoint CA volume optional: nothing projects that Secret into
the app namespace, so a required mount wedged every backup Pod on
FailedMount; a missing ca.crt now falls through to the system CA store
and fails the handshake fast, which is not the same as skipping verify.

Promote the Pod-level activeDeadlineSeconds to the Job so a wedged run is
bounded once, not (backoffLimit+1) times. Use find, not ls, for the base
AOF so its no-match diagnostic is not dead code under pipefail.

Pin the object key reaching the rendered Job: both fixtures now render
--key={{ .ObjectKey }} and assert it, so zeroing the key (which left the
package green) is caught. Document .ObjectKey and the cleanup mode in the
CRD render-context, list cozy-default-redis among the Helm-managed CRs,
and mirror the CA/TLS and find fixes into the example strategy.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-09 12:19:02 +04:00
Andrey Kolkov 84fe2ecf6f docs(backups): mirror the redis TLS fix into the example strategy 
The demo strategy CR states its in-Pod script is identical to the shipped
cozy-default-redis strategy, but it kept the plaintext Sentinel and master
dial after the shipped template gained server-side TLS. Add the same
optional redis-<app>.ca-cert mount and RTLS auto-detection so a tenant who
copies this example to back up a tls.enabled Redis does not reproduce the
plaintext-against-a-TLS-only-listener failure.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-08 19:13:50 +04:00
Andrey Kolkov 2d82c1ea26 test(backups): keep main's redis-tls e2e Tests; fast-fail redis backup waits 
The branch was cut before main gained the redis-tls and
redis-tls-flip-rejected Chainsaw Tests, so its copy of the redis
chainsaw-test.yaml carried only the plaintext Test plus the new
redis-2-backup-roundtrip. Carry main's two TLS Tests here too, so a conflict
resolution at merge time cannot drop the TLS end-to-end coverage regardless
of which side it takes. The file still needs a rebase onto current main to
merge cleanly; this only removes the risk of losing the Tests.

Also pass the terminal Failed value to the BackupJob/RestoreJob phase waits
so a failed run aborts immediately with a legible cause instead of polling to
the 600s timeout.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-08 17:57:06 +04:00
Andrey Kolkov ac79c3757b test(backups): self-contained Redis backup round-trip e2e 
The Redis driver shipped no round-trip backup/restore e2e, unlike its
ClickHouse, MariaDB and RabbitMQ siblings, so the backup to S3 and restore
data path had no automated coverage. The existing demo backed up through the
platform cozy-default BackupClass, whose endpoint is the shared system
bucket; in the e2e environment that bucket's advertised external address is
not routable, so a cozy-default round-trip cannot run there.

Rework examples/backups/redis to provision its OWN Bucket + Redis strategy +
BackupClass (mirroring the MariaDB and RabbitMQ demos), projecting the
bucket's keys and the seaweedfs CA into the Secrets the strategy references
and honouring an S3_ENDPOINT override so the backup Pod targets the
in-cluster seaweedfs. Add a redis-2-backup-roundtrip Chainsaw test that
drives run-all.sh in tenant-root and asserts the sentinel round-trips
through object storage in place and into a copy.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
 2026-09-08 17:57:06 +04:00
Andrey Kolkov b0eb104276 feat(backups): add Redis backup/restore strategy driver 
The spotahome RedisFailover operator ships no backup CRD, so add a
dedicated strategy.backups.cozystack.io/Redis driver that runs a one-shot
Job: it dumps an RDB snapshot from the current master (discovered via the
operator's sentinel Service) to the system bucket over S3, and on restore
replays it (RDB -> RESP via redis-cli --pipe) into the target's master.

Structurally a specialisation of the Job/Altinity script-Job pattern with
a Redis applicationRef gate and an app-readiness precondition. Backup
parameters round-trip through the Backup's DriverMetadata (no
underlyingResources snapshot); restore drives from .Backup.ApplicationRef
so a to-copy restore reads the source's object-storage prefix while
writing into a differently-named target. Both in-place and to-copy are
supported; PITR is not (RDB is a discrete snapshot).

- api: Redis/RedisList strategy type + generated CRD and deepcopy
- driver: reconcileRedis / reconcileRedisRestore + pure helpers, wired
  into both dispatch switches, supportedBackupStrategyKinds and the no-op
  restore cleanup case
- default: gated cozy-default-redis strategy (system-bucket coords via
  cozy-backups-creds env secretKeyRef, https:// scheme + optional CA) and
  the Redis route in the cozy-default BackupClass
- image: images/redis-backup bundling redis-cli + curl + an RDB->RESP
  converter, wired via backupStrategyController.redisBackupClientImage
- tests: driver unit tests (gate, precondition, render, param round-trip,
  both reconcile paths) and the helm rendering-gate coverage
- example: examples/backups/redis honest marker round-trip for both
  restore flows

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
 2026-09-08 17:56:20 +04:00
Myasnikov Daniil 186547d835 test(e2e): run the merge-gating lanes on Talos containers 
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.

The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.

Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.

Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.

The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.

Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.

One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.

Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-09-05 13:19:17 +05:00
Andrey Kolkov e94fcf8219 feat(backups): RabbitMQ definitions backup via a dedicated strategy driver (#3990) 
## What this PR does

Adds backup/restore support for the managed `RabbitMQ` application.
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so this backs up broker
**definitions** (vhosts, users, permissions, queues, exchanges,
bindings, policies, parameters) over the management HTTP API
(`GET`/`POST /api/definitions`) and stores them in the system bucket.
Message payloads and point-in-time recovery are intentionally out of
scope; a full-data backup would be a Velero volume snapshot, which is a
separate strategy.

It is delivered as a **first-class
`strategy.backups.cozystack.io/Rabbitmq` driver**, following the
Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

- `reconcileRabbitmq` / `reconcileRabbitmqRestore` render the strategy's
`PodTemplateSpec` — `curl` `GET`/`POST /api/definitions`,
pushing/pulling the JSON to S3 with `curl --aws-sigv4` — as a
`batch/v1.Job`, covering both restore flows: in-place (into the source
broker) and to-copy (into a separate, differently-named RabbitMQ in the
same namespace, authenticating with the target's own default-user
credentials and reading the source's object key).
- A **Go-read precondition** gates each run on the `RabbitmqCluster`
reporting `AllReplicasReady` (read through a small `rabbitmqtypes`
partial type), so a not-yet-ready broker surfaces a precise
`Ready=False` wait on the BackupJob/RestoreJob rather than a Job that
starts and fails against an unreachable management API.
- The produced `Backup` records the stored object's location on
`status.artifact.uri`, rendered from the strategy's
`spec.artifactURITemplate`. The `Backup` CRD has no status subresource,
so this persists on Create, mirroring the Etcd driver.

Dispatch is wired in both controllers (plus
`supportedBackupStrategyKinds`), `rabbitmqtypes` is registered in the
scheme, and RBAC grants read on `rabbitmqclusters.rabbitmq.com` (the Job
itself is already covered by the existing `batch/jobs` grant). It ships
`cozy-default-rabbitmq` (a `$bucketName`-gated `Rabbitmq` strategy) and
routes `apps.cozystack.io/RabbitMQ` to it in the `cozy-default`
BackupClass, which also enrols it in the `DefaultObjectsGate`. The
RabbitMQ application chart is left untouched — it had no prior backup
wiring, and the strategy carries the S3 coordinates itself.

Unit tests cover the driver's pure helpers (applicationRef validation,
cluster-name derivation, the readiness-message builder, artifact-URI
rendering, parameter round-trip) and the reconcile precondition gate
(defers with `Ready=False` and no Job until the cluster is ready, then
proceeds). A runnable example under `examples/backups/rabbitmq/` and an
honest-roundtrip Chainsaw e2e under `hack/e2e-chainsaw/rabbitmq/` seed a
sentinel vhost, back it up, drop it, and assert it round-trips through
object storage on both restore flows.

The Job pod reuses the existing `platform-migrations` image (curl +
ca-certificates) under a new
`backupStrategyController.rabbitmqBackupClientImage` key, stamped in
lockstep with `chBackupClientImage` in `packages/core/platform/Makefile`
so the repository stays pinned at a single digest
(`hack/image-pin-consistency.bats`). A dedicated slim image built by
this package is a follow-up.

**Validation:** `go build`/`go vet`/`gofmt`/`go test
./internal/backupcontroller/... ./internal/template/...`, deepcopy +
Rabbitmq CRD generation, `helm template` proving the CRD-ordering gate
both on and off, `helm lint`, and `helm unittest` (18/18). The backup +
in-place + to-copy roundtrip was verified end-to-end on a dev cluster (a
real management-API export and a real S3 `--aws-sigv4` upload/download
against SeaweedFS), so the driver's core contract is exercised on a real
cluster; the in-CI Chainsaw run is the remaining automated evidence. The
object-cleanup, terminating-namespace teardown and readiness-deadline
paths added during review are unit-tested (including `Reconcile`-level
finalizer retention and the terminating-namespace release) but not yet
cluster-exercised. (Note found while testing: RabbitMQ 4.2 is OOMKilled
on the `nano` preset, so the example/e2e use `c1.small`.)

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map against the diff. This is an additive backup
strategy plus a new strategy CRD; no downstream repository is
mechanically forced by the change. A documentation page for RabbitMQ
backup on [cozystack/website](https://github.com/cozystack/website)
would be a reasonable follow-up, but it is not required by this diff —
leaving the boxes unticked for a human to decide rather than ticking "no
downstream" to clear the checklist.

### Release note

```release-note
feat(backups): add RabbitMQ definitions backup/restore via a dedicated strategy.backups.cozystack.io/Rabbitmq driver; it gates each run on the RabbitmqCluster being ready and records the stored object on Backup.status.artifact.uri.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added RabbitMQ backup and restore support for definitions, including
vhosts, users, queues, exchanges, bindings, policies, and related
settings.
  * Added in-place and restore-to-copy workflows with readiness checks.
  * Added scheduled and on-demand backup examples.
  * Integrated RabbitMQ backups with the default backup configuration.
* Backups store definitions only; message payloads and point-in-time
recovery are not included.
* **Documentation**
  * Added a complete RabbitMQ backup and restore demonstration guide.
* **Tests**
* Added end-to-end coverage for readiness and backup/restore round
trips.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-03 16:03:46 +04:00
Andrey Kolkov 6eba4d9fa7 test(backups): use .svc FQDN for rabbitmq backup e2e S3 endpoint 
The reworked rabbitmq backup roundtrip reached the upload step and failed
TLS verification: `curl (60) no alternative certificate subject name matches
target hostname 'seaweedfs-s3.tenant-root'`. The demo Bucket, in-cluster
endpoint resolution and the copied SeaweedFS CA all worked; the remaining
gap is the hostname. The seaweedfs filer serving cert's SANs are
`*.tenant-root.svc` and `*.tenant-root.svc.cluster.local`, which the 2-label
`seaweedfs-s3.tenant-root` matches neither; the 3-label `.svc` FQDN does (and
still resolves via search-domain expansion). Point S3_ENDPOINT at
`https://seaweedfs-s3.tenant-root.svc:8333`.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-02 21:25:21 +04:00
Andrey Kolkov 329ef9cf3f test(backups): make rabbitmq backup e2e self-contained via demo bucket 
The rabbitmq backup roundtrip was the only backup e2e routing through the
platform cozy-default system bucket, whose advertised endpoint in the CI
sandbox is the unroutable s3.example.org placeholder, so the backup Job
failed with "Could not resolve host: s3.example.org". Mirror the
clickhouse/mariadb backup examples: provision a demo Bucket, project its
access keys into rabbitmq-backup-creds and copy the seaweedfs CA into
rabbitmq-backup-ca, and render a self-contained Rabbitmq strategy +
rabbitmq-default BackupClass whose curl uploads to the in-cluster seaweedfs
endpoint (S3_ENDPOINT override, CURL_CA_BUNDLE for the self-signed cert).

The in-Pod script is kept identical to the shipped cozy-default-rabbitmq
strategy, which is left unchanged; the demo only reuses its client image.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-02 15:37:30 +04:00
Andrey KolkovandClaude Opus 4.8 ba32df69c0 feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver 
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.

Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

  - reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
    PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
    --aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
  - A Go-read precondition gates each run on the RabbitmqCluster reporting
    AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
    precise Ready=False wait instead of a Job that starts and fails.
  - The produced Backup records the object location on status.artifact.uri
    (rendered from spec.artifactURITemplate; the Backup CRD has no status
    subresource, so it persists on Create, as in the Etcd driver).

Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.

Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.

The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-01 18:21:10 +04:00
Andrey KolkovandClaude Opus 4.8 5bcdb8cfb8 fix(postgres): let in-place restore re-bootstrap instead of wedging 
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.

1. Archive-vs-recovery serverName collision. The driver set the restored
   cluster's WAL-archive serverName equal to the recovery source
   (bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
   chart pinned the archiver serverName to the release name. In place those
   coincide, so the cluster archived onto the source's own WAL prefix and
   barman-cloud-check-wal-archive refused to start ("Expected empty
   archive"). Fixed by giving the restored cluster a fresh
   bootstrap.newServerName, keyed by the RestoreJob UID (unique per
   restore, stable across reconciles); the chart archives there while
   externalClusters still recovers from bootstrap.serverName. This is what
   the manual CNPG strategies hit (serverName defaults to the cluster
   name); the cozy-default flow dodged it (serverName is <ns>-<app>).

2. useSystemBucket guard. The restore populates explicit S3 coordinates
   (destinationPath / s3CredentialsSecret / endpointCA) but left
   backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
   incompatible with useSystemBucket" guard failed the HelmRelease upgrade
   and the cluster never re-rendered. This is what the cozy-default
   (system-bucket) flow hits on an in-place restore. Fixed by clearing
   useSystemBucket in the restore patch, since the app is now on the
   explicit-creds path. The driver's app view (postgresapp.Backup) gains
   the field, without omitempty so a false survives the merge patch.

Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-08-25 17:31:53 +04:00
Aleksei Sviridkin e441edc2aa test(e2e): stop the mongodb backup suite before the restore leg 
A to-copy restore replays the source cluster's system users into the
target, so the target's own generated credentials stop authenticating
and the harness cannot read the sentinel back out of the restored copy.
The suite has never passed for that reason, holding the in-tree e2e run
red on every branch that selects it.

Drive run-all.sh with SKIP_RESTORE=1 so the suite still proves the
backup half end to end, and pin the skip by asserting the skip branch's
own log line: exit zero alone is also what a completed round-trip
returns, so without that assertion the step could not tell a skipped
flow from a full one, nor notice SKIP_RESTORE failing to reach the
harness.

The to-copy manifest and the example README both stated that CI covers
the restore, which stops being true here, so both now say where
coverage ends. The asserted log line is annotated at its source, since
rewording it would red the suite with a bare match failure.

Coverage returns by dropping SKIP_RESTORE together with the assertion
on that line, once the credential handling behind the restore is
fixed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-14 21:18:09 +03:00