mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-30 08:37:31 +09:00
feat/vm-disk-http-auth
56
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Andrey Kolkov
|
caae13a60a |
docs(backups): drop the inaccurate NAMESPACE-override note from the demo runner
run-all.sh said NAMESPACE could be overridden through the environment, but the numbered manifests pin namespace: tenant-root while only the waits read $NAMESPACE, so an override would create resources in one namespace and wait in another. The demo is tenant-root-specific anyway: the system bucket credentials and the derived strategy live there. Say that instead. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
Andrey Kolkov
|
bd7d4abe84 |
fix(backups): size the Kafka metadata demo brokers at the chart default
The demo's source and to-copy Kafka fixtures requested resourcesPreset "small", the deprecated legacy alias worth 1 CPU / 512Mi — half the Kafka chart's own c1.small default (1 CPU / 1Gi). 512Mi is too little for the broker: it crashlooped before the HelmRelease went Ready, so the demo never reached the backup. Use c1.small on the kafka and zookeeper blocks of both fixtures, matching the chart default and the in-tree kafka e2e fixture that comes up cleanly. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
Andrey Kolkov
|
c5a8a39bcd |
docs(backups): name the demo BackupClass in the Kafka flow headers
The run-all.sh flow header and the chainsaw harness header both said the ad-hoc BackupJob runs against the platform cozy-default BackupClass. It routes to this demo's kafka-metadata BackupClass instead, which maps Kafka to the kafka-strategy-default strategy that run-all.sh derives from the shipped cozy-default-kafka at run time. The numbered manifests already described this; only the two summary headers overstated the one piece of wiring the demo exists to exercise. Reword them to match. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
Andrey Kolkov
|
2cb2b459fa |
fix(backups): reject a non-numeric recorded partition count on Kafka restore
The restore path validated only the shape parsed from the live cluster,
not the partition count and replication factor read from the stored
object. A non-numeric recorded partition count made `[ "${lp}" -lt
"${parts}" ]` and its `elif` exit 2; an `if` condition is exempt from
errexit, so both read as false, the topic was skipped, and the loop still
printed its per-topic success line - so the RestoreJob reached Succeeded
without ever touching the topic. A well-formed but wrong-content object,
which curl -f cannot catch, must fail loudly like the unrecognised-record
guard.
Validate the recorded parts and rf are numeric before the comparisons and
--create, with a bats case that reddens when the guard is dropped.
Also correct a stale to-copy example comment: the strategy reads the exact
object the source backup recorded in status.artifact.uri (injected as
ARTIFACT_URI), not a key derived from .Backup.ApplicationRef.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
|
||
|
Andrey Kolkov
|
4925a84c26 |
fix(backups): pin the restore-side guards and bound a running Kafka Job
- delete_topic waited with `grep -qx` (BRE), so deleting `audit.events` matched the live `audit-events` and spun to its timeout — the e2e failed at step 25 on every run that reached it. Use `grep -qxF`. - The two -k/--connect-to guards were written as `! grep`, which cannot fail under hack/cozytest.sh (it wraps each @test in `set -e` + a trailing `return 0`, and bash exempts a `!`-negated command from errexit). Rewrite them as positive `if grep ...; then return 1; fi`; verified the -k reinstatement now reddens the suite under cozytest. - Cover the restore-side literal-topic guard that was unpinned at every level: a bats case runs the existing-topic branch with a colliding sibling and asserts the `--alter` argv carries `\Q…\E`, and run-all.sh now leaves `audit.events` live so the e2e exercises that branch instead of only `--create`. Add bats cases for the partition-shrink refusal, the `-f` upload guard, and `--exclude-internal`. - Bound a running Job: reconcileKafka / reconcileKafkaRestore now fail the BackupJob/RestoreJob once kafkaDefaultBackupDeadline elapses in the Job-running branch (buildJobStrategyBatchJob sets no activeDeadlineSeconds), so an unschedulable pod no longer pins it in Running forever. Pinned by TestReconcileKafka_FailsWhenJobNeverCompletes. - Fail closed on an unknown record kind and an unknown MODE, and validate the live partition/RF parsed on restore (an empty value silently skipped reconciliation). Correct the stale "image pinned in the strategy" comment in the example helpers and the chainsaw test (the controller now resolves the target broker's image as .ClientImage). Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
Andrey Kolkov
|
ec26f1171d |
fix(backups): make Kafka metadata e2e reach an in-cluster S3 endpoint
The e2e drove run-all.sh against the shipped cozy-default-kafka strategy, whose endpoint is the platform's EXTERNAL S3 ingress. In CI that host (s3.example.org) is an unroutable placeholder with no issued cert, so the backup Pod died on `curl: (6) Could not resolve host` and the BackupJob went Failed — the chainsaw round-trip failed there, not in the driver. Mirror the rabbitmq e2e model, but without duplicating the 150-line driver script: run-all.sh now derives a demo strategy from the shipped cozy-default-kafka at run time (jq-patch: override S3_ENDPOINT, and when an endpoint override is given, mount the copied self-signed seaweedfs CA and point CURL_CA_BUNDLE at it), applies a demo BackupClass, and routes the BackupJob/Plan to it. The chainsaw run sets S3_ENDPOINT=https://seaweedfs-s3.tenant-root.svc:8333 (the .svc FQDN the seaweedfs cert SAN covers). No -k; TLS is verified against the CA. With no override (a real cluster whose external endpoint resolves and uses a public CA) the shipped endpoint is reused and no CA is mounted, since CURL_CA_BUNDLE replaces the system bundle rather than extending it. The shipped cozy-default-kafka strategy is untouched. jq derivation verified on a sample in both CA modes; example scripts pass bash -n. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
Andrey Kolkov
|
0270650237 |
fix(backups): harden Kafka metadata S3 handling and own the artifact
- Drop `-k`: mirror the RabbitMQ S3 layer (ARTIFACT_URI as the single source of truth, forcePathStyle honoured, whole base URL kept) so curl --aws-sigv4 verifies TLS against the ACME endpoint the helper returns. The derived endpoint is default-port, so the curl 7.76.1 --connect-to port workaround is gone too; this also fixes the IPv6 and path-prefix mangling in the old host-splitting. - Own the artifact: a cleanup mode in the script plus cleanupKafkaBackup wired into BackupReconciler.cleanupOnDelete delete a deleted Backup's object (best-effort, waited on) instead of orphaning it, matching the RabbitMQ precedent. Kafka has the identical own-your-object property. - Restore probes existence with --list (fail closed), so an unreachable broker aborts instead of being misread as "topic absent" and created. - Hold the fixes with tests at the right level: a colliding topic pair (audit.events / audit-events, distinct partition counts) in the e2e so a revert of the \Q...\E literal-match breaks the round-trip, and Go tests pinning the scope stamp and the cleanup Job. - Correct the docs/comments: endpoint-shape row, the "cached on nodes" claim, and the narrower curl 7.76.1 S3 envelope (empty-body payload hash) that makes a small modern-curl client image the right follow-up. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
Andrey Kolkov
|
537aadf389 |
fix(backups): fix Kafka backup S3 signing and literal topic names
Reproduced end-to-end against a real Kafka 3.9.1 broker: - [CRITICAL] The Strimzi image ships curl 7.76.1, whose --aws-sigv4 omits a non-default port from the SigV4 canonical Host while sending it in the Host header, so every S3 PUT/GET 403s and the driver never uploads or downloads. Sign/send the bare host and redirect the connection to the real port with --connect-to (the same fix the topic-data example already carries), and accept the in-cluster self-signed cert with -k. - [CRITICAL] kafka-topics --topic is a Java regex, not a literal: a backup entry for "audit.events" also matched a live "audit-events", so restore repartitioned an unrelated topic irreversibly and backup recorded the wrong partition/replication values. Wrap the name in \Q...\E on every filter call (--describe on both paths, --alter on restore) so it matches exactly one topic; --create and kafka-configs --entity-name are already literal. Applied in the example helpers too. - [MAJOR] cozy-default's Kafka route looked like a data backup at runtime. Stamp scope=topic-metadata on the produced Backup's DriverMetadata and say "topic definitions and configs only; no message data" in the Ready condition, so kubectl describe backup is honest. - [MAJOR] Add reconcile tests for the two Go behaviours a mutation could silently drop: the Strimzi-Ready precondition on backup (BackupJob stays out of Succeeded and creates no Job) and the applicationRef Kind gate on the restore target. - [MINOR] Fail closed on the config-capture grep (exit != 1 aborts) and clarify the image pin has no automated producer. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
Andrey Kolkov
|
405eb25095 |
fix(backups): harden Kafka metadata backup
- Fail closed on topic listing: set -o pipefail, capture kafka-topics --list --exclude-internal exit separately, and abort on failure so an unreachable broker can no longer produce an empty backup reported as Succeeded. An empty list after a successful call stays legitimate. - Check each kafka-topics --describe exit and validate the parsed partition count / replication factor before writing a record, so a timed-out describe cannot write a corrupt orders\t\t\t line. - Store topic configs one per line with key and value in separate tab-delimited fields, sourced from kafka-configs --describe (one config per line), so a value containing a comma (cleanup.policy=compact,delete) round-trips; on restore each config is applied with the value bracketed only when it contains a comma. - Detect topic divergence on restore instead of a silent --if-not-exists no-op: grow partitions with --alter when the backup asks for more, and fail loudly (naming the topic and both values) when it asks for fewer partitions or a different replication factor. A restore that cannot reach the recorded state no longer reports Succeeded. - Add KafkaStrategyKind to the RestoreJob cleanup-dispatch no-op group so deleting a Kafka RestoreJob does not run a pointless Velero cleanup; add a regression test alongside the siblings, plus a reconcile Kind-gate test and a rendering test that actually renders the strategy template. - Wire the kafka-metadata Chainsaw suite into both selectors (select-install suite_to_source, select-e2e src_to_suites) so editing the kafka app or the suite selects it and the bats unit tests pass. - Pin the client image to the operator's default broker version (kafka 3.9.1) with a digest, and align the example default; fix the values comment. Point the RabbitMQ comment references (which shipped in the CRD description) at the Job strategy actually mirrored, and trim the in-pod comments that rendered into the manifest. - Document the Strimzi Topic Operator caveat: for a topic owned by a KafkaTopic CR the operator reconciles its config from the CR, so an in-place config restore is durable only for out-of-band topics. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
Andrey Kolkov
|
a410a3f48b |
feat(backups): Kafka topic-metadata backup via a dedicated strategy driver
Add a strategy.backups.cozystack.io/Kafka driver that backs up the topic metadata of a Cozystack-managed Kafka application — partition counts, replication factors and non-default topic configs — through the Kafka Admin API, and restores it with kafka-topics --create + kafka-configs --alter. Modelled on the Job/Rabbitmq strategies: the driver renders a one-shot batch Job, gates on the Strimzi Kafka cluster reporting Ready, and keys each backup by .BackupName so runs do not overwrite each other. Scope is deliberately narrow: message payloads, consumer-group offsets, ACLs, quotas and KafkaUsers are out of scope, and there is no PITR. Restored topics are created directly (unmanaged on the target) — the accepted tradeoff of an Admin-API-only driver. Both restore flows are implemented: in-place recreates dropped topics into the source, to-copy applies the source's topics onto a freshly-bootstrapped empty cluster. Wire it into cozy-default (strategy-kafka-default.yaml gated on the bucket name, a BackupClass binding, read RBAC on kafka.strimzi.io/kafkas for the readiness gate, and the Strimzi Kafka client image), document it in backup-classes.md, and add an honest round-trip example plus Chainsaw e2e (examples/backups/kafka-metadata, kafka-3-metadata-roundtrip) that proves a per-run retention.ms sentinel survives both restores. Unit tests cover the driver's pure helpers (application-ref validation, readiness verdict, parameter round-trip, per-run artifact-URI keying). Not yet verified on a live cluster. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
Andrey Kolkov
|
1289f522fa |
fix(backups): unbreak install, kill destructive cleanup, reach the resource knob
Store the Redis strategy Template schemaless, as the Rabbitmq strategy already does: without the markers controller-gen emitted the full PodTemplateSpec schema and the CRD reached ~602 KB, pushing the chart's Helm-release Secret past the 1 MiB cap so backupstrategy-controller failed to install at all. Give the example strategy a cleanup branch, and both scripts an explicit `elif restore` / `else exit 1`, so a deleted Backup rendered with Mode=cleanup deletes its object instead of falling into the restore path and FLUSHALLing the live master. Forward a backupStrategyController block through the platform bundle alongside backupStorage, so redisBackupResources — the restore-Pod sizing knob the loader-not-ready message points at — is reachable via spec.components.platform.values.backupStrategyController, not merely settable on a Helm-managed Strategy CR that reverts on the next upgrade. Correct the endpointCASecretName doc row to match the now-optional mount, and `int` the activeDeadlineSeconds interpolation so a large values-file override renders as an integer, not exponent form. Assisted-By: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
651a6e3023 |
fix(backups): harden the redis restore AOF wait and image build
Wait for the throwaway loader's AOF rewrite to finish - no rewrite in flight, last one ok, base AOF on disk - instead of a bare aof_rewrite_in_progress:0 check, which right after `appendonly yes` can fall through before the scheduled rewrite starts and replay an absent or half-written base AOF. Same fix in the example strategy. Declare the package image targets PHONY and serialize image-redis-backup after image-backupstrategy-controller so their `yq -i values.yaml` digest stamps cannot race under `make -j`. Run the demo cleanup under `set -e` (every delete is already --ignore-not-found) so a real API failure aborts instead of printing "Cleanup complete" over a half-torn-down demo. Assisted-By: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
b4d621f4f7 |
fix(backups): delete Redis backup objects on cleanup and size the restore Pod
Delete the backup object when its Backup is removed. The Redis driver
owns its S3 object (one per BackupJob, no engine retention prunes it), so
it now spawns a cleanup-mode Job that deletes the object and holds the
Backup's finalizer until it completes, mirroring the Rabbitmq driver.
Without this a retention-pruned Plan grew the bucket unboundedly and the
key was lost with the CR.
Size the strategy Pod from backupStrategyController.redisBackupResources
so an admin can raise the restore ceiling (the throwaway loader holds the
whole dataset in memory) via chart values, which a direct edit of the
cluster-scoped, Helm-managed Strategy CR could not survive. The
loader-not-ready diagnostic now names the memory limit, the likeliest
cause, instead of only the image version.
Make the endpoint CA volume optional: nothing projects that Secret into
the app namespace, so a required mount wedged every backup Pod on
FailedMount; a missing ca.crt now falls through to the system CA store
and fails the handshake fast, which is not the same as skipping verify.
Promote the Pod-level activeDeadlineSeconds to the Job so a wedged run is
bounded once, not (backoffLimit+1) times. Use find, not ls, for the base
AOF so its no-match diagnostic is not dead code under pipefail.
Pin the object key reaching the rendered Job: both fixtures now render
--key={{ .ObjectKey }} and assert it, so zeroing the key (which left the
package green) is caught. Document .ObjectKey and the cleanup mode in the
CRD render-context, list cozy-default-redis among the Helm-managed CRs,
and mirror the CA/TLS and find fixes into the example strategy.
Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey Kolkov
|
84fe2ecf6f |
docs(backups): mirror the redis TLS fix into the example strategy
The demo strategy CR states its in-Pod script is identical to the shipped cozy-default-redis strategy, but it kept the plaintext Sentinel and master dial after the shipped template gained server-side TLS. Add the same optional redis-<app>.ca-cert mount and RTLS auto-detection so a tenant who copies this example to back up a tls.enabled Redis does not reproduce the plaintext-against-a-TLS-only-listener failure. Assisted-By: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
2d82c1ea26 |
test(backups): keep main's redis-tls e2e Tests; fast-fail redis backup waits
The branch was cut before main gained the redis-tls and redis-tls-flip-rejected Chainsaw Tests, so its copy of the redis chainsaw-test.yaml carried only the plaintext Test plus the new redis-2-backup-roundtrip. Carry main's two TLS Tests here too, so a conflict resolution at merge time cannot drop the TLS end-to-end coverage regardless of which side it takes. The file still needs a rebase onto current main to merge cleanly; this only removes the risk of losing the Tests. Also pass the terminal Failed value to the BackupJob/RestoreJob phase waits so a failed run aborts immediately with a legible cause instead of polling to the 600s timeout. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
Andrey Kolkov
|
ac79c3757b |
test(backups): self-contained Redis backup round-trip e2e
The Redis driver shipped no round-trip backup/restore e2e, unlike its ClickHouse, MariaDB and RabbitMQ siblings, so the backup to S3 and restore data path had no automated coverage. The existing demo backed up through the platform cozy-default BackupClass, whose endpoint is the shared system bucket; in the e2e environment that bucket's advertised external address is not routable, so a cozy-default round-trip cannot run there. Rework examples/backups/redis to provision its OWN Bucket + Redis strategy + BackupClass (mirroring the MariaDB and RabbitMQ demos), projecting the bucket's keys and the seaweedfs CA into the Secrets the strategy references and honouring an S3_ENDPOINT override so the backup Pod targets the in-cluster seaweedfs. Add a redis-2-backup-roundtrip Chainsaw test that drives run-all.sh in tenant-root and asserts the sentinel round-trips through object storage in place and into a copy. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
Andrey Kolkov
|
b0eb104276 |
feat(backups): add Redis backup/restore strategy driver
The spotahome RedisFailover operator ships no backup CRD, so add a dedicated strategy.backups.cozystack.io/Redis driver that runs a one-shot Job: it dumps an RDB snapshot from the current master (discovered via the operator's sentinel Service) to the system bucket over S3, and on restore replays it (RDB -> RESP via redis-cli --pipe) into the target's master. Structurally a specialisation of the Job/Altinity script-Job pattern with a Redis applicationRef gate and an app-readiness precondition. Backup parameters round-trip through the Backup's DriverMetadata (no underlyingResources snapshot); restore drives from .Backup.ApplicationRef so a to-copy restore reads the source's object-storage prefix while writing into a differently-named target. Both in-place and to-copy are supported; PITR is not (RDB is a discrete snapshot). - api: Redis/RedisList strategy type + generated CRD and deepcopy - driver: reconcileRedis / reconcileRedisRestore + pure helpers, wired into both dispatch switches, supportedBackupStrategyKinds and the no-op restore cleanup case - default: gated cozy-default-redis strategy (system-bucket coords via cozy-backups-creds env secretKeyRef, https:// scheme + optional CA) and the Redis route in the cozy-default BackupClass - image: images/redis-backup bundling redis-cli + curl + an RDB->RESP converter, wired via backupStrategyController.redisBackupClientImage - tests: driver unit tests (gate, precondition, render, param round-trip, both reconcile paths) and the helm rendering-gate coverage - example: examples/backups/redis honest marker round-trip for both restore flows Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-By: LLM |
||
|
Myasnikov Daniil
|
186547d835 |
test(e2e): run the merge-gating lanes on Talos containers
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos containers instead of QEMU guests, so a tenant worker sits at L2 rather than L3. Measured in CI on commits where both substrates ran the same tree: whole-job wall clock down 24-49 minutes, essentially all of it in Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to 466-809s and stop depending on the runner's vgif flag. The node-join soft-red gate goes with the substrate, which is what it was for. It existed because nested virtualisation on the shared runners degraded far enough that a worker registering in minutes elsewhere could miss any deadline this test could afford; #3513 established that the discriminator was the host's kvm_amd vgif flag, and removing a nesting level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from all four workflows, and a missed deadline is an ordinary red again. The ::warning annotation survives, because the reason a run went red should be legible without opening the log, and so does the 124-versus-everything -else test, because "the workers were slow" and "the wait never ran" send a reader to different places. Done by swapping the substrate inside the existing e2e jobs rather than by promoting a separate container job. Those jobs carry more than a substrate -- the GitHub App token, the report-overrun guard, the images list, the SSH breakpoint -- and promoting a job that never had them would have dropped four features without saying so. Three properties of container mode drive the design, and each fails quietly rather than loudly. machine.kernel.modules is a silent no-op -- kernel_module_spec.go returns early on ModeContainer with no error and no event -- so the host loads openvswitch and zfs before the nodes start and the workflow step asserts both rather than trusting them; a missing module surfaces an hour later looking like a CNI or a storage regression. Docker gives a privileged container its own tmpfs /dev seeded once at creation, so /dev/kvm works but ZFS zvols created later never appear and linstor-csi fails ControllerPublishVolume; the compose file binds the host devtmpfs, which is also why talosctl cluster create docker can never carry LINSTOR. And there is no ARP VIP, so 192.168.123.11 stands in for the QEMU lane's .10. The QEMU machinery stays in the tree and keeps running: nightly.yaml and e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated StorageClass, its Immediate binding mode, the tenant StorageClass propagation that only applies to remotely-accessible classes, and the Cozystack Talos node image with its extensions all keep nightly and release-candidate coverage. None of them is exercised per pull request any more. That is the trade. Nothing on the PR path builds or downloads a nocloud disk now, so build-talos drops the talos-nocloud step and the talos-image artifact. build-talos itself stays: finalize consumes its digest fragment. One property this does not fix, recorded beside the runner class because it will be read off a red run eventually: a container node reports the HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters that with kubelet systemReserved. That corrects the scheduler and not what a pod reads from /proc, so a workload sizing itself from /proc/cpuinfo or /proc/meminfo is configured for the whole runner. Also carried here because the same files carry them: the ghcr.io pull-through mirror removal (see #4007 -- whichever lands first makes the other a no-op), the authoritative HelmRelease readiness gate, the backup-credential preflight, the prepull guard against an image-less container fragment, and folding the two standalone OIDC suites into the latest-version tenant cluster. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey Kolkov
|
e94fcf8219 |
feat(backups): RabbitMQ definitions backup via a dedicated strategy driver (#3990)
## What this PR does Adds backup/restore support for the managed `RabbitMQ` application. RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no consistent online export of message payloads, so this backs up broker **definitions** (vhosts, users, permissions, queues, exchanges, bindings, policies, parameters) over the management HTTP API (`GET`/`POST /api/definitions`) and stores them in the system bucket. Message payloads and point-in-time recovery are intentionally out of scope; a full-data backup would be a Velero volume snapshot, which is a separate strategy. It is delivered as a **first-class `strategy.backups.cozystack.io/Rabbitmq` driver**, following the Altinity pattern (a dedicated driver that renders a one-shot Job for an HTTP-API engine): - `reconcileRabbitmq` / `reconcileRabbitmqRestore` render the strategy's `PodTemplateSpec` — `curl` `GET`/`POST /api/definitions`, pushing/pulling the JSON to S3 with `curl --aws-sigv4` — as a `batch/v1.Job`, covering both restore flows: in-place (into the source broker) and to-copy (into a separate, differently-named RabbitMQ in the same namespace, authenticating with the target's own default-user credentials and reading the source's object key). - A **Go-read precondition** gates each run on the `RabbitmqCluster` reporting `AllReplicasReady` (read through a small `rabbitmqtypes` partial type), so a not-yet-ready broker surfaces a precise `Ready=False` wait on the BackupJob/RestoreJob rather than a Job that starts and fails against an unreachable management API. - The produced `Backup` records the stored object's location on `status.artifact.uri`, rendered from the strategy's `spec.artifactURITemplate`. The `Backup` CRD has no status subresource, so this persists on Create, mirroring the Etcd driver. Dispatch is wired in both controllers (plus `supportedBackupStrategyKinds`), `rabbitmqtypes` is registered in the scheme, and RBAC grants read on `rabbitmqclusters.rabbitmq.com` (the Job itself is already covered by the existing `batch/jobs` grant). It ships `cozy-default-rabbitmq` (a `$bucketName`-gated `Rabbitmq` strategy) and routes `apps.cozystack.io/RabbitMQ` to it in the `cozy-default` BackupClass, which also enrols it in the `DefaultObjectsGate`. The RabbitMQ application chart is left untouched — it had no prior backup wiring, and the strategy carries the S3 coordinates itself. Unit tests cover the driver's pure helpers (applicationRef validation, cluster-name derivation, the readiness-message builder, artifact-URI rendering, parameter round-trip) and the reconcile precondition gate (defers with `Ready=False` and no Job until the cluster is ready, then proceeds). A runnable example under `examples/backups/rabbitmq/` and an honest-roundtrip Chainsaw e2e under `hack/e2e-chainsaw/rabbitmq/` seed a sentinel vhost, back it up, drop it, and assert it round-trips through object storage on both restore flows. The Job pod reuses the existing `platform-migrations` image (curl + ca-certificates) under a new `backupStrategyController.rabbitmqBackupClientImage` key, stamped in lockstep with `chBackupClientImage` in `packages/core/platform/Makefile` so the repository stays pinned at a single digest (`hack/image-pin-consistency.bats`). A dedicated slim image built by this package is a follow-up. **Validation:** `go build`/`go vet`/`gofmt`/`go test ./internal/backupcontroller/... ./internal/template/...`, deepcopy + Rabbitmq CRD generation, `helm template` proving the CRD-ordering gate both on and off, `helm lint`, and `helm unittest` (18/18). The backup + in-place + to-copy roundtrip was verified end-to-end on a dev cluster (a real management-API export and a real S3 `--aws-sigv4` upload/download against SeaweedFS), so the driver's core contract is exercised on a real cluster; the in-CI Chainsaw run is the remaining automated evidence. The object-cleanup, terminating-namespace teardown and readiness-deadline paths added during review are unit-tested (including `Reconcile`-level finalizer retention and the terminating-namespace release) but not yet cluster-exercised. (Note found while testing: RabbitMQ 4.2 is OOMKilled on the `nano` preset, so the example/e2e use `c1.small`.) ### Screenshots Not applicable — no UI changes. ### Downstream repositories Walked the trigger map against the diff. This is an additive backup strategy plus a new strategy CRD; no downstream repository is mechanically forced by the change. A documentation page for RabbitMQ backup on [cozystack/website](https://github.com/cozystack/website) would be a reasonable follow-up, but it is not required by this diff — leaving the boxes unticked for a human to decide rather than ticking "no downstream" to clear the checklist. ### Release note ```release-note feat(backups): add RabbitMQ definitions backup/restore via a dedicated strategy.backups.cozystack.io/Rabbitmq driver; it gates each run on the RabbitmqCluster being ready and records the stored object on Backup.status.artifact.uri. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added RabbitMQ backup and restore support for definitions, including vhosts, users, queues, exchanges, bindings, policies, and related settings. * Added in-place and restore-to-copy workflows with readiness checks. * Added scheduled and on-demand backup examples. * Integrated RabbitMQ backups with the default backup configuration. * Backups store definitions only; message payloads and point-in-time recovery are not included. * **Documentation** * Added a complete RabbitMQ backup and restore demonstration guide. * **Tests** * Added end-to-end coverage for readiness and backup/restore round trips. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
6eba4d9fa7 |
test(backups): use .svc FQDN for rabbitmq backup e2e S3 endpoint
The reworked rabbitmq backup roundtrip reached the upload step and failed TLS verification: `curl (60) no alternative certificate subject name matches target hostname 'seaweedfs-s3.tenant-root'`. The demo Bucket, in-cluster endpoint resolution and the copied SeaweedFS CA all worked; the remaining gap is the hostname. The seaweedfs filer serving cert's SANs are `*.tenant-root.svc` and `*.tenant-root.svc.cluster.local`, which the 2-label `seaweedfs-s3.tenant-root` matches neither; the 3-label `.svc` FQDN does (and still resolves via search-domain expansion). Point S3_ENDPOINT at `https://seaweedfs-s3.tenant-root.svc:8333`. Assisted-By: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
329ef9cf3f |
test(backups): make rabbitmq backup e2e self-contained via demo bucket
The rabbitmq backup roundtrip was the only backup e2e routing through the platform cozy-default system bucket, whose advertised endpoint in the CI sandbox is the unroutable s3.example.org placeholder, so the backup Job failed with "Could not resolve host: s3.example.org". Mirror the clickhouse/mariadb backup examples: provision a demo Bucket, project its access keys into rabbitmq-backup-creds and copy the seaweedfs CA into rabbitmq-backup-ca, and render a self-contained Rabbitmq strategy + rabbitmq-default BackupClass whose curl uploads to the in-cluster seaweedfs endpoint (S3_ENDPOINT override, CURL_CA_BUNDLE for the self-signed cert). The in-Pod script is kept identical to the shipped cozy-default-rabbitmq strategy, which is left unchanged; the demo only reuses its client image. Assisted-By: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
ba32df69c0 |
feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.
Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):
- reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
--aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
- A Go-read precondition gates each run on the RabbitmqCluster reporting
AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
precise Ready=False wait instead of a Job that starts and fails.
- The produced Backup records the object location on status.artifact.uri
(rendered from spec.artifactURITemplate; the Backup CRD has no status
subresource, so it persists on Create, as in the Etcd driver).
Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.
Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.
The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey KolkovandClaude Opus 4.8
|
5bcdb8cfb8 |
fix(postgres): let in-place restore re-bootstrap instead of wedging
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.
1. Archive-vs-recovery serverName collision. The driver set the restored
cluster's WAL-archive serverName equal to the recovery source
(bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
chart pinned the archiver serverName to the release name. In place those
coincide, so the cluster archived onto the source's own WAL prefix and
barman-cloud-check-wal-archive refused to start ("Expected empty
archive"). Fixed by giving the restored cluster a fresh
bootstrap.newServerName, keyed by the RestoreJob UID (unique per
restore, stable across reconciles); the chart archives there while
externalClusters still recovers from bootstrap.serverName. This is what
the manual CNPG strategies hit (serverName defaults to the cluster
name); the cozy-default flow dodged it (serverName is <ns>-<app>).
2. useSystemBucket guard. The restore populates explicit S3 coordinates
(destinationPath / s3CredentialsSecret / endpointCA) but left
backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
incompatible with useSystemBucket" guard failed the HelmRelease upgrade
and the cluster never re-rendered. This is what the cozy-default
(system-bucket) flow hits on an in-place restore. Fixed by clearing
useSystemBucket in the restore patch, since the app is now on the
explicit-creds path. The driver's app view (postgresapp.Backup) gains
the field, without omitempty so a false survives the merge patch.
Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Aleksei Sviridkin
|
e441edc2aa |
test(e2e): stop the mongodb backup suite before the restore leg
A to-copy restore replays the source cluster's system users into the target, so the target's own generated credentials stop authenticating and the harness cannot read the sentinel back out of the restored copy. The suite has never passed for that reason, holding the in-tree e2e run red on every branch that selects it. Drive run-all.sh with SKIP_RESTORE=1 so the suite still proves the backup half end to end, and pin the skip by asserting the skip branch's own log line: exit zero alone is also what a completed round-trip returns, so without that assertion the step could not tell a skipped flow from a full one, nor notice SKIP_RESTORE failing to reach the harness. The to-copy manifest and the example README both stated that CI covers the restore, which stops being true here, so both now say where coverage ends. The asserted log line is annotated at its source, since rewording it would red the suite with a bare match failure. Coverage returns by dropping SKIP_RESTORE together with the assertion on that line, once the credential handling behind the restore is fixed. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Andrey Kolkov
|
b97c443a16 |
fix(backups): harden MongoDB restore auth + PITR option handling from review
Address the two blocking findings in PR #3562: - Restore into a differently-named target now authenticates with the TARGET cluster's own S3 credentials instead of the source's. The backupSource carried the source release's -s3-creds Secret, which is deleted together with the source app — the disaster-recovery case backups exist for — leaving a to-copy restore unable to reach S3. The restore replays into the target (gated on backup.enabled just above), whose own -s3-creds Secret is present and grants access to the same bucket; only the credential reference is swapped, bucket/prefix/endpoint still come from the source archive. Falls back to the source reference when the target declares no discoverable s3 credentials (in-place restore resolves to the same Secret, so it is a no-op there). - A non-string recoveryTime (spec.options is free-form runtime.RawExtension, so {"recoveryTime": 20260805} passes admission) no longer silently degrades a PITR request to a plain snapshot restore reported Succeeded. When the typed decode fails and the blob carries a recoveryTime key, the RestoreJob now fails terminally — the same guarantee pitrSpec already makes for a malformed RFC3339 string. Minor findings also addressed: - BackupJob tolerates a not-yet-present MongoDB app CR with the same bounded grace the psmdb-CR existence gate uses, instead of failing terminally on a GitOps single-commit ordering race. - examples: the to-copy target writes into its OWN S3 prefix (mongodb-target/) rather than the source's, so the two clusters' oplog and backup metadata no longer interleave under shared keys (both run replset rs0). - examples: wait_psmdb_ready no longer treats a transient state=error as terminal — the cluster state is recomputed each reconcile and can blip through error during replset init before converging to ready. Unit tests cover the target-credential selector (preferred/default/sole storage) and the load-bearing recoveryTime-key detection. Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
0d4a3e3134 |
refactor(backups): make MongoDB PITR option uniform with CNPG + warn on unknown restore options
PITR: replace the bespoke spec.options.pitr{type,date} blob with a single
spec.options.recoveryTime RFC3339 string, matching the CNPG strategy's option
exactly (CNPGRestoreOptions.RecoveryTime). The driver converts recoveryTime to
the psmdb Restore's pitr{type:date,date} in the operator's "YYYY-MM-DD HH:MM:SS"
UTC format. An empty recoveryTime restores the snapshot as-taken (documented
difference from CNPG, whose empty value means latest WAL — a psmdb logical
backup is already consistent). A non-RFC3339 recoveryTime fails terminally.
Unknown options: parseMongoDBRestoreOptions now also returns the set of
spec.options keys the driver does not recognise; reconcileMongoDBRestore emits a
UnknownRestoreOption Warning event per unknown key (and logs it) instead of
silently ignoring it — so a "recoverytime" typo no longer silently degrades a
PITR restore to a snapshot restore. (BackupJob carries no spec.options blob, so
there is no backup-path equivalent; the only backup-time input is the free-form
BackupClass parameters used as template vars.)
Docs + examples updated to recoveryTime; tests cover the RFC3339→UTC mapping,
empty/invalid recoveryTime, and unknown-key reporting.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey KolkovandClaude Opus 4.8
|
dff56aa998 |
feat(backups): add a native MongoDB (psmdb) backup-strategy driver
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so `apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and ClickHouse native drivers — logical backups, restore-to-differently-named instances, and point-in-time recovery through the unified BackupJob/RestoreJob/BackupClass/Plan interface (closes #3545). The driver orchestrates the operator's PerconaServerMongoDBBackup / PerconaServerMongoDBRestore CRDs: - BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls status.state, and persists the S3 destination + storage descriptor (by-reference, no credentials) onto the Cozystack Backup artifact. - RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back from the source backup status, snapshot fallback) so the same artifact restores in-place or into a differently-named instance; PITR via spec.options.pitr (date/latest). Because psmdb resolves the backup storage by name against the cluster's spec.backup.storages, the driver names the chart's storage (s3-storage) via the cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear Ready=False precondition when a target has backups disabled. Wiring: - api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy. - internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp partial-schema mirrors; dispatch, supported-kinds and scheme registration. - backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb Strategy CR, and psmdb.percona.com RBAC. - apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no CA-bundle field) so backups can target Cozystack's self-signed seaweedfs. Tests + e2e: - unit tests for the driver (validation, ensure-idempotency, precondition, backupSource resolution, snapshot round-trip, PITR option parsing). - examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that drives run-all.sh end-to-end (sentinel document round-trips through S3 into a restored copy while the source stays untouched), per docs/agents/e2e-testing.md. - docs/operations/backup-classes.md updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
1b1a6b2f1d |
fix(clickhouse): address review — trim template docs, harden backup demo CA secret
Review feedback on #3385: - Move the endpointCA rationale out of the Helm template (lllamnyp): the SSL_CERT_DIR and CA-volume blocks were multi-paragraph essays. Trim each to a few lines stating only the non-obvious decision (SSL_CERT_DIR vs AWS_CA_BUNDLE; the deliberate non-optional mount) and point at the values.yaml endpointCA field for the full description. - wait_deleted (00-helpers.sh): poll with 'get --ignore-not-found' and treat only an empty result as deletion, so an RBAC/API-server/transport failure is no longer read as success and cleanup can't settle on a false positive. - Backup demo CA Secret: scope the default name to CLICKHOUSE_NAME, stamp an ownership label on the copy, refuse to overwrite a Secret the demo doesn't own, and delete on cleanup only when that label is present — so a foreign Secret sharing the name is never clobbered or removed. - README: document CH_BACKUP_CA_SECRET and the release-scoped, demo-owned CA Secret; spell out the admin-copy path (export CH_BACKUP_CA_SECRET before steps 04 and 07). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
9379729b31 |
fix(clickhouse): address review notes on endpointCA and the CA copy
Review notes from IvanHunters on #3385 (LGTM with non-blocking findings). 1. The backup-s3-ca volume carries no `optional: true` while the adjacent useSystemBucket credential refs deliberately do, and the asymmetry was undocumented — an operator naming a Secret that does not exist gets the whole ClickHouse Pod stuck in ContainerCreating, not just the sidecar. Kept the fail-closed behaviour and wrote down why the two cases differ: credentials are projected by the platform after install, so a strict ref would wedge every fresh release until the first BackupJob, whereas this CA is named by the tenant, must already exist, and is trust material — optional would mount an empty dir, the Pod would look healthy, and the first backup would fail deep inside clickhouse-backup with an x509 error naming neither the Secret nor the chart. The comment also states the blast radius plainly. 2. The CA probe swallowed stderr, so an RBAC denial on the cross-namespace read was reported as "CA not found" followed by advice (set S3_CA_SECRET) that cannot fix a permissions problem. It now separates present / absent / not-allowed-to-look and says so, and the README no longer labels step 03 simply "tenant": copying the CA reads a Secret in tenant-root, which is a platform capability. Added a section spelling out the three ways to satisfy that (admin credentials, a one-off admin copy, or skipping the CA entirely on a publicly-trusted endpoint). 3. Caveat, and a usability regression this branch introduced: on a cluster with a publicly-trusted endpoint and no seaweedfs, step 03 hard-exited on a default the user never chose. It now distinguishes an explicitly named Secret (still an error when missing — a typo should not be guessed past) from the unset default, where it warns and continues with backup.endpointCA unset. Verified all four paths against a fake kubectl: Forbidden, absent+default, absent+explicit, and present. 4. Caveat: endpointCA reaches the clickhouse-backup sidecar only. The legacy `schedule` CronJob writes to S3 through restic and does not consume it, so a private-CA endpoint combined with `schedule` still fails TLS. Documented on the field itself, so it lands in values.schema.json, the README and the cozyrds openAPISchema. The PR body's claim about select-e2e.sh/select-install.sh mappings was stale after the restructure that removed them; corrected there rather than in code. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
0c14d6cf43 |
test(e2e): sharpen round-trip diagnostics and make teardown settle
Recommendations from the third review pass (which returned LGTM; none of these blocked, all three are cheap and make the failure paths honest). 1. The round-trip's catch named nothing. An unnamed `describe HelmRelease` in tenant-root dumps every release on the root tenant of a live cluster and buries the two that matter, so name them (mariadb-2-backup-roundtrip names its release for the same reason). Raised podLogs tail 100 -> 200, since the sidecar logs every action and an S3 error sits well back from the tail by the time a step fails, and added the Keeper Pods: chkeeper.yaml labels them `app: <release>-keeper`, so the existing app.kubernetes.io/instance selectors missed them entirely — and a source ClickHouse that never reaches a ready replica is usually Keeper failing to form a quorum, which explains itself only there. 2. SSL_CERT_DIR is now `/etc/ssl/cozy-s3-ca:/etc/ssl/certs`. The single-directory form works — Go reads the default bundle FILE regardless, which is what the earlier measurement showed — but that left the public-CA half of the contract resting on a file this image happens to ship. Naming the system directory costs nothing and survives a base-image bump that moves the bundle; verified against public AWS S3 that both forms still verify (403 auth, no x509). The mount path stays the bare directory, and the unit test now pins both halves separately — it caught the over-broad edit that had briefly made mountPath a colon list. 3. cleanup.sh returned while the uninstalls it triggered were still running, so the next suite (etcd) could start with helm-controller still draining this one, and the pre-clean could hand step 04 an application whose predecessor was still going away. It now waits for both ClickHouse releases and the Bucket to actually disappear, and — since the script deliberately runs without `set -e` so one absent resource cannot abandon the rest — records those failures and exits non-zero, rather than reporting a teardown that did not settle as success. wait_deleted polls `get` instead of using `kubectl wait --for=delete`, which errors on an already-absent resource: the normal case here, and indistinguishable from a real failure. Sized the two cleanup step budgets to 12m for the same reason the run step is 60m: cleanup.sh's waits now sum to 540s, which the previous 8m did not clear. The Keeper-sized-from-the-server-resources bug the review found outside this diff is filed as #3487. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
82aa967674 |
test(e2e): correct the round-trip budget and its stale comments
Second review pass on this branch.
1. The op timeout equalled the inner sum instead of exceeding it: the comment
enumerated 3900s of inner waits and then set 65m, and 3900s IS 65m — so there
was zero headroom for the applies and SQL calls between those waits, and the
claim of "room" was false. Sized the inner budgets against what CI actually
takes instead (the bucket suite completes in ~37s and a backup-enabled
ClickHouse plus Keeper reaches Ready in ~98s, measured on run 30353894332):
bucket claim/access 300->180 and the three job waits 600->480, summing to
3300s, under a 60m op timeout. Shrinking the inner budgets rather than
raising the ceiling also keeps this suite closer to its postgres and mariadb
siblings on a 180-minute job cap.
2. The strategy Pod's securityContext comment still justified root with "Pod
runs apk add at startup", which the previous commit made conditional — the
default path now runs the platform's pre-baked image and installs nothing.
Scoped the note to the alpine fallback and said explicitly that the field set
matches cozy-default-altinity, runAsNonRoot deliberately unset in both.
3. clickhouse-backup.yaml still referenced clickhouse-backup-{1-contracts,
2-roundtrip}: the file moved unchanged, so the Test rename never reached its
own header and a reader following either name found nothing.
4. wait_sts_ready's timeout diagnostic could not print anything. It selected
clickhouse.altinity.com/chi=${name#chi-}, which yields
clickhouse-<app>-clickhouse-0-0 while the label carries the CHI name
(clickhouse-<app>), and 2>/dev/null then swallowed kubectl's "No resources
found". Ask for the StatefulSet's own first replica by name instead and let
both streams through; verified it now surfaces the Pod events that explain
why the replica never came up.
Also dropped the clickhouse bats case added in the previous commit: it asserted
what the generic examples/backups/<app> -> <app> mapping already guarantees, so
it did not lock the invariant its comment claimed. That invariant is now
recorded where it belongs — in the mapping's own comment in select-e2e.sh.
The dedicated ch-backup-client image this example now inherits is tracked in
#3420; noted there that the command: override is load-bearing in two files.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey KolkovandClaude Opus 4.8
|
c1fd7db51d |
test(e2e): fold the clickhouse backup Tests into the app's own suite
Review findings on the previous two commits. 1. TIA no longer selected the suite that runs the harness. select-e2e.sh maps examples/backups/<app>/** to suite <app>, so with the round-trip living in a separate clickhouse-backup/ dir an edit to the harness selected `clickhouse` — which did not execute those scripts — while the suite that does was left out. Fixed the way postgres/mariadb/etcd already do it: both Tests move into hack/e2e-chainsaw/clickhouse/ as clickhouse-1-backup-contracts and clickhouse-2-backup-roundtrip. That also deletes every special case the earlier commits added (src_to_suites, suite_to_source, and the bats case that guarded them — both scripts are now byte-identical to main), and the suite list in hack/e2e-chainsaw/README.md stays correct because no dir is added. The bats case is replaced by one asserting the invariant that actually matters: an examples/backups/clickhouse edit selects the suite that runs it. 2. The docs claimed a fail-fast the harness did not have, and the budget was sized as if it did. wait_for_field polled to the full timeout with no terminal-value escape, and the HelmRelease/StatefulSet waits were bare `kubectl wait` with no existence backstop and no Stalled fail-fast. Ported wait_for_field's fail_value parameter and wait_hr_ready from the postgres helpers, added wait_sts_ready, and routed all eight waits through them. Verified against a fake kubectl: a Failed BackupJob now bails in 3s instead of 600s, a Stalled HelmRelease immediately. The inner budgets sum to 3900s, which the old 30m op timeout did not cover — a hang would have been SIGKILLed with no diagnostics, the failure mode .chainsaw.yaml calls strictly worse — so the timeout is now 65m with the arithmetic written out. 3. The strategy Pod ran `apk add --no-cache curl jq` on every backup and restore. Harmless while nothing in CI ran it; a per-PR dependency on the Alpine CDN once it does, and broken outright air-gapped — which is exactly why the shipped strategy bakes curl+jq into its image. Step 01 now reads that image off cozy-default-altinity and drops the apk add, falling back to alpine only when the platform strategy is absent. Also bounded the two polling curls with --max-time; a hung connection previously stalled the loop until the outer timeout. 4. 91-scenario-user-backup.md still documented the old tenant-test default. 5. .chainsaw.yaml claimed all app tests run in tenant-test, which the postgres and mariadb round-trips already contradicted before this branch. The round-trip header now also states its scope plainly: it exercises the example's strategy, not the shipped cozy-default-altinity. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
dac34b7469 |
test(e2e): actually run the clickhouse backup round-trip in CI
The round-trip Test was skipping: it exited 0 behind CLICKHOUSE_E2E_S3_ROUNDTRIP, so "PASS (1.01s)" meant the gate fired, not that backup and restore worked. Only the contracts Test ran, which proves the chart still exposes the sidecar and its auth Secret — not that a byte ever reaches S3 and comes back. The gate existed because the sidecar could not verify the in-cluster SeaweedFS certificate. backup.endpointCA closes that, so adopt the shape mariadb and postgres already use to run their round-trips un-gated: - examples: resolve the S3 endpoint from S3_ENDPOINT when set (BucketInfo advertises the external ingress URL, an unroutable placeholder in CI), copy the seaweedfs CA (auto-discovered from its cert-manager Certificate when the fullname default is absent) into a per-release Secret, and reference it from backup.endpointCA on both the source and the to-copy target. NAMESPACE now defaults to tenant-root, matching the other backup demos. - suite: drop the gate, run in tenant-root — an isolated tenant's Cilium egress allowlist blocks its Pods from reaching tenant-root's seaweedfs — and pre-clean with cleanup.sh, because tenant-root is shared and a leftover Succeeded BackupJob would satisfy the harness's wait without a backup running. cleanup.sh also drops the copied CA Secret; the platform's own is never touched. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
8aeedde088 |
feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383)
## What this PR does Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported, tested capability for the new `backups.cozystack.io` API. Fixes #2774. Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the `recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime` plumbing; #3313 has since merged and this branch is rebased on `main`. This PR adds the reliability + verification layer #2774 asked for. ### Changes - **Precise failure reason for an unreachable target.** The restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the sole authority for "this restore is stuck" — the client waits the full window either way. What changes is the *diagnosis*: once the deadline has elapsed, if `recoveryTime` was set and the recovery Cluster never went healthy, the CNPG driver reads the `full-recovery` pod logs and — when several distinct attempts exited with PostgreSQL's `FATAL: recovery ended before configured recovery target was reached` — fails the RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason `RecoveryTargetUnreachable`) and an actionable message, instead of a bare generic timeout. Timing is unchanged; only the reason/message differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.) Design points: - **We deliberately do NOT trip early off the FATAL.** A valid near-now target emits that same FATAL transiently — repeatedly, for many attempts, on a slow archiver — before it converges, so any earlier trip would wrongly reject a recoverable restore (observed twice in CI). The FATAL is used only to *explain* a failure the deadline has already declared. - **Health wins unconditionally** — the deadline/classification check precedes the healthy check, but a recovery that converges within the window reaches success and is never classified as failed (a live dev7 restore did exactly this). - **Counting distinct FATAL-ing attempts** (not pod failures) tolerates a one-off archive-lag miss on a near-now target and ignores transient recovery-pod crashes (a node blip / brief API-server unreachability never emits that FATAL). - Adds the `pods/log` RBAC grant the log read needs (distinct subresource from `pods`); a `Forbidden` is logged loudly and surfaced in the failure message rather than silently disabling the classification. - **e2e** (`examples/backups/postgres/run-all.sh`, wired into the Chainsaw `postgres-2-backup-roundtrip` suite): - Step 45 — write `before`, capture server timestamp (µs precision), write `after`, restore to that timestamp, assert `before` present and `after` absent. - Step 46 — a `recoveryTime` an hour ahead must fail with reason `RecoveryTargetUnreachable` once the deadline elapses (asserts the *reason*, so it proves the classification fired, not just that a timeout occurred). - Chainsaw budget raised 25m → 40m to cover three sequential restores + the post-deadline classification. - **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a `docs/operations/backup-classes.md` PITR section (recoverable window, discovering earliest/latest restorable time from the backup catalog since `firstRecoverabilityPoint` is unreliable under the plugin, archive-lag tolerance, idempotency under GitOps). ### Verification - `go build` / `go vet` clean; unit tests pass (`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`, `recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level success-path test asserting `Succeeded` + `RecoveryConverged=True`, and — added per review — `TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`, a reconcile-level test driving the deadline branch through all three sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL absent → generic timeout, `Forbidden` log read → generic timeout + missing-`pods/log`-RBAC note. It fails if the call-site reason is reverted to the generic one). - Reviewed via adversarial branch-review; fixes include a critical missing-RBAC dead-on-arrival, a classification false-positive on transient recovery-pod crashes, and a sub-second e2e flake. - **dev7**: backup + WAL archiving + recovery convergence verified live on the #3313 build. The load-bearing assumption of the classification — CNPG **accumulates** failed `full-recovery` pods rather than reusing one — was confirmed (observed 7 simultaneous failed recovery pods). A fully-green live step-46 run was blocked by intermittent COSI bucket-provisioning flakiness on dev7; the negative path is covered by the CI Chainsaw e2e. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The only user-facing surface is the new `docs/operations/backup-classes.md` PITR section; the website keeps a hand-mirrored copy per docs version. - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: PITR docs mirror warranted (not yet opened; same pending mirror as #3313's backup-classes changes) - [x] No other downstream repository is affected by this change ```release-note feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added PostgreSQL point-in-time recovery (PITR) support via RFC3339 `recoveryTime`. * Introduced a PITR RestoreJob example, plus helper variables and an unreachable-target scenario. * **Bug Fixes** * Improved restore deadline handling with deterministic, log-based fail-fast classification for unreachable recovery targets, including clearer conditions. * **Documentation** * Added detailed PITR operational documentation (recovery windows, failure behavior, and GitOps/idempotent behavior). * Expanded README and example comments for PITR verification and failure expectations. * **Tests** * Extended unit tests and end-to-end chainsaw workflow to cover PITR success and unreachable failures. * **Chores** * Updated RBAC to allow pod log reads; refined cleanup and run scripts for additional RestoreJob artifacts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey KolkovandClaude Opus 4.8
|
2829f7a300 |
docs(mariadb): address review notes on backup example
- 00-bucket.yaml: correct the Secret-name comment to include the "bucket-" release prefix (bucket-<bucket>-<user>), matching the README and 00-helpers.sh. - README: note that S3_CA_NAMESPACE defaults to tenant-root independently of NAMESPACE, so a non-root tenant run must override it (and S3_CA_SECRET) too. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
d1f2822018 |
test(mariadb): add backup/restore round-trip e2e (chainsaw) via example harness
Reworks the never-merged backup-mariadb-e2e branch onto the modern Chainsaw approach (docs/agents/e2e-testing.md). The bats port that branch carried (hack/e2e-apps/backup-mariadb.bats) is dropped — the whole hack/e2e-apps/*.bats app suite has been replaced by hack/e2e-chainsaw/, and the mariadb strategy controller + example manifests it depended on have since landed on main independently. Only the e2e coverage was still missing. Convert examples/backups/mariadb/ from static numbered YAML into the same executable harness the other backup demos use (00-helpers.sh, run-all.sh, cleanup.sh), keeping the numbered manifests as the human-readable flow that run-all.sh drives — so the documented flow and the automated test can never drift. run-all.sh provisions a Bucket, derives the two per-app Secrets the strategy references from BucketInfo + the seaweedfs CA, deploys a source MariaDB with a sentinel row, fires an ad-hoc BackupJob, restores to a copy with a to-copy RestoreJob, and asserts the sentinel round-tripped through S3 into the copy while the source is left untouched. Add a second Chainsaw Test (mariadb-2-backup-roundtrip) to hack/e2e-chainsaw/mariadb/ that drives run-all.sh + cleanup.sh, mirroring postgres-2-backup-roundtrip: runs in tenant-root (the isolated e2e tenant cannot reach tenant-root's seaweedfs across the Cilium egress policy), skipDelete since the harness (not Chainsaw) owns the resources, fail-fast on terminal BackupJob/RestoreJob phases, and scoped catch diagnostics. Cleanup also prunes the operator-side k8s.mariadb.com Backup/Restore CRs the Cozystack jobs own only by label. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
6a1e2d0d06 |
feat(backups): support PostgreSQL point-in-time recovery (PITR)
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested capability for the new backups.cozystack.io API (issue #2774). The recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing already existed on the barman-cloud-plugin branch; this adds the reliability and verification layer the issue asked for: - Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set and the recovery Cluster is not healthy, the driver reads the CNPG full-recovery pod logs and, once several distinct attempts have exited with PostgreSQL's "recovery ended before configured recovery target was reached" FATAL, fails the RestoreJob with conditions RecoveryConverged=False / Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the restore deadline. Health is checked first, so a converged recovery always wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a transient archive-lag miss on a near-now target and ignores transient recovery-pod crashes. Adds the pods/log RBAC grant the log read requires. - e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures the server timestamp, writes an after marker, restores to that timestamp and asserts the before row survived and the after row did not; step 46 submits a recoveryTime an hour ahead and asserts it fails fast with reason RecoveryTargetUnreachable. - Working, uncommented example (45-restorejob-pitr.yaml) and a docs section (docs/operations/backup-classes.md) covering the recoverable window, how to discover the earliest/latest restorable time from the backup catalog, the archive-lag tolerance, and idempotency under GitOps. Refs #2774 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Fable 5
|
b897b53107 |
test(postgres): fix S3 CA secret name in backup roundtrip e2e
The backup roundtrip e2e copied the seaweedfs S3 CA from a Secret named seaweedfs-system-ca-cert in tenant-root, a name captured off a dev cluster. On current main the seaweedfs chart pins fullnameOverride: seaweedfs, so the cert-manager CA Certificate's secret is seaweedfs-ca-cert; the old name is absent and the first CI run of the e2e failed at step 05a with 'secrets "seaweedfs-system-ca-cert" not found'. Default S3_CA_SECRET to seaweedfs-ca-cert and, when that Secret is absent, auto-discover the seaweedfs CA Certificate (the seaweedfs-labelled one with spec.isCA=true) and read its secretName, so a future fullnameOverride change does not silently break the copy again. Fail with a clear message when neither resolves. README and chainsaw comment updated to match. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Fable 5
|
50fb3c87c2 |
test(postgres): run the backup round-trip e2e in CI under TIA
Ungate postgres-2-backup-roundtrip: instead of an opt-in env flag it now runs whenever TIA selects the postgres suite (postgres changes, and every release cut), like the other chainsaw tests. The blocker was network placement, not capability: the S3 client is the barman-cloud sidecar inside the Postgres Pod, and the isolated e2e tenant's Cilium egress allowlist blocks it from tenant-root's seaweedfs — so the Test runs in tenant-root (same-tenant egress) against the in-cluster S3 endpoint, trusting its self-signed cert via the copied seaweedfs-system-ca-cert. This is exactly the recipe validated end to end on a real cluster. Also map examples/backups/<app>/ edits to that app's suite in hack/select-e2e.sh (the round-trip executes those scripts as its harness; previously such edits were silently ignored), with bats coverage for both the mapped and the no-suite case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Fable 5
|
f2f39d8f8c |
test(postgres): materialise restore-target backup Secrets in the example harness
The CNPG strategy template renders <app>-cnpg-backup-creds / -ca against whichever application it drives, so a to-copy restore looks the pair up under the TARGET's name. The harness only created the source's pair; on a cluster with stale target Secrets from an earlier run the restore failed TLS verification against the wrong CA. Materialise the pair for the target too (kubectl apply, so stale content is corrected) and delete both pairs in cleanup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Fable 5
|
67db8f7da7 |
fix(backups): make the barman-cloud serverName stable across every flow
Two defects of the same class — the WAL-archive S3 prefix (serverName) changing or defaulting differently between components: Recovery default regression (review blocker): native CNPG defaulted an absent externalClusters serverName to the entry name (= bootstrap.oldName), but the barman-cloud plugin defaults it to the NEW cluster's name, so a legacy-flow restore with bootstrap.serverName unset would read the wrong S3 prefix (hard failure, or silently restoring another cluster's data). The chart now always emits serverName, defaulting to bootstrap.oldName. Live-cluster serverName flip (found by the gated postgres backup e2e on a real cluster): the chart renders spec.plugins without serverName (effective prefix = cluster name) while a strategy template may name a different one; the driver's ForceOwnership patch then flipped the prefix mid-stream, splitting the WAL archive across two prefixes — the WAL holding the backup's start was lost between them and the restore failed with "WAL not found". The driver now preserves the live Cluster's effective serverName (explicit parameter, or the cluster name when the plugin is attached without one) and only applies the strategy's value to a Cluster with no plugin attached; both charts pin serverName explicitly, and the example strategy no longer sets one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
bdc0b564e2 |
test(postgres): add gated barman-cloud backup/restore e2e round-trip
Port examples/backups/postgres into an automated e2e, following the etcd backup precedent: the numbered example manifests become a runnable harness (00-helpers.sh, run-all.sh, cleanup.sh) that a human reads and the test drives, so the documented flow and the test can never drift. run-all.sh provisions a Bucket, deploys the source Postgres (writing a sentinel row), derives the two Secrets the barman-cloud ObjectStore references from the Bucket's BucketInfo and seaweedfs' CA, submits an ad-hoc BackupJob and waits for Succeeded, then restores to a copy and asserts the sentinel round-tripped through S3 — the in-cluster witness that a plugin backup is both taken and restorable end to end. A second Test document (postgres-2-backup-roundtrip) in the existing hack/e2e-chainsaw/postgres suite drives run-all.sh, gated out of CI behind POSTGRES_E2E_S3_ROUNDTRIP=1 (kind has no reachable, CA-trusted S3 endpoint, and a tenant Pod cannot cross the Cilium egress policy to tenant-root's seaweedfs). Placing it in the postgres suite (rather than a new dir) keeps it triggered by packages/apps/postgres changes under Test Impact Analysis. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Myasnikov DaniilandClaude Fable 5
|
9297e51170 |
Merge origin/main into feat/e2e-chainsaw-example
Reconcile the Chainsaw e2e migration with ~500 commits of main drift: - keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's _lib/ rename with the Chainsaw finally/snapshot deltas re-applied - resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted; their post-branch assertion drift is re-ported into the Chainsaw suites in follow-up commits - drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup, serviceexposure) from _lib/ relocation; ported as Chainsaw suites in follow-up commits - docs/agents/e2e-testing.md: merge main's plan-job TIA wording with the Chainsaw suite phrasing Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Timofei Larkin
|
5f76ac0d76 |
test(etcd): capture backup Job Pod logs, fail fast on terminal phase
The backup round-trip test could wedge for 20 min and then report a bare EtcdSnapshot phase=Started with no root cause: dump_diagnostics dumped the (healthy) EtcdCluster and the operator log, but the snapshot agent runs in its own Job Pod, so the etcd-read / S3-upload error lives only in that Pod's log and in the BackupJob/EtcdSnapshot status. - dump_diagnostics now dumps the whole backup chain: BackupJob/Backup and EtcdSnapshot describe, each snapshot Job's describe + Pod logs (selected by the operator's etcd-operator.cozystack.io/cluster label), and recent namespace events (the primary signal when a hung Pod logs nothing). - wait_for_field gains an optional terminal-failure value; steps 04/05 pass Failed so the BackupJob/RestoreJob waits return immediately on a terminal phase instead of polling to the timeout — which also keeps the snapshot Job Pod within its 600s TTL so its log is still readable at dump time. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Andrey Kolkov
|
cc5d688e15 |
feat(platform): migrate to etcd-operator v1alpha2
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
c82b55c621 |
docs(agents): align E2E guide and references with the Chainsaw migration
Rewrite docs/agents/e2e-testing.md for the Chainsaw suite: declarative
asserts replace the until-get/wait backstops, Chainsaw cleanup replaces
test-level EXIT traps, catch blocks replace the `|| { ...; false; }` idiom,
and the TIA/diagnostics sections describe suites + `make test-chainsaw
CHAINSAW_SUITES`. Move the BATS->Chainsaw bullet out of 'in-flight' since it
is now the standard.
Fix references the migration left stale: hack/e2e-apps/ -> hack/e2e-chainsaw/
in the gemini styleguide and the nats/etcd backup READMEs. The
backup-postgres.bats / backup-foundationdb.bats files cited by the postgres
examples, the postgres README, and the foundationdb controller comment never
existed -- drop the false 'CI e2e drives this' claims and describe those
backup flows as the manual reference flows they are.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Andrey Kolkov
|
f17ead3d26 |
feat(platform): add default backupclass
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Timofei Larkin
|
72c4eb9c08 |
feat(backups): implement Job backup strategy (#1721)
## What Implements the `strategy.backups.cozystack.io/Job` backup strategy — a generic, application-agnostic driver that runs a user-supplied Kubernetes Job to create (and restore) a backup. ## Background This PR began as a draft written against an earlier shape of the backups API. The API has since moved to a `BackupClass`-based dispatch model, which broke the original draft on three counts: - `BackupJob.Spec.StrategyRef` and `BackupJob.Spec.StorageRef` were removed; the strategy and its parameters are now resolved from a `BackupClass` into a `ResolvedBackupConfig`. - Strategy handling moved from standalone per-strategy reconcilers to a dispatch model: `BackupJobReconciler.Reconcile` / `RestoreJobReconciler.Reconcile` resolve the strategy and switch into per-strategy methods (`reconcileVelero`, `reconcileCNPG`, `reconcileAltinity`, …). - The draft's template context (`.Values`, `.Storage`) diverged from the convention the other strategies adopted, and its `.Storage` half targeted a storage API that does not exist. ## What changed - **`reconcileJob` / `reconcileJobRestore`** (`internal/backupcontroller/jobstrategy_controller.go`): replaces the standalone `BackupJobStrategyReconciler` with the two dispatch methods the `BackupJob` / `RestoreJob` reconcilers call. The Job strategy is the generic counterpart of the Altinity driver — same lifecycle (render `PodTemplateSpec` → run a one-shot `batch/v1.Job` → poll its terminal condition → emit a `Backup`), minus the ClickHouse-specific `applicationRef` gate, so it runs for any application. Shared package helpers (`renderJobTemplate`, `jobConditionState`, `getApplicationUnstructured`, status helpers, deterministic Job naming, controller refs) are reused. - **Template context** follows the in-tree convention used by the other drivers: `.Application` (live application object), `.Release.Name` / `.Release.Namespace`, `.Mode` (`backup` / `restore`), `.Parameters` (resolved `BackupClass` parameters), and `.Backup` (restore runs only). Strategy parameters round-trip through the `Backup`'s `driverMetadata` so a later restore re-renders against the same values. - **Restore** re-renders the same template with `.Mode == "restore"`. Cross-namespace restore is unsupported (the RestoreJob target reference is namespace-local), matching the Altinity driver. - **`JobSpec.Template` doc + regenerated CRD**: the field doc previously advertised `.Values`; it now documents the real context, and `strategy.backups.cozystack.io_jobs.yaml` is regenerated to match (the change is limited to that field's description). - **Tests** (`jobstrategy_controller_test.go`): backup-Job creation / args / labels / status, completion → `Backup` artifact with parameter round-trip, Job-failure surfacing, missing-strategy terminal failure, restore into the target namespace, restore completion, and the parameter-prefix unit. A deliberately generic application kind pins that the driver never gates on `applicationRef.kind`. No RBAC changes are required: the controller already holds `strategy.backups.cozystack.io/*`, `batch/jobs`, `apps.cozystack.io/*`, and `backups` permissions. ## Notes for reviewers - Backup and restore share one template, so a strategy author's `PodTemplateSpec` must branch on `.Mode` to do anything restore-specific. This matches the existing convention; worth confirming it's the intended contract for the Job strategy's docs/examples. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Implemented full backup/restore job strategy with deterministic Job creation, idempotent execution, and status/condition updates; added demo example scripts and reusable shell helpers for a NATS JetStream backup/restore workflow. * **Documentation** * Clarified Job template rendering context and listed supported template variables; added README and step-by-step scenario docs for the NATS example. * **Tests** * Added comprehensive unit tests covering job creation, completion/failure handling, parameter extraction, and unmappable-target failure cases. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/1721?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Timofei Larkin
|
727343d259 |
docs(backups): add NATS Job-strategy example
Add examples/backups/nats/, the first worked example of the generic Job
backup strategy. It backs up a NATS JetStream stream that has no dedicated
driver, using a stock natsio/nats-box image: 'nats stream backup' -> tar ->
curl --aws-sigv4 PUT to the COSI/seaweedfs S3 endpoint, and the reverse for
restore. One PodTemplateSpec serves both directions, branching at runtime on
{{ .Mode }} (the engine templates each string leaf independently and cannot
restructure the Pod per mode).
Mirrors the clickhouse example's structure: numbered scripts (bucket -> NATS
src + seeded stream -> strategy -> BackupClass -> BackupJob -> in-place and
to-copy RestoreJob), shared helpers, run-all.sh, README, and scenario docs.
Like every examples/backups/* dir it is a runnable demo, not CI-gated.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
|
||
|
Andrey Kolkov
|
d80f67e799 |
feat(platform): add implementation of backup-strategy controller for etcd
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
