Commit Graph
59 Commits
Author SHA1 Message Date
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
Myasnikov Daniil f47bab7280 test(backups): fail CI on any default Strategy CR rendered without its CRD gate
The CRD gate has now been missed twice: MongoDB shipped without it, and
Kafka, RabbitMQ and Redis followed after it existed. helm-unittest only
checks the templates a case lists, so a new strategy-*-default.yaml
that forgets the gate stays green until an upgrade wedges.

Render the chart with the bucket gate open and no CRD served, and fail
on any strategy.backups.cozystack.io object from any template. Also
check that every rendered Strategy CR gates on the CRD this chart ships
for its kind, because a gate naming the wrong CRD passes the first
check and keeps the CR out of every live render.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-14 13:31:48 +00:00
Myasnikov Daniil 7204992452 fix(backups): gate every default Strategy CR on its CRD
The chart ships its strategy CRDs as ordinary templates, and Helm maps
every document of a release through the RESTMapper before it applies
any. A default Strategy CR gated only on the bucket name therefore
fails Build() with "no matches for kind" whenever the bucket name
resolves before the CRD is served, the release applies nothing, the
CRDs are never created, and every retry renders the same manifest.

That happens on the upgrade that first ships a kind: the Kafka,
Rabbitmq and Redis CRs leave a v1.6 cluster's HelmRelease stuck in
UpgradeFailed on the way to v1.7. It also happens on the first render
of an external-S3 install (provisionBucket=false), where the bucket
name comes from values and every older default CR renders next to its
absent CRD.

MongoDB already gates on crdEstablished for this reason. Give every
default Strategy the same gate. On an installed cluster the CRDs are
established and the render is unchanged; otherwise the first revision
creates the CRDs and skips the CRs, and DefaultObjectsGate, which reads
the cozy-default BackupClass routes, forces the follow-up upgrade that
renders them once the kinds are served.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-14 13:31:45 +00:00
Andrey Kolkov 0c2f131192 feat(backups): back up Valkey with the Redis strategy
Valkey runs on the same spotahome RedisFailover operator as Redis and
speaks the same replication protocol and RDB format, so it needs no
strategy kind of its own. The only difference the backup Job sees is the
chart's resource-name prefix: valkey-<app> instead of redis-<app> for the
sentinel Service and the auth Secret.

The Redis driver now accepts applicationRef kind Valkey as well as Redis
and hands the template .Release.Prefix, derived from the app's kind, which
cozy-default-redis uses for every name it builds. A kind with no known
prefix fails the render instead of producing an empty one. cozy-default
routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must
target an app of the same kind as the backup's source: replaying a Redis
dump into Valkey, or the reverse, is not something the strategy can
promise across engine releases.

The shipped template reads the prefix as `.Release.Prefix | default
"redis-"`: during an upgrade the previous controller, which has no Prefix
in its render context and serves only Redis, renders the new CR, and a
bare reference would print "<no value>" into the names.

The redis:8 image already loads RDB v11, which both shipped Valkey
versions (7.2, 8.1) write.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-10-13 12:34:24 +04:00
Andrey Kolkov 647407545d fix(backupstrategy-controller): run backup clients on a package-owned image
The Altinity and RabbitMQ strategy Pods ran platform-migrations, a 349 MB
image carrying kubectl, helm and etcd-migrate whose ENTRYPOINT is the
migration runner, and whose pins were stamped from packages/core/platform
so every migration change moved the backup clients too.

Build images/backup-http-client (alpine + curl + jq + ca-certificates,
non-root) from this package's Makefile and stamp both client keys from
it. The cross-package stamp in packages/core/platform/Makefile and the
platform-migrations-specific check in hack/image-pin-consistency.bats go
away; the general one-digest-per-repository check still covers the two
keys sharing one image.

Fixes #3420

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-12 09:54:52 +04:00
Aleksei Sviridkin f4bde9e50d fix(backups): delete the original PVC of an in-place VM restore without keepOriginalPVC (#4672)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

An in-place VMInstance restore with `keepOriginalPVC: false` reports
`Succeeded` without restoring the disks, and leaves every disk it
touched unable to come back. `prepareForRestore` deletes each DataVolume
with Orphan propagation and leaves the PVC in place, on the assumption,
stated in the `deleteDataVolume` comment, that Velero overwrites it.
Velero does not overwrite a PVC. Its CSI restore action returns early
for a claim that already exists, so no DataDownload is ever created, and
with `existingResourcePolicy: update` the merge patch it attempts
instead is rejected because a bound claim's spec is immutable. All that
reaches the claim is the labels-only fallback patch, which also drops
the `cdi.kubevirt.io/allowClaimAdoption` annotation the resource
modifier added.

The claim left behind then blocks the disk for good. Orphan propagation
has removed its owner reference to the DataVolume, and it lacks the
`cdi.kubevirt.io/storage.populatedFor` annotation that
kubevirt-velero-plugin writes on the backed-up copy, so when the
HelmRelease resumes, the CDI webhook refuses the DataVolume it creates
with `Destination PVC <ns>/<name> already exists`, on every reconcile.
Deleting the disk through the API leaves the PVC behind, and a new disk
of the same name fails the same way.

Observed on a v1.6.3 cluster, for a VMInstance restore with
`keepOriginalPVC: false` whose disk was attached to the VM. The Velero
log for the restore, trimmed:

```text
level=info msg="Retrieved per-restore resource modifiers from configmap cozy-velero/restore-modifiers-..."
level=warning msg="PVC already exists and ExistingVolumeDataPolicy is not in-place restore. Skip restore this PVC." Action=PVCCSIRestoreItemAction PVC=tenant-datalab/vm-disk-e2edisk
level=info msg="Applying resource modifier patch on tenant-datalab/vm-disk-e2edisk"
level=warning msg="patch attempt failed for /v1, Kind=PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk: PersistentVolumeClaim \"vm-disk-e2edisk\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims"
level=info msg="backup/restore labels successfully updated for PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk"
```

`velero restore describe --details` lists the PVC as `(failed)` and the
Restore is `Completed` with two warnings. Afterwards the claim had its
original creation timestamp and UID, no owner reference, no
`allowClaimAdoption`, no `populatedFor`, and the disk's HelmRelease was
`Ready=False` with the webhook error above. That cluster runs a Velero
build from `main`, whose wording is quoted above; the released v1.17.1
and v1.18.3 return early at the same point with `PVC already exists.
Skip restore this PVC.` (`pkg/restore/actions/csi/pvc_action.go`).

The fix deletes the PVC when `keepOriginalPVC` is false, only once its
DataVolume is confirmed gone so that CDI cannot recreate it, and holds
the Velero Restore until the claim has actually disappeared. A claim
still mounted stays `Terminating` behind `kubernetes.io/pvc-protection`,
and creating the Restore then would hit the same existing-claim path.
Velero then creates the PVC from the backup, the data mover fills it,
and the recreated DataVolume adopts it through `populatedFor`. A failed
deletion now fails the RestoreJob instead of letting it report
`Succeeded` over the old data. Before 2234824fe the DataVolume was
deleted with background propagation, so garbage collection removed the
claim, but nothing waited for it.

Deleting the claim is only safe when Velero can put the disk back, so
before any preparation the reconciler now gets the Velero Backup and
fails the RestoreJob unless it exists and is `Completed`. The Backup
artifact outlives the Velero Backup, which Velero garbage-collects at
its TTL, and without the check a restore of an expired backup would halt
the VM and delete its disks before failing. The check runs for every
Velero restore, whatever `keepOriginalPVC` says. A DataVolume that
cannot be deleted now fails the RestoreJob too, and the preparation
waits for every DataVolume to be gone before it goes further: a
surviving DataVolume makes CDI recreate the claim, which Velero then
skips. Reading the DataVolume back needs `get` on `datavolumes`, which
the backupstrategy-controller ClusterRole now grants next to `delete`.

`TestPrepareForRestore_KeepOriginalPVCFalse_SkipsRename` asserted the
old behaviour, that the original PVC survives, and is replaced by
`TestPrepareForRestore_KeepOriginalPVCFalse_DeletesPVC`.
`TestPrepareForRestore_KeepOriginalPVCFalse_WaitsForTerminatingPVC`
covers the wait. Both fail on `upstream/main` without the controller
change. `TestReconcileVeleroRestore_UnusableVeleroBackupTouchesNothing`
covers an expired Velero Backup and one being deleted,
`TestPrepareForRestore_KeepOriginalPVCFalse_KeepsPVCWhileDataVolumeRemains`
a failed and a pending DataVolume deletion, and
`TestPrepareForRestore_KeepOriginalPVCTrue_FailedRenameKeepsPVC` a
failed rename under `keepOriginalPVC: true`, which must leave the claim
in place. A helm-unittest suite, `tests/restore_rbac_test.yaml`, checks
the new verb. Each test fails with its fix reverted, with one exception:
`TestPrepareForRestore_KeepOriginalPVCTrue_FailedRenameKeepsPVC` still
passes with the Step 5 condition cut to `vmRes != nil`, because a failed
rename holds the restore back before Step 5 is reached. That guard is
pinned by `TestPrepareForRestore_KeepOriginalPVC_CacheLagsBehindWrites`,
which fails with the same cut. `helm unittest
packages/system/backupstrategy-controller` passes. `go test
./internal/backupcontroller/...` passes.

Related: #4601 reports the same Velero mechanism for a standalone
VMDisk, where `prepareForRestore` does nothing at all because
`collectUnderlyingResources` records no disk. #3427 makes a VMDisk
report its own disk; with `keepOriginalPVC: false` that path then runs
into this defect too, and this change covers it once both are in. This
PR does not touch the VMDisk path itself.

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map against the diff, which is the restore
preparation in `internal/backupcontroller/velerostrategy_controller.go`
and its unit tests, plus one verb in the backupstrategy-controller
ClusterRole and its helm-unittest: no package added or renamed, no
`values.yaml` or `values.schema.json` change, no CRD,
ApplicationDefinition, label, annotation, namespace or metric change.
`keepOriginalPVC` lives in the opaque `RestoreJob.spec.options`, and a
search of the organisation finds it only in this repository, so neither
the website nor the Terraform provider documents or models it.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): an in-place VMInstance RestoreJob with keepOriginalPVC=false now deletes each original disk PVC, and waits for it to be gone, before Velero restores it. Previously the PVC was left in place, Velero skipped restoring its data, the RestoreJob still reported Succeeded, and the disk's HelmRelease could never recreate its DataVolume ("Destination PVC already exists"). A Velero RestoreJob now also fails before touching the application when its Velero Backup is gone or not Completed, for example after the Backup expired at its TTL.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* When a restore is configured not to keep the original disk claim, the
claim is deleted before Velero restores it, allowing Velero to recreate
it.
* Restores wait while the original disk claim or its DataVolume is being
deleted. Deletion or status-check errors stop restore preparation.
* Restores configured to keep the original disk claim continue to
preserve it.
* Restores stop before changing resources if the referenced Velero
backup is missing or has not completed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 22:26:10 +03:00
Mattia Eleuteri 6d064a8602 fix(backups): delete a restored disk's PVC only once the restore can run
Deleting the original PVC of a keepOriginalPVC=false restore is only
safe when Velero can put the disk back. The Backup artifact outlives
the Velero Backup it points to, which Velero garbage-collects at TTL,
and nothing checked it before prepareForRestore halted the VM and
deleted the disks: a restore of an expired backup lost the volume where
it used to fail with the disk intact. The Velero Backup must now exist
and be Completed before any preparation starts.

A DataVolume that survives its deletion makes CDI recreate the claim
under the original name, which Velero then skips as an existing PVC,
so the restore would end Succeeded on a freshly imported disk. A failed
DataVolume deletion now fails the restore, and the PVCs are deleted
only once every DataVolume is gone. Reading the DataVolume back needs
get on datavolumes, which the ClusterRole did not grant.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-07 11:23:37 +02:00
Mattia Eleuteri 5956b459fb fix(backups): give each Postgres instance its own default WAL prefix
The default CNPG strategy archived a Postgres app under
<namespace>-<name>, inside a destinationPath that is also keyed by
namespace and name, and the driver fell back to the Cluster name when a
strategy set no serverName. Deleting the app does not empty the bucket,
so an app recreated under the same name archived its WALs and base
backups into its predecessor's stream: both start on timeline 1, and a
later restore of the old archive could pick the new cluster's base
backup or WAL segments.

barman-cloud refuses a non-empty archive only while
.check-empty-wal-archive exists in PGDATA. CNPG removes that file as
soon as ContinuousArchiving turns True, and with no plugin attached
archive_command is a successful no-op, so the file is gone minutes
after initdb, before the first BackupJob attaches the plugin.

Suffix the default serverName with the application UID, in the
strategy template and in the driver's fallback. The template guards
the UID with `with`, so an object without one renders the previous name
rather than "<no value>". A Cluster that already has the plugin keeps
its serverName, so only clusters never backed up take the new prefix,
and restores keep reading the source prefix from driverMetadata.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-10-06 16:35:42 +03:00
Andrey Kolkov 6555ae16b8 fix(mongodb): start a backup only once the operator has pushed the storage into pbm
The wait after a storage injection watched the cluster's spec, which is
not what pbm writes to. At the pinned operator version the backup
reconciler hands pbm only a storage name and records status.s3 from the
spec entry it read, while the storage pbm uses is whatever the cluster
reconciler last pushed into pbm's own config; that push flips PBMReady
False and back to True and the reconcile stamps observedGeneration when
it ends. A backup started between the apply and that push dumped to the
previous coordinates while its Backup recorded the new ones, so both its
restore and its prune pointed at the wrong bucket. The wait was also no
wait at all: the status write that named it re-queued the job at once,
and the live read always showed the apply. The mint now waits for
observedGeneration to reach the cluster's generation and for PBMReady
to be True, and fails at the backup deadline if the operator never gets
there.

The strategy, app and cluster reads still ran ahead of the operator CR
lookup and were bounded by the backup deadline, so a strategy absent for
one reconcile 40 minutes into a streaming dump failed the job and left
the finalized CR unreachable. The CR is resolved first now; with one in
hand the cluster is not read at all, and the reads the artifact needs
are bounded by the running ceiling, past which the job fails naming the
CR left in place.

The settle check treated a field the entry lacked as agreement, so a
strategy that newly set forcePathStyle minted in the same pass as the
apply that changed addressing. A missing field now counts as its zero
value, which keeps an entry the operator rewrote without an empty region
settled while catching a value the apply adds; the storage type is
compared outright.

The runbook still said the finalizer keeps a Plan from accumulating
objects, the RBAC comment still said the timeout cancel runs on either
flow, and four wait messages read a duration as a delay. All corrected;
the runbook also describes the pbm wait.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-29 11:02:57 +04:00
Andrey Kolkov 0aaf5e1e05 docs(mongodb): condense the psmdb-backups RBAC comment so it does not ship into the ClusterRole
The delete/patch grant justification had grown to an 11-line `#` block, which
(unlike a `{{/* */}}` comment) renders verbatim into every cluster's ClusterRole
and drifts out of date as the code moves. Condense it to a short pointer at
reconcileMongoDB / cleanupMongoDBBackup and the docs, where the detail is
versioned; keep only enough to explain the non-obvious grant.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov d091c9951c fix(mongodb): requeue after a fresh storage injection so the operator's cache can observe it
The first BackupJob of a useSystemBucket app SSA-injected the storage and minted
the operator PerconaServerMongoDBBackup in one reconcile. The psmdb operator
resolves spec.backup.storages from a cached cluster read when it services that
CR, and a cache that has not yet observed the apply takes the not-found branch
and latches the CR at State=error — which the operator never re-drives (it
returns for a terminal-state CR and watches only the Backup CR and Pods) and the
driver reads as a terminal failure. Only the first job per app is exposed; later
ones re-apply an already-declared storage.

When the pre-apply cluster did not already declare the storage, requeue after
the apply instead of minting in the same pass. The next reconcile finds the
storage in this driver's own cache (a good proxy that the operator's has caught
up too) and only then mints the CR — one 5s poll per app, well inside the
deadline. A fake client has no informer lag, so this is pinned by asserting the
first pass requeues and mints nothing rather than by reproducing the timing.

Also correct the RBAC delete-grant comment: the 30m timeout branch cancels a
not-yet-started operator CR on either flow, not just the useSystemBucket
cleanup, so a Failed BackupJob never leaves a CR the operator would run later.
And pin the wantBucket=="" restore-credential short-circuit with a no-bucket
source case.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov e653a4fa5d fix(mongodb): correct the finalizer-prune claims, pin the pre-read release CR, and fix the retention field text
The delete-backup finalizer prunes storage only for a `ready` backup at
v1.22.0; a timed-out backup never reaches `ready`, so the deadline-cancel
comment no longer claims the finalizer "takes anything partially written" —
nothing is written until the backup runs, which the cancel prevents. The
psmdbBackupTimedOut godoc no longer says the operator "has not started" the
backup, which is wrong for `requested` (the operator sets it as it dispatches);
it now describes the not-yet-running states it actually gates.

The pre-read CR handed to releaseMongoDBCleanup on the terminating path had no
test that failed when the argument was dropped, because the existing cases
either call the function directly or let the second Get succeed. Added a
cleanupMongoDBBackup case whose operator-CR Get fails after the first: the
release must strip from the passed object, not re-read, or the teardown wedges.

The useSystemBucket field said "retention move[s] to the Plan", but a Plan has
no retention field; corrected to "scheduling moves to a Plan; this flow has no
retention yet" and regenerated the schema, README and mongodb-rd. Softened the
"never pins a namespace in Terminating" wording in the cleanup godoc and RBAC
comment: the driver only bounds its own finalizer, the operator's release-lock
finalizer can still hold the CR while the operator is down.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov f8554d289d fix(mongodb): stop the system-bucket flow stranding archives, and reject an inert retentionPolicy
Two ways the shared cozy-backups bucket could accumulate objects nothing prunes.

A dump that outran the 30-minute deadline was declared failed while pbm kept
writing. The BackupJob's terminal-phase guard then never runs the branch that
mints the Cozystack Backup, so the archive lands with no Backup object to
represent it and no retention can reach it (the operator CR carries no ownerRef
either). Leave a running dump to finish — a real dataset routinely outruns 30m,
and a broken one still terminates through the operator's error/rejected state.
The deadline keeps bounding the case it was written for: the operator never
starts the backup (state stays "", requested or waiting), where nothing has
been written and failing strands nothing.

retentionPolicy is inert on this flow (the chart's tasks[].keep is gated out and
no Plan-level retention exists yet), so a value a tenant set expecting it to work
would grow the bucket silently. Reject a retentionPolicy changed from the chart
default when useSystemBucket is on, rather than ignore it; the default still
renders so an untouched install is unaffected.

Also on the cleanup path: hand the terminating-namespace release the operator CR
the caller already read and confirmed owned, so the finalizer strip is not gated
on a second Get whose transient failure would silently drop it and leave the CR
finalized in a namespace being torn down. Correct the ownership comments — the
delete-backup finalizer is not one the operator never sets (it finalizes
task-minted backups on the legacy flow); what keeps it unambiguous is that
cleanup only ever resolves the one CR named in driverMetadata. Add the missing
isolation test that a legacy app's own s3-storage is never overwritten by the
injection call site, and move the strategy's storage comment into a Go-template
comment so it no longer ships into every rendered Strategy CR.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 542f42ef67 fix(mongodb): make the system-bucket cleanup teardown-safe and single-source its ownership
Four issues in the system-bucket cleanup path.

Teardown wedge. The cleanup requeued forever waiting for the operator to
prune, and the escape-hatch annotation released the Cozystack Backup without
touching the delete-backup finalizer the driver stamps on the operator CR.
So a namespace teardown, where the operator's delete-backup step cannot run
(it execs against a pbm agent that is going away), left the operator CR — and
the namespace — stuck in Terminating with no way out but hand-patching
finalizers. Cleanup now detects a terminating namespace (as Redis/Rabbitmq
do) and, on that or the annotation, strips its own finalizer and releases,
leaving the object for a bucket lifecycle policy. Needs patch on
perconaservermongodbbackups for the finalizer strip.

Ownership was read from two places written at different times: the finalizer
(stamped when the operator CR is created) and a driverMetadata flag (written
when the artifact is created, a later reconcile). If the app's useSystemBucket
flag flipped between them they disagreed. Read ownership from the finalizer on
the one CR named in driverMetadata (the one the driver minted): resolving it by
name, never by scan, keeps the operator's own cron-* task-minted backups (which
do carry percona.com/delete-backup at v1.22.0) out of scope, so the finalizer on
that named CR is an unambiguous single marker. Drops the driverMetadata flag.

Injection error was terminal. A transient server-side apply — an apiserver
hiccup, or a conflict while the operator writes the same cluster — failed the
BackupJob outright. Requeue with backoff instead, bounded by the existing
deadline, matching the cluster/app read errors alongside it.

Injection ran before the enabled check, mutating a cluster that can never
back up. Gate it on backup.enabled so a disabled cluster is left untouched.

Also: reframe the restore S3 backfill as the defensive guard it is (the
pinned operator always echoes status.s3, so the live path normally carries
coordinates), and correct the psmdbtypes doc that still claimed the driver
never patches the operator CRs.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov f27a02f1b6 docs(mongodb): correct the inlined-storage story and surface the TLS knob
The strategy godoc still said the storage config is NOT inlined and the
strategy only names the storage — which controller-gen copies verbatim into
the CRD, so kubectl explain served the new s3 field and its own denial from
one object. Rewrite it (and the StorageName field doc) to describe both
flows, and regenerate the CRD.

Render insecureSkipTLSVerify on the default MongoDB strategy: the field is
plumbed end to end into the driver-injected storage entry and backed by a
platform knob, but the template set only forcePathStyle, pinning it to
false so a self-signed external S3 that Redis backups honour would fail
MongoDB x509 verification. Pin it in the rendering test alongside the other
s3 coordinates.

Fix the useSystemBucket field description: angle brackets rendered as empty
HTML tags in the README table and the dashboard schema, so wrap the path in
backticks (as the postgres field already does) and note that schedule,
retentionPolicy and PITR are ignored on this flow. Regenerate the chart
schema, README and mongodb-rd.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 1d05dafb2e fix(mongodb): own and prune the system-bucket archive, and heal the restore read path
Two correctness gaps on the useSystemBucket flow.

Retention. Gating spec.backup.tasks out of the chart also dropped the keep
that psmdb task retention used, and an on-demand PerconaServerMongoDBBackup
is not covered by task retention, so nothing pruned the shared cozy-backups
bucket — a retention-swept Plan grew it unboundedly for every tenant at
once. Own the archive the way the Redis/Rabbitmq drivers own theirs: stamp
the operator's percona.com/delete-backup finalizer onto the backups minted
on this flow, and on Cozystack Backup deletion delete that operator CR and
wait for the finalizer to free the object before releasing. A legacy backup
(tenant's own bucket) keeps the pre-existing hands-off no-op. Adds the
delete verb on perconaservermongodbbackups and a skip-artifact-cleanup
escape hatch for a wedged operator, matching the sibling drivers. Also drops
the unused update verb from the psmdb cluster RBAC (server-side apply is a
patch, not an update).

Restore. resolveMongoDBBackupSource returned early off the live operator
Backup CR whenever it reported a destination, copying status.s3 as-is. On
this flow the operator can echo a destination with status.s3 empty (the
storage was injected, not declared), and the target has no declared storage
to resolve from either, so the restore stranded with no coordinates.
Backfill a missing live S3 block from the snapshot the BackupJob recorded.

Also correct the precondition godoc: the pbm agents start on
spec.backup.enabled alone; the declared storage is required by the backup
CR's resolution, not by the agents.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 4ea4888685 fix(mongodb): key system-bucket injection off the app flag, not live storage
The MongoDB backup driver decided what to inject and what to record from
the live cluster's storage set rather than from the app's opt-in flag. That
had three consequences.

The snapshot fallback fired for legacy apps too. Its gate was
`snap.S3 == nil && rendered.S3 != nil`, but the coordinates moved onto the
shared cozy-default strategy, so rendered.S3 is now non-nil for every
MongoDB app. A legacy backup written to the tenant's own bucket got its
snapshot rewritten to the platform bucket, sending restore to the wrong
place. Gate the fallback on the app's backup.useSystemBucket flag instead,
threaded through createMongoDBBackupArtifact, and pin the legacy case.

The injected storage never caught up with the strategy. Injection was
gated on the named storage being absent, so it ran once per cluster and
froze the coordinates at the first backup; a later endpoint/region change
or bucket re-provision was a silent no-op, and Helm could not repair it
because the useSystemBucket chart renders no storages. Key injection off
the app flag and re-apply on every BackupJob (the prefix is deterministic,
so this never splits the archive the way CNPG's serverName would). Add
UseSystemBucket to mongodbapp.MongoDBSpec so a legacy app is protected by
the flag, not by whatever happens to be on the cluster.

The docs said the opposite of what the driver does. backup-classes.md
claimed "the MongoDB driver cannot inject its S3 target per-backup";
correct it and add an opt-in-to-the-system-bucket section that names which
values stop being read and that scheduled backups + PITR must move to a
Plan. Fix the endpoint table row, the tuning reference, the RBAC comment
that said the driver never patches, the strategy header/inline comments,
and the example comment. Also assert notExists on spec.backup.pitr in the
helm test, and lift the duplicated cozy-backups-creds default into a const.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-22 12:03:58 +04:00
Andrey Kolkov 730592f323 feat(mongodb): support the system-bucket backup flow (useSystemBucket)
MongoDB was the only cozy-default app with no useSystemBucket flow: its
backup storage was always static on the app CR (bucket/endpoint/creds
from backup.* values + a chart-emitted <release>-s3-creds Secret), so a
tenant could not back up to the platform system bucket out of the box the
way postgres/clickhouse/etc. do.

PBM takes bucket/endpointUrl/region/prefix as static CR fields (not
secret refs) and the platform bucket name/endpoint are only known at
BackupJob time, so - unlike ClickHouse's runtime env secretKeyRef - the
chart cannot render them. The fix mirrors the CNPG driver: the chart adds
a useSystemBucket flag that keeps spec.backup.enabled (so the pbm agents
run) but omits the static storages/tasks/pitr and the creds Secret; the
cozy-default MongoDB strategy carries the coordinates (endpointUrl already
scheme'd by the endpoint helper, credentialsSecret cozy-backups-creds);
and the psmdb driver SSA-injects spec.backup.storages[s3-storage] onto the
live PerconaServerMongoDB at BackupJob time under its own field manager
with ForceOwnership, so a Flux re-render never reverts it. Injection is
gated on the storage being absent, so a legacy app that ships its own
static storage is left untouched. The backup snapshot falls back to the
strategy coordinates when the operator status echo is empty, so the
restore path rebuilds backupSource (credentialsSecret cozy-backups-creds
is projected into the restore namespace before dispatch).

RBAC grows patch/update on perconaservermongodbs for the injection. A
bootstrap.enabled + useSystemBucket render guard mirrors postgres.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-22 12:03:58 +04:00
Andrey Kolkov 01967b56d3 docs(backups): correct the Kafka .Mode doc and the embedded-script templating note
The KafkaSpec.Template doc listed .Mode as "backup" or "restore", but the
driver also renders it as "cleanup" when reaping a removed Backup's object;
the comment is copied verbatim into the CRD, so kubectl explain and the
dashboard showed two values while a custom strategy's restore branch would
run on the cleanup pass. List all three, matching the Redis sibling, and
regenerate the CRD.

The strategy template also claimed the embedded script "reads only env
vars, never Helm/controller values, so no tpl is applied". The controller
renders every string leaf of the PodTemplateSpec through internal/template,
args[0] included, so the script is templated on every run. State that
instead, without literal template braces that Helm would itself expand.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-14 17:55:48 +04:00
Andrey Kolkov 0d918e1f91 fix(backups): resolve Kafka backup client image from the target broker
The Kafka metadata strategy pinned kafkaBackupClientImage to an upstream
Strimzi tag with no producer: nothing stamped it and Renovate does not
manage helm-values, so it drifted from the broker version the operator
provisions. A dedicated small client image (the sibling curl-only drivers'
follow-up) is not viable here because the driver runs the Kafka Admin-API
CLI, which only the broker image ships.

Resolve the client image from a live broker pod of the target Strimzi Kafka
at reconcile time instead, so the CLI always matches the broker (any version
the operator picks, not only the chart default), and record it on the Backup
so cleanup can reuse it once the source cluster is gone. The chart no longer
pins the image; the strategy template renders it as the .ClientImage
placeholder the controller fills.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:52 +04:00
Andrey Kolkov 14c6af122e fix(backups): restore Kafka S3 port workaround and hold the script with a test
- Regression fix: an earlier change dropped the curl --connect-to port workaround along
  with -k, but curl 7.76.1 (the image's) 403s on ANY ported endpoint,
  which is the shape backupStorage.endpoint ships (:8333) and what
  provisionBucket:false and s3.ingress.enabled:false produce. Restore
  --connect-to (sign a portless URL, redirect to the real port), keeping
  -k off so TLS is still verified. IPv6-safe host/port split, path prefix
  preserved. The port-less ingress passed; the ported case is the one that regressed.
- Correct the S3 envelope claim in values.yaml and backup-classes.md: the
  403 is decided by the port, not a body-hash the backend recomputes, and
  --connect-to handles it on SeaweedFS and MinIO alike.
- Hold the script fixes at the layer the contract lives at. Externalise
  the driver to files/kafka-backup.sh (embedded verbatim via .Files.Get,
  render byte-identical) and add hack/kafka-backup-script.bats, which
  EXECUTES it against stubbed kafka CLIs + curl and asserts behaviour: \Q
  literal match on the colliding pair, --connect-to with no -k on a ported
  endpoint, fail-closed --list, comma-value bracket grouping, RF-divergence
  abort, and the cleanup delete's HTTP handling. Not a template grep.
- Pin the Backup-delete dispatch and the readiness deadline in Go:
  TestCleanupOnDelete_Kafka_RoutesToKafkaCleanup and
  TestReconcileKafka_FailsAfterReadinessDeadline.
- Docs: config restore is additive (no --delete-config); export time
  scales with topic count and the Job has no wall clock (as its siblings).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:32 +04:00
Andrey Kolkov 0270650237 fix(backups): harden Kafka metadata S3 handling and own the artifact
- Drop `-k`: mirror the RabbitMQ S3 layer (ARTIFACT_URI as the single
  source of truth, forcePathStyle honoured, whole base URL kept) so
  curl --aws-sigv4 verifies TLS against the ACME endpoint the helper
  returns. The derived endpoint is default-port, so the curl 7.76.1
  --connect-to port workaround is gone too; this also fixes the IPv6
  and path-prefix mangling in the old host-splitting.
- Own the artifact: a cleanup mode in the script plus cleanupKafkaBackup
  wired into BackupReconciler.cleanupOnDelete delete a deleted Backup's
  object (best-effort, waited on) instead of orphaning it, matching the
  RabbitMQ precedent. Kafka has the identical own-your-object property.
- Restore probes existence with --list (fail closed), so an unreachable
  broker aborts instead of being misread as "topic absent" and created.
- Hold the fixes with tests at the right level: a colliding topic pair
  (audit.events / audit-events, distinct partition counts) in the e2e so
  a revert of the \Q...\E literal-match breaks the round-trip, and Go
  tests pinning the scope stamp and the cleanup Job.
- Correct the docs/comments: endpoint-shape row, the "cached on nodes"
  claim, and the narrower curl 7.76.1 S3 envelope (empty-body payload
  hash) that makes a small modern-curl client image the right follow-up.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:06 +04:00
Andrey Kolkov 537aadf389 fix(backups): fix Kafka backup S3 signing and literal topic names
Reproduced end-to-end against a real Kafka 3.9.1 broker:

- [CRITICAL] The Strimzi image ships curl 7.76.1, whose --aws-sigv4
  omits a non-default port from the SigV4 canonical Host while sending it
  in the Host header, so every S3 PUT/GET 403s and the driver never
  uploads or downloads. Sign/send the bare host and redirect the
  connection to the real port with --connect-to (the same fix the
  topic-data example already carries), and accept the in-cluster
  self-signed cert with -k.
- [CRITICAL] kafka-topics --topic is a Java regex, not a literal: a
  backup entry for "audit.events" also matched a live "audit-events", so
  restore repartitioned an unrelated topic irreversibly and backup
  recorded the wrong partition/replication values. Wrap the name in
  \Q...\E on every filter call (--describe on both paths, --alter on
  restore) so it matches exactly one topic; --create and
  kafka-configs --entity-name are already literal. Applied in the example
  helpers too.
- [MAJOR] cozy-default's Kafka route looked like a data backup at
  runtime. Stamp scope=topic-metadata on the produced Backup's
  DriverMetadata and say "topic definitions and configs only; no message
  data" in the Ready condition, so kubectl describe backup is honest.
- [MAJOR] Add reconcile tests for the two Go behaviours a mutation could
  silently drop: the Strimzi-Ready precondition on backup (BackupJob
  stays out of Succeeded and creates no Job) and the applicationRef Kind
  gate on the restore target.
- [MINOR] Fail closed on the config-capture grep (exit != 1 aborts) and
  clarify the image pin has no automated producer.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:21:25 +04:00
Andrey Kolkov 405eb25095 fix(backups): harden Kafka metadata backup
- Fail closed on topic listing: set -o pipefail, capture kafka-topics
  --list --exclude-internal exit separately, and abort on failure so an
  unreachable broker can no longer produce an empty backup reported as
  Succeeded. An empty list after a successful call stays legitimate.
- Check each kafka-topics --describe exit and validate the parsed
  partition count / replication factor before writing a record, so a
  timed-out describe cannot write a corrupt orders\t\t\t line.
- Store topic configs one per line with key and value in separate
  tab-delimited fields, sourced from kafka-configs --describe (one config
  per line), so a value containing a comma (cleanup.policy=compact,delete)
  round-trips; on restore each config is applied with the value bracketed
  only when it contains a comma.
- Detect topic divergence on restore instead of a silent --if-not-exists
  no-op: grow partitions with --alter when the backup asks for more, and
  fail loudly (naming the topic and both values) when it asks for fewer
  partitions or a different replication factor. A restore that cannot
  reach the recorded state no longer reports Succeeded.
- Add KafkaStrategyKind to the RestoreJob cleanup-dispatch no-op group so
  deleting a Kafka RestoreJob does not run a pointless Velero cleanup; add
  a regression test alongside the siblings, plus a reconcile Kind-gate
  test and a rendering test that actually renders the strategy template.
- Wire the kafka-metadata Chainsaw suite into both selectors
  (select-install suite_to_source, select-e2e src_to_suites) so editing
  the kafka app or the suite selects it and the bats unit tests pass.
- Pin the client image to the operator's default broker version
  (kafka 3.9.1) with a digest, and align the example default; fix the
  values comment. Point the RabbitMQ comment references (which shipped in
  the CRD description) at the Job strategy actually mirrored, and trim the
  in-pod comments that rendered into the manifest.
- Document the Strimzi Topic Operator caveat: for a topic owned by a
  KafkaTopic CR the operator reconciles its config from the CR, so an
  in-place config restore is durable only for out-of-band topics.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:21:24 +04:00
Andrey Kolkov a410a3f48b feat(backups): Kafka topic-metadata backup via a dedicated strategy driver
Add a strategy.backups.cozystack.io/Kafka driver that backs up the topic
metadata of a Cozystack-managed Kafka application — partition counts,
replication factors and non-default topic configs — through the Kafka
Admin API, and restores it with kafka-topics --create + kafka-configs
--alter. Modelled on the Job/Rabbitmq strategies: the driver renders a
one-shot batch Job, gates on the Strimzi Kafka cluster reporting Ready,
and keys each backup by .BackupName so runs do not overwrite each other.

Scope is deliberately narrow: message payloads, consumer-group offsets,
ACLs, quotas and KafkaUsers are out of scope, and there is no PITR.
Restored topics are created directly (unmanaged on the target) — the
accepted tradeoff of an Admin-API-only driver. Both restore flows are
implemented: in-place recreates dropped topics into the source, to-copy
applies the source's topics onto a freshly-bootstrapped empty cluster.

Wire it into cozy-default (strategy-kafka-default.yaml gated on the
bucket name, a BackupClass binding, read RBAC on kafka.strimzi.io/kafkas
for the readiness gate, and the Strimzi Kafka client image), document it
in backup-classes.md, and add an honest round-trip example plus Chainsaw
e2e (examples/backups/kafka-metadata, kafka-3-metadata-roundtrip) that
proves a per-run retention.ms sentinel survives both restores.

Unit tests cover the driver's pure helpers (application-ref validation,
readiness verdict, parameter round-trip, per-run artifact-URI keying).

Not yet verified on a live cluster.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:20:27 +04:00
Aleksei Sviridkin 1f04e31ed3 feat(backups): add Redis backup/restore strategy driver (#3992)
## What this PR does

Adds backup/restore support for the `Redis` app (spotahome
`RedisFailover`) via a dedicated `strategy.backups.cozystack.io/Redis`
strategy controller. The spotahome operator ships no backup CRD, so the
driver follows the generic-script-Job pattern (a specialisation of the
Job/Altinity drivers): it runs a one-shot Job that dumps an RDB snapshot
from the current master to the system bucket and replays it on restore.

- **Backup**: discover the current master via the operator's sentinel
Service (`rfs-<name>`, master `mymaster`) → `redis-cli --rdb` → `curl
--aws-sigv4` PUT to `s3://<bucket>/<namespace>/<app>/dump.rdb`.
- **Restore**: GET the object → convert RDB → RESP (`hdt3213/rdb`) →
`FLUSHALL` + `redis-cli --pipe` into the target's master. Both
**in-place** and **to-copy** are implemented; a to-copy restore reads
the source's prefix via `.Backup.ApplicationRef.Name` while writing into
a differently-named target.
- **PITR**: not supported — RDB is a discrete snapshot with no
continuous log to replay to an arbitrary time (the etcd-like case).

The driver is a specialisation of the Job/Altinity script-Job flow with
a `Redis` applicationRef gate and an app-readiness precondition
(surfaces a precise `Ready=False` instead of a silent hang; blocks only
on an explicit `Ready=False` so a missing condition never wedges
backups). Strategy parameters round-trip through the Backup's
`DriverMetadata`; there is no `underlyingResources` snapshot (the Job
pattern re-renders on restore).

### System bucket

Coordinates and credentials reach the Job as env `secretKeyRef` from the
projected `cozy-backups-creds` (the ClickHouse/Altinity delivery model),
so **the app chart is untouched** — no `backup.useSystemBucket`, no new
RBAC (`batch/jobs` and `apps.cozystack.io/*` are already granted). The
bare-host `endpoint` gets `https://` prepended when unscheme'd; against
a self-signed endpoint the strategy verifies with the CA from
`backupStorage.endpointCASecretName` when set, otherwise against the
image's system CA store; skip-verify is a separate explicit
`backupStorage.insecureSkipTLSVerify` opt-in, never a fallback. The
`cozy-default-redis` strategy is gated on `$bucketName` (CRD-ordering)
and bound in the `cozy-default` BackupClass.

### Changes

- `api/backups/strategy/v1alpha1/redis_types.go` + generated CRD and
deepcopy.
- `internal/backupcontroller/redisstrategy_controller.go` (+ unit tests)
wired into both dispatch switches, `supportedBackupStrategyKinds`, and
the no-op restore-cleanup case.
- `cozy-default-redis` strategy template + `cozy-default` BackupClass
route + helm rendering-gate coverage.
- `images/redis-backup` (redis-cli + curl + RDB→RESP converter) +
Makefile target + `backupStrategyController.redisBackupClientImage`.
- `examples/backups/redis/` — an honest marker round-trip for both
restore flows.

### Validation

`go build`/`vet`, `go test ./internal/backupcontroller/...
./api/backups/...`, `make generate` (clean), and `helm unittest` (10/10)
all pass. Not yet cluster-verified.

## Downstream

None.

```release-note
Add backup and restore support for the Redis application (spotahome RedisFailover) via a new Redis backup strategy, wired into the cozy-default BackupClass. Supports in-place and to-copy restore; PITR is not supported.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Redis backup and restore support using RDB snapshots in
S3-compatible storage.
  * Supports restoring data in place or to a separate Redis instance.
* Added configuration for Redis backup images, custom HTTPS certificate
authorities, and TLS verification.
* Added a complete Redis backup and restore demonstration with automated
setup, verification, and cleanup.

* **Documentation**
* Documented Redis storage, credentials, endpoints, TLS, and restore
behavior.
  * Added administrator and tenant scenario guides.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-10 19:17:26 +04:00
Andrey Kolkov 92edfb810b fix(backups): run the strategy bats under bash; move template essays to {{/* */}}
The behavioural bats test ran the rendered `set -euo pipefail` script via
`sh`; the CI unit-tests runner's /bin/sh is dash, which rejects `-o pipefail`
at line 1 and reddened the whole suite (production is busybox ash, which
supports it, so the script itself is fine). Run it via `bash`, matching ash
semantics and the hack/migration-50-etcd-adopt.bats precedent.

Convert the multi-paragraph `#` rationale blocks in the shipped strategy
template to `{{/* */}}` so Helm strips them at render instead of copying the
prose verbatim into every rendered Strategy CR, where it drifts as the code
changes. Short one- to three-line guard comments stay as `#`.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 19:23:37 +04:00
Andrey Kolkov 1289f522fa fix(backups): unbreak install, kill destructive cleanup, reach the resource knob
Store the Redis strategy Template schemaless, as the Rabbitmq strategy
already does: without the markers controller-gen emitted the full
PodTemplateSpec schema and the CRD reached ~602 KB, pushing the chart's
Helm-release Secret past the 1 MiB cap so backupstrategy-controller failed
to install at all.

Give the example strategy a cleanup branch, and both scripts an explicit
`elif restore` / `else exit 1`, so a deleted Backup rendered with
Mode=cleanup deletes its object instead of falling into the restore path
and FLUSHALLing the live master.

Forward a backupStrategyController block through the platform bundle
alongside backupStorage, so redisBackupResources — the restore-Pod sizing
knob the loader-not-ready message points at — is reachable via
spec.components.platform.values.backupStrategyController, not merely
settable on a Helm-managed Strategy CR that reverts on the next upgrade.

Correct the endpointCASecretName doc row to match the now-optional mount,
and `int` the activeDeadlineSeconds interpolation so a large values-file
override renders as an integer, not exponent form.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 16:53:50 +04:00
Andrey Kolkov 651a6e3023 fix(backups): harden the redis restore AOF wait and image build
Wait for the throwaway loader's AOF rewrite to finish - no rewrite in
flight, last one ok, base AOF on disk - instead of a bare
aof_rewrite_in_progress:0 check, which right after `appendonly yes` can
fall through before the scheduled rewrite starts and replay an absent or
half-written base AOF. Same fix in the example strategy.

Declare the package image targets PHONY and serialize image-redis-backup
after image-backupstrategy-controller so their `yq -i values.yaml` digest
stamps cannot race under `make -j`. Run the demo cleanup under `set -e`
(every delete is already --ignore-not-found) so a real API failure aborts
instead of printing "Cleanup complete" over a half-torn-down demo.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 16:23:12 +04:00
Andrey Kolkov b4d621f4f7 fix(backups): delete Redis backup objects on cleanup and size the restore Pod
Delete the backup object when its Backup is removed. The Redis driver
owns its S3 object (one per BackupJob, no engine retention prunes it), so
it now spawns a cleanup-mode Job that deletes the object and holds the
Backup's finalizer until it completes, mirroring the Rabbitmq driver.
Without this a retention-pruned Plan grew the bucket unboundedly and the
key was lost with the CR.

Size the strategy Pod from backupStrategyController.redisBackupResources
so an admin can raise the restore ceiling (the throwaway loader holds the
whole dataset in memory) via chart values, which a direct edit of the
cluster-scoped, Helm-managed Strategy CR could not survive. The
loader-not-ready diagnostic now names the memory limit, the likeliest
cause, instead of only the image version.

Make the endpoint CA volume optional: nothing projects that Secret into
the app namespace, so a required mount wedged every backup Pod on
FailedMount; a missing ca.crt now falls through to the system CA store
and fails the handshake fast, which is not the same as skipping verify.

Promote the Pod-level activeDeadlineSeconds to the Job so a wedged run is
bounded once, not (backoffLimit+1) times. Use find, not ls, for the base
AOF so its no-match diagnostic is not dead code under pipefail.

Pin the object key reaching the rendered Job: both fixtures now render
--key={{ .ObjectKey }} and assert it, so zeroing the key (which left the
package green) is caught. Document .ObjectKey and the cleanup mode in the
CRD render-context, list cozy-default-redis among the Helm-managed CRs,
and mirror the CA/TLS and find fixes into the example strategy.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 12:19:02 +04:00
Andrey Kolkov 56e10b1580 fix(backupstrategy-controller): gate the default MongoDB Strategy on its CRD (#3980)
## What breaks today

Upgrading any existing cluster to a build that carries the new MongoDB
backup driver wedges the `backupstrategy-controller` HelmRelease
permanently, with `resource mapping not found for name:
"cozy-default-mongodb" ... no matches for kind "MongoDB" in version
"strategy.backups.cozystack.io/v1alpha1"; ensure CRDs are installed
first`.

This chart ships its `strategy.backups.cozystack.io` CRDs as ordinary
templates, and Helm resolves every document in a release manifest
through the cluster's RESTMapper in `Build()` **before** it applies any
of them. A Strategy CR whose CRD is not served yet therefore fails
`Build()`, so the release applies **zero** objects — including the CRD
that would have made the mapping resolve — and every retry renders the
same manifest. Nothing recovers on its own.

## Why the gate that exists does not cover it

The chart already knows this trap: it is why every default Strategy
gates on the bucket name. That gate covers the **first install** of a
fresh cluster, where nothing is resolvable yet and no Strategy renders
at all. It cannot cover shipping a **new kind** to a chart that is
already installed — there the bucket name resolved releases ago, so the
first upgraded revision renders the new CRD and the new Strategy CR
together and wedges.

## The fix

Gate the CR on its CRD actually being served, through a `crdEstablished`
helper. The helper asks about the **CRD object** rather than listing the
custom kind, because `lookup` on a kind the apiserver does not serve is
a render *error*, not an empty result, which would trade this wedge for
a different permanent failure; `apiextensions.k8s.io/v1` is always
served, so that lookup can only answer yes or no. It requires
`Established=True` rather than mere existence, because that is when the
kind enters the RESTMapper.

Convergence afterwards is the mechanism the bucket gate already relies
on, and is **not** the HelmRelease interval: once the CRD is served, the
controller's `DefaultObjectsGate` sees the kind mapped and the CR absent
and forces a real Helm upgrade whose render includes it. Kinds that do
not map at all are skipped there (`meta.IsNoMatchError`), which is what
stops the gate forcing upgrades against a render that cannot yet produce
the object.

`bucketNameOverride` short-circuits the CRD lookup for the same reason
it short-circuits the bucket lookup: it marks an offline render — `helm
template`, a CI diff, a unit test — where there is no apiserver to ask.
Live deploys go through Flux and never set it.

## How it was found

The release upgrade E2E lane in #3276, on the `v1.6.2 → main` path.
`mongodbs.strategy.backups.cozystack.io` is the only definition added to
this chart since the tag, so this is reachable only by upgrading an
existing install — no install-only suite can see it.

## Verification

Reproduced and fixed against a live cluster that happens to be in
exactly the pre-upgrade state (the seven older strategy CRDs present,
`mongodbs` absent). `helm template --dry-run=server` of main's chart
fails with the error above and produces zero documents; the same render
with this change succeeds with 21 documents and the Strategy correctly
withheld. The helper's two branches were probed the same way against
that cluster: a served CRD answers `true`, an absent one answers empty.

The new unit case pins the gate offline through the external-S3 path,
where the bucket name resolves without the offline override so that only
the CRD gate can hold the CR back. Removing the gate turns exactly that
case red; the suite is 19/19 green with it.

```release-note
Fixed a permanent failure of the backupstrategy-controller release when upgrading a cluster to a version that adds a new backup-strategy kind: the default MongoDB Strategy is now withheld until its CRD is served, instead of failing the whole release with "no matches for kind".
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved backup strategy rendering during upgrades by waiting until
the required MongoDB resource definition is available.
* Preserved offline rendering when an explicit bucket name is provided.
* Prevented invalid default strategies from being generated when the
required resource definition is unavailable.

* **Tests**
* Added coverage for offline rendering and unavailable
resource-definition scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-09 11:17:28 +04:00
Andrey Kolkov 45a2f54c62 feat(backups): back up TLS-enabled Redis apps via the default strategy
The shipped cozy-default-redis strategy connected to Sentinel and the
master in plaintext, so a BackupJob against a Redis app with tls.enabled
(the operator moves both to a TLS-only listener) failed at master
resolution. Mount the app CA the operator publishes at
redis-<app>.ca-cert as an optional Secret, and dial redis-cli with
--tls --cacert exactly when its ca.crt is present, so a non-TLS app
keeps the plaintext path and the throwaway restore loader stays
local-only. Covers the default authClients: no; mutual TLS is out of
scope, since the platform issues no client certificate.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-08 18:47:13 +04:00
Andrey Kolkov b0eb104276 feat(backups): add Redis backup/restore strategy driver
The spotahome RedisFailover operator ships no backup CRD, so add a
dedicated strategy.backups.cozystack.io/Redis driver that runs a one-shot
Job: it dumps an RDB snapshot from the current master (discovered via the
operator's sentinel Service) to the system bucket over S3, and on restore
replays it (RDB -> RESP via redis-cli --pipe) into the target's master.

Structurally a specialisation of the Job/Altinity script-Job pattern with
a Redis applicationRef gate and an app-readiness precondition. Backup
parameters round-trip through the Backup's DriverMetadata (no
underlyingResources snapshot); restore drives from .Backup.ApplicationRef
so a to-copy restore reads the source's object-storage prefix while
writing into a differently-named target. Both in-place and to-copy are
supported; PITR is not (RDB is a discrete snapshot).

- api: Redis/RedisList strategy type + generated CRD and deepcopy
- driver: reconcileRedis / reconcileRedisRestore + pure helpers, wired
  into both dispatch switches, supportedBackupStrategyKinds and the no-op
  restore cleanup case
- default: gated cozy-default-redis strategy (system-bucket coords via
  cozy-backups-creds env secretKeyRef, https:// scheme + optional CA) and
  the Redis route in the cozy-default BackupClass
- image: images/redis-backup bundling redis-cli + curl + an RDB->RESP
  converter, wired via backupStrategyController.redisBackupClientImage
- tests: driver unit tests (gate, precondition, render, param round-trip,
  both reconcile paths) and the helm rendering-gate coverage
- example: examples/backups/redis honest marker round-trip for both
  restore flows

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-08 17:56:20 +04:00
Andrey KolkovandClaude Opus 4.8 ba32df69c0 feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.

Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

  - reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
    PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
    --aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
  - A Go-read precondition gates each run on the RabbitmqCluster reporting
    AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
    precise Ready=False wait instead of a Job that starts and fails.
  - The produced Backup records the object location on status.artifact.uri
    (rendered from spec.artifactURITemplate; the Backup CRD has no status
    subresource, so it persists on Create, as in the Etcd driver).

Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.

Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.

The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-01 18:21:10 +04:00
Myasnikov Daniil 50ed307fef fix(backupstrategy-controller): gate the default MongoDB Strategy on its CRD
Upgrading a cluster to a build that carries the new MongoDB backup driver
wedges the backupstrategy-controller release permanently:

  Helm upgrade failed: resource mapping not found for name:
  "cozy-default-mongodb" ... no matches for kind "MongoDB" in version
  "strategy.backups.cozystack.io/v1alpha1"; ensure CRDs are installed first

This chart ships its strategy.backups.cozystack.io CRDs as ordinary
templates, and Helm resolves every document in a release manifest through
the cluster's RESTMapper in Build() BEFORE applying any of them. A
Strategy CR whose CRD is not served yet therefore fails Build(), so the
release applies ZERO objects — including the CRD that would have made the
mapping resolve — and every retry renders the same manifest. Nothing
recovers on its own.

The chart already knows about this trap: it is why every default Strategy
gates on the bucket name. That gate covers the first install of a fresh
cluster, where nothing is resolvable and no Strategy renders at all. It
cannot cover shipping a NEW kind to a chart that is already installed,
which is what happened here: the bucket name resolved releases ago, so
the first upgraded revision renders the new CRD and the new Strategy CR
together and wedges.

Gate the CR on its CRD being served, via a crdEstablished helper that
asks about the CRD OBJECT rather than listing the custom kind — `lookup`
on a kind the apiserver does not serve is a render error, not an empty
result, which would trade this wedge for a different permanent failure.
apiextensions.k8s.io/v1 is always served, so that lookup can only answer
yes or no. Established=True rather than mere existence, because that is
when the kind enters the RESTMapper.

Convergence afterwards is the mechanism the bucket gate already relies on
and is NOT the HelmRelease interval: once the CRD is served, the
controller's DefaultObjectsGate sees the kind mapped and the CR absent
and forces a real Helm upgrade whose render includes it. Kinds that do
not map at all are skipped there (meta.IsNoMatchError), which is what
stops the gate forcing upgrades against a render that cannot yet produce
the object.

Found by the release upgrade E2E lane on the v1.6.2 → main path, where
`mongodbs.strategy.backups.cozystack.io` is the only definition added
since the tag.

Verified against a live cluster that is in exactly the pre-upgrade state
(seven older strategy CRDs, no mongodbs): `helm template --dry-run=server`
of main's chart fails with the error above and produces zero documents,
while the same render with this change succeeds with 21 documents and the
Strategy correctly withheld. The helper's two branches were probed the
same way — a served CRD answers true, an absent one answers empty.

The new unit case pins the gate offline through the external-S3 path,
where the bucket name resolves without the offline override so only the
CRD gate can hold the CR back; removing the gate turns exactly that case
red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-28 11:29:59 +05:00
Myasnikov Daniil 4f9c20cac0 fix(backupstrategy-controller): gate the default MongoDB strategy on bootstrap
A fresh install of packages/system/backupstrategy-controller fails
permanently and applies nothing:

  Helm install failed for release cozy-backup-controller/backupstrategy-controller
  resource mapping not found for name: "cozy-default-mongodb"
  no matches for kind "MongoDB" in version "strategy.backups.cozystack.io/v1alpha1"

The chart ships its own strategy.backups.cozystack.io CRDs as ordinary
templates (templates/crds.yaml globs definitions/*.yaml), and Helm
resolves every document in a release manifest through the cluster's
RESTMapper in Build() before it applies any of them. An ungated Strategy
CR therefore has no mapping on the first install, Build() fails, and the
release applies ZERO objects, including the CRDs that would have made
the mapping resolve. Nothing recovers: every retry renders the same
manifest, so this is permanent rather than a race, and Helm's kind
sorter cannot help because it orders the apply, not the Build.

The seven sibling strategies never hit this because each gates on a
resolved $bucketName, which is empty until the COSI BucketClaim status
is reconciled. That gate is documented as "the CR needs the bucket
name", and it silently doubles as the CRD-ordering guard. MongoDB
genuinely does not need the bucket name, so the gate was left off and
the accidental protection went with it.

Gate it too, for the ordering reason rather than the value, and record
that reason in the template so the next bucket-independent strategy
keeps the gate. Convergence is unchanged: DefaultObjectsGate already
routes apps.cozystack.io/MongoDB to cozy-default-mongodb and forces a
second revision once any routed object is missing, which is how the
seven siblings materialise today.

Verified by rendering both windows. Before: the bootstrap render emits
one Strategy, MongoDB, alongside the eight not-yet-applied CRDs. After:
it emits none, and with backupStorage.bucketNameOverride set it emits
all eight.

tests/rendering_test.yaml asserted count: 1 for MongoDB under a case
named "bootstrap window (no resolvable bucket name)", so the defect was
codified green. helm-unittest has no RESTMapper and cannot see it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 14:53:13 +05:00
Andrey KolkovandClaude Opus 4.8 dff56aa998 feat(backups): add a native MongoDB (psmdb) backup-strategy driver
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so
`apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and
ClickHouse native drivers — logical backups, restore-to-differently-named
instances, and point-in-time recovery through the unified
BackupJob/RestoreJob/BackupClass/Plan interface (closes #3545).

The driver orchestrates the operator's PerconaServerMongoDBBackup /
PerconaServerMongoDBRestore CRDs:

- BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls
  status.state, and persists the S3 destination + storage descriptor
  (by-reference, no credentials) onto the Cozystack Backup artifact.
- RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back
  from the source backup status, snapshot fallback) so the same artifact
  restores in-place or into a differently-named instance; PITR via
  spec.options.pitr (date/latest).

Because psmdb resolves the backup storage by name against the cluster's
spec.backup.storages, the driver names the chart's storage (s3-storage) via the
cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear
Ready=False precondition when a target has backups disabled.

Wiring:
- api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy.
- internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp
  partial-schema mirrors; dispatch, supported-kinds and scheme registration.
- backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb
  Strategy CR, and psmdb.percona.com RBAC.
- apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no
  CA-bundle field) so backups can target Cozystack's self-signed seaweedfs.

Tests + e2e:
- unit tests for the driver (validation, ensure-idempotency, precondition,
  backupSource resolution, snapshot round-trip, PITR option parsing).
- examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that
  drives run-all.sh end-to-end (sentinel document round-trips through S3 into a
  restored copy while the source stays untouched), per docs/agents/e2e-testing.md.
- docs/operations/backup-classes.md updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Mattia Eleuteri f859b710a4 fix(backupstrategy-controller): repair the bucket credentials Secret from the gate, not a chart fail
Addresses the four findings on #3524.

**The `fail` in the bucket chart (MAJOR).** The <bucket>-<user>-credentials
Secret was made to `fail` the render while its COSI lookup was empty, so the
release would retry and converge instead of skipping the Secret permanently.
Helm cannot render a partial set, so that fail aborts the WHOLE
<bucket>-system release: the other users' Secrets, and the bucket UI
Deployment/Service/Ingress/HTTPRoute with them. One declared user whose
BucketAccess never provisions takes down every other user of that bucket,
and on an already-installed release parks it in Failed, blocking every later
upgrade.

**No per-bucket escape (MAJOR).** Rather than add one, remove the need for
it. The gate already forces the release that renders the lookup-gated
Strategy CRs; the credentials Secret is the same trap one release earlier,
and the gate reads that Secret to resolve the bucket name in the first
place. So while it is absent, or carries no bucket name, force the bucket's
<bucket>-system release instead of doing nothing. That cannot deadlock its
own precondition: the BucketClaim and the BucketAccess whose COSI Secret the
lookup reads are rendered unconditionally by the parent release, not by the
one being forced.

The two releases are throttled independently — they are forced in sequence
on a bootstrap, so a shared timestamp would delay the second by a full
MinForceInterval for no reason. Coordinates come from the chart under
provisionBucket, and are omitted on external S3 where the Secret is
admin-managed and no release renders it. No RBAC delta: patch on
helmreleases is already cluster-scoped.

The chart goes back to a partial render, with the comment now explaining why
the skip is not self-healing and why failing is the wrong lever, and a test
that renders the UI with every user unresolvable so a reintroduced `fail`
fails the suite. Tenant buckets keep their pre-existing behaviour;
generalising the repair to every bucket needs a controller that owns them
and is tracked separately.

**Stale gauge (MINOR).** cozystack_backup_default_objects_missing was only
written on the happy path, so the state the gate exists to catch reported 0
and the documented alert never fired. It is now written on the unresolved
path too — the credentials Secret counts as one of the objects the default
backups depend on, which is what it always was. On an API error the gauge is
deliberately left alone rather than flapping, so a new
cozystack_backup_default_objects_check_errors_total marks it stale, and the
doc pairs the two instead of overstating the gauge.

**Suspended releases (MINOR).** forceHelmRelease now does a point Get and
skips a release with spec.suspend: true. helm-controller ignores forceAt and
requestedAt while suspended (`cozyhr suspend` sets exactly that), so the gate
was re-stamping every MinForceInterval for the whole suspension and climbing
the force counter — which the runbook attributes to a render that is not
producing the objects, the wrong diagnosis. The skip is logged and not
counted, so a climbing counter keeps its documented meaning.

Reported-by: IvanHunters <xorokhotnikov@gmail.com>
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-By: Claude <noreply@anthropic.com>
2026-08-07 11:10:41 +02:00
Mattia Eleuteri 51ad226706 fix(backupstrategy-controller): repair lookup-gated backup objects
The default Strategy CRs and the Velero BackupStorageLocation are
Helm-templated behind a `lookup` of the BucketClaim the same chart
creates, and the <bucket>-<user>-credentials Secret behind a `lookup` of
the COSI Secret. When those lookups are empty at install time the objects
were silently skipped, and helm-controller does not re-render a release
whose chart and values did not change (drift detection is off on
operator-generated HelmReleases), so the skip was permanent: clusters ran
for months with only BackupClass cozy-default and no Strategy CRs at all.
That also fail-closes the pre-adoption snapshot in the v1.6.0 etcd
migration, which reads the projected credentials Secret.

Add a DefaultObjectsGate runnable to backupstrategy-controller. Once the
bucket name is resolvable from the projector's source Secret, it checks
that every object cozy-default routes to exists and forces one real Helm
upgrade (reconcile.fluxcd.io/forceAt + requestedAt) when any is missing.
Helm remains the objects' only author; the gate only makes sure the render
that produces them actually happens. It cannot be a render-time `fail` in
this chart: the chart is the producer of the Bucket its own lookup reads,
so a failed render would deadlock the condition.

The bucket chart's user-credentials template CAN fail safely — the
BucketAccess that produces the COSI Secret belongs to the parent release
and that release retries forever — so make it fail loudly instead of
skipping, with requireUserCredentials=false as the offline-render escape.

Also correct the chart and doc comments that claimed Flux re-renders on
its interval, and document the two-release manual recovery for clusters
already affected.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-08-03 17:16:55 +02:00
Andrey Kolkov 8aeedde088 feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383)
## What this PR does

Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported,
tested capability for the new `backups.cozystack.io` API. Fixes #2774.

Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the
`recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime`
plumbing; #3313 has since merged and this branch is rebased on `main`.
This PR adds the reliability + verification layer #2774 asked for.

### Changes

- **Precise failure reason for an unreachable target.** The restore
deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the
sole authority for "this restore is stuck" — the client waits the full
window either way. What changes is the *diagnosis*: once the deadline
has elapsed, if `recoveryTime` was set and the recovery Cluster never
went healthy, the CNPG driver reads the `full-recovery` pod logs and —
when several distinct attempts exited with PostgreSQL's `FATAL: recovery
ended before configured recovery target was reached` — fails the
RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason
`RecoveryTargetUnreachable`) and an actionable message, instead of a
bare generic timeout. Timing is unchanged; only the reason/message
differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.)
Design points:
- **We deliberately do NOT trip early off the FATAL.** A valid near-now
target emits that same FATAL transiently — repeatedly, for many
attempts, on a slow archiver — before it converges, so any earlier trip
would wrongly reject a recoverable restore (observed twice in CI). The
FATAL is used only to *explain* a failure the deadline has already
declared.
- **Health wins unconditionally** — the deadline/classification check
precedes the healthy check, but a recovery that converges within the
window reaches success and is never classified as failed (a live dev7
restore did exactly this).
- **Counting distinct FATAL-ing attempts** (not pod failures) tolerates
a one-off archive-lag miss on a near-now target and ignores transient
recovery-pod crashes (a node blip / brief API-server unreachability
never emits that FATAL).
- Adds the `pods/log` RBAC grant the log read needs (distinct
subresource from `pods`); a `Forbidden` is logged loudly and surfaced in
the failure message rather than silently disabling the classification.
- **e2e** (`examples/backups/postgres/run-all.sh`, wired into the
Chainsaw `postgres-2-backup-roundtrip` suite):
- Step 45 — write `before`, capture server timestamp (µs precision),
write `after`, restore to that timestamp, assert `before` present and
`after` absent.
- Step 46 — a `recoveryTime` an hour ahead must fail with reason
`RecoveryTargetUnreachable` once the deadline elapses (asserts the
*reason*, so it proves the classification fired, not just that a timeout
occurred).
- Chainsaw budget raised 25m → 40m to cover three sequential restores +
the post-deadline classification.
- **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a
`docs/operations/backup-classes.md` PITR section (recoverable window,
discovering earliest/latest restorable time from the backup catalog
since `firstRecoverabilityPoint` is unreliable under the plugin,
archive-lag tolerance, idempotency under GitOps).

### Verification

- `go build` / `go vet` clean; unit tests pass
(`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`,
`recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level
success-path test asserting `Succeeded` + `RecoveryConverged=True`, and
— added per review —
`TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`,
a reconcile-level test driving the deadline branch through all three
sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL
absent → generic timeout, `Forbidden` log read → generic timeout +
missing-`pods/log`-RBAC note. It fails if the call-site reason is
reverted to the generic one).
- Reviewed via adversarial branch-review; fixes include a critical
missing-RBAC dead-on-arrival, a classification false-positive on
transient recovery-pod crashes, and a sub-second e2e flake.
- **dev7**: backup + WAL archiving + recovery convergence verified live
on the #3313 build. The load-bearing assumption of the classification —
CNPG **accumulates** failed `full-recovery` pods rather than reusing one
— was confirmed (observed 7 simultaneous failed recovery pods). A
fully-green live step-46 run was blocked by intermittent COSI
bucket-provisioning flakiness on dev7; the negative path is covered by
the CI Chainsaw e2e.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The only user-facing surface is the new
`docs/operations/backup-classes.md` PITR section; the website keeps a
hand-mirrored copy per docs version.

- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: PITR docs mirror warranted (not yet opened; same pending
mirror as #3313's backup-classes changes)
- [x] No other downstream repository is affected by this change

```release-note
feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added PostgreSQL point-in-time recovery (PITR) support via RFC3339
`recoveryTime`.
* Introduced a PITR RestoreJob example, plus helper variables and an
unreachable-target scenario.
* **Bug Fixes**
* Improved restore deadline handling with deterministic, log-based
fail-fast classification for unreachable recovery targets, including
clearer conditions.
* **Documentation**
* Added detailed PITR operational documentation (recovery windows,
failure behavior, and GitOps/idempotent behavior).
* Expanded README and example comments for PITR verification and failure
expectations.
* **Tests**
* Extended unit tests and end-to-end chainsaw workflow to cover PITR
success and unreachable failures.
* **Chores**
* Updated RBAC to allow pod log reads; refined cleanup and run scripts
for additional RestoreJob artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-23 14:14:58 +04:00
Myasnikov Daniil a302bd0ddc docs(backupstrategy-controller): reframe the migrations-image reuse as a workaround
Running the platform migrations image as the ClickHouse backup client is
not a design decision worth defending, and the in-tree comments read as
though it were. Reframe them.

The strategy script uses exactly curl, jq, sleep and date. For that it
pulls a 349 MB image carrying kubectl, helm, git, cozyhr, the
etcd-migrate binary and every migration script into a tenant-adjacent
Pod, and that image's ENTRYPOINT is run-migrations.sh -- so the
`command:` override in the strategy template is load-bearing rather than
incidental. A dedicated alpine + curl + jq + ca-certificates image
measures 23 MB.

The standing objection to a dedicated image -- "a second tag release CI
would have to re-digest each cut" -- does not survive contact with the
actual defect. Every other first-party image already pays that cost
automatically. What broke was a pin with NO producer, which an image
owned by its own package cannot reproduce; that fix would also delete
the cross-package stamp introduced here.

Comment-only. No rendered output changes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-22 12:29:15 +05:00
Andrey KolkovandClaude Opus 4.8 6a1e2d0d06 feat(backups): support PostgreSQL point-in-time recovery (PITR)
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested
capability for the new backups.cozystack.io API (issue #2774). The
recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing
already existed on the barman-cloud-plugin branch; this adds the reliability
and verification layer the issue asked for:

- Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set
  and the recovery Cluster is not healthy, the driver reads the CNPG
  full-recovery pod logs and, once several distinct attempts have exited with
  PostgreSQL's "recovery ended before configured recovery target was reached"
  FATAL, fails the RestoreJob with conditions RecoveryConverged=False /
  Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the
  restore deadline. Health is checked first, so a converged recovery always
  wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a
  transient archive-lag miss on a near-now target and ignores transient
  recovery-pod crashes. Adds the pods/log RBAC grant the log read requires.

- e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw
  postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures
  the server timestamp, writes an after marker, restores to that timestamp and
  asserts the before row survived and the after row did not; step 46 submits a
  recoveryTime an hour ahead and asserts it fails fast with reason
  RecoveryTargetUnreachable.

- Working, uncommented example (45-restorejob-pitr.yaml) and a docs section
  (docs/operations/backup-classes.md) covering the recoverable window, how to
  discover the earliest/latest restorable time from the backup catalog, the
  archive-lag tolerance, and idempotency under GitOps.

Refs #2774

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-22 10:40:18 +04:00
Andrey KolkovandClaude Opus 4.8 2212367842 fix(backups): own driver-created ObjectStore + correct stale barman references
Review follow-ups (cozystack#3300):

- Lifecycle/leak: applyClusterPluginBackup now sets an ownerReference from the
  driver-created barmancloud.cnpg.io/ObjectStore to its cnpg.io/Cluster, so
  Kubernetes garbage-collects it when the Cluster is deleted (tenant app
  teardown or the RestoreJob purge). In the platform useSystemBucket flow the
  chart does not render this ObjectStore, so previously it was orphaned in the
  tenant namespace. Drop the now-accurate-only 'delete' verb on objectstores
  from the driver RBAC (GC handles removal; the driver never deletes it) and
  correct the RBAC comment that falsely claimed the purge deletes it.
- Diagnostics/comments: the RestoreJob WAL-archive-deadline failure message and
  several doc comments still told operators to check spec.backup.barmanObjectStore
  / externalClusters[].barmanObjectStore, fields no longer emitted after the
  migration. Point them at the barman-cloud plugin (spec.plugins + ObjectStore,
  externalClusters[].plugin) instead so they don't mislead during an incident.
- Test: TestApplyClusterPluginBackup_PatchesExistingCluster now asserts the
  ObjectStore carries an ownerReference to the Cluster (kind/name/uid).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00
Andrey Kolkov 13ff340277 fix(backups): migrate CNPG backups to the barman-cloud plugin
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00
Myasnikov DaniilandClaude Opus 4.8 4af5a82796 fix(etcd-operator,backups): address review nits — trim endpoint + hook resources
- backupstrategy-controller: trim() the decoded S3 endpoint before scheme
  stripping so a trailing newline in the Secret can't yield a malformed URL
- etcd-operator: give the pre-upgrade selector-fix hook container resource
  requests/limits (matches postgres-operator webhook-ready), +unittest

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-15 19:11:36 +05:00
Myasnikov DaniilandClaude Opus 4.8 e160a0dc87 fix(backups): derive default S3 endpoint from the provisioned bucket
The default backupStorage.endpoint is http://seaweedfs-s3...svc:8333, but
Cozystack ships SeaweedFS with global.seaweedfs.enableSecurity=true, so its
in-cluster S3 serves TLS on :8333 behind the self-signed "SeaweedFS CA".
Every cozy-default backup (etcd/mariadb/velero/fdb/cnpg) therefore hits a TLS
listener over plaintext and fails the handshake. This is fatal for the etcd
v1alpha2 adoption migration: its mandatory pre-upgrade safety snapshot writes
to this endpoint, so migration 50 hard-fails and blocks the 1.5->1.6 upgrade
on any cluster with a legacy etcd. The Etcd Strategy S3 schema has no
caCert/insecureSkipVerify field, so it cannot target the self-signed
in-cluster endpoint at all.

Add a "backupstrategy-controller.endpoint" helper that, for a provisioned
bucket, resolves the endpoint from the COSI bucket's system credentials
Secret (backupStorage.systemSecretName) — the external S3 ingress with an
ACME cert, the same trusted endpoint COSI advertises and every backup
operator can verify — and forces the https:// scheme. All Strategy CRs, the
Velero BackupStorageLocation and the controller Deployment env are routed
through it. It falls back to .Values.backupStorage.endpoint for external S3
(provisionBucket=false) and for offline `helm template`/unit renders and the
pre-reconcile first install, where the Secret lookup returns nothing (Flux
re-renders on spec.interval once the Secret exists).

helm-unittest extended to assert every consumer picks up the resolved
endpoint (CNPG/Etcd/Velero keep the full URL, MariaDB/FDB strip the scheme
and derive their secure flag) plus the external-S3 verbatim fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-15 19:11:36 +05:00
Andrey Kolkov cc5d688e15 feat(platform): migrate to etcd-operator v1alpha2
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
Andrey Kolkov f17ead3d26 feat(platform): add default backupclass
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-28 13:49:59 +04:00
Andrey Kolkov d80f67e799 feat(platform): add implementation of backup-strategy controller for etcd
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-22 14:35:08 +04:00
Andrey Kolkov 92930ad1aa feat(platform): add implementation of backup-strategy controller for foundationdb
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-13 09:49:19 +04:00