Files
cozystack/docs/operations/backup-classes.md
T
Aleksei Sviridkin 9b56e70ca6 feat(mongodb): support the system-bucket backup flow (useSystemBucket) (#3967)
## What this PR does

MongoDB was the only `cozy-default` app with **no `useSystemBucket`
flow**: its backup storage was always static on the
`PerconaServerMongoDB` CR (bucket/endpoint/credentials from `backup.*`
values plus a chart-emitted `<release>-s3-creds` Secret), so a tenant
could not back up to the platform system bucket out of the box the way
Postgres/ClickHouse/MariaDB/etcd do. This closes that gap so the default
MongoDB backup-restore works OOTB.

PBM takes `bucket`/`endpointUrl`/`region`/`prefix` as **static CR
fields** (not secret refs), and the platform bucket name/endpoint are
only known at BackupJob time — so, unlike ClickHouse's runtime
`secretKeyRef` env, the chart cannot render them. The implementation
mirrors the CNPG driver's pattern:

- **Chart** (`packages/apps/mongodb`): a new `backup.useSystemBucket`
flag keeps `spec.backup.enabled` (so the pbm agents run) but omits the
static `storages`/`tasks`/`pitr` and the `<release>-s3-creds` Secret.
Helm never owns `spec.backup.storages`, so a Flux re-render can't fight
the driver. A `bootstrap.enabled + useSystemBucket` render guard mirrors
Postgres, and `useSystemBucket: true` with `backup.enabled: false` is
refused both at render time and by an `x-kubernetes-validations` rule on
`backup` in the schema (the same mechanism as the `storageClass`
immutability rule). Note on the schema rule: the aggregated apiserver
does not evaluate CEL on application-backed resources yet (#2657), so
today it is a UI/schema-level marker and the render-time refusal is what
stops a Flux release; it also trips the API review gate (“validation
rule added”), so this PR needs an API-owner approval on top of the code
review.
- **Strategy** (`cozy-default-mongodb`): now carries the S3 coordinates
(`endpointUrl` already scheme'd by the
`backupstrategy-controller.endpoint` helper, `credentialsSecret:
cozy-backups-creds`, `prefix: <namespace>/<application>`).
- **Driver** (`psmdbstrategy_controller.go`): SSA-injects
`spec.backup.storages[s3-storage]` onto the live `PerconaServerMongoDB`
at BackupJob time under a dedicated field manager
(`cozystack-psmdb-backup-driver`) with `ForceOwnership`. Injection is
**gated on the app's `backup.useSystemBucket` flag** (a legacy app
leaves it false and is never touched; a useSystemBucket app is
re-injected once per mint (once per BackupJob, plus once per
storage-race re-mint), before its operator CR is created, so a later
coordinate change catches up on the next job while a streaming dump is
never patched under). The backup CR is minted only once the operator
reports the storage pushed into pbm (`status.observedGeneration` caught
up and `PBMReady` True): pbm writes to what its own config says, not to
the spec, so a CR minted before that push would land at the previous
coordinates while its Backup recorded the new ones; the job waits at
`PerconaServerMongoDBPBMConfigPending` meanwhile, bounded by the backup
deadline. Once a job has its CR, none of the mint-time reads or holds
can fail it: the strategy and app reads the artifact still needs are
bounded by the 24h running ceiling instead, and the cluster is not read
at all **and on the release having rendered that flag**: the app CR is
projected from the HelmRelease's desired values, so the flag reads true
before helm-controller renders the revision, while the tenant's own
storage, scheduled tasks and PITR stream from the previous render still
sit on the cluster. As long as the cluster carries `spec.backup.tasks`
or `spec.backup.pitr`, the driver does not inject and holds the
BackupJob at `Ready=False PerconaServerMongoDBLegacyRender` (failing it
at the deadline), so a wedged legacy revision never has its nightly dump
and oplog redirected into the shared bucket. The backup snapshot falls
back to the strategy coordinates when the operator status echo is empty;
that fallback is gated on the CR's own `percona.com/delete-backup`
finalizer (the flow the CR was minted on), not on the app flag re-read
at completion, so a legacy backup never records the platform bucket and
a flag flipped mid-dump does not change the artifact. The restore path
rebuilds `backupSource` from it (the `cozy-backups-creds` reference is
projected into the restore namespace before dispatch).
- **Retention/cleanup**: on the `useSystemBucket` flow the driver owns
the shared-bucket archive (the chart's psmdb `tasks[].keep` retention is
gated out, and an on-demand backup is not covered by task retention). It
stamps the operator's `percona.com/delete-backup` finalizer on each
backup it mints **against a storage that carries the platform
coordinates** — the strategy's bucket and `credentialsSecret`, read back
from the live cluster after the injection — never on the flag alone: a
strategy without an `s3` block, or a storage that points anywhere else,
holds the job (`MongoDBStrategyHasNoS3` /
`PerconaServerMongoDBStorageForeign`) instead of minting a finalized
dump against a bucket the platform does not own. On Cozystack `Backup`
deletion (by hand today; by platform retention once it lands, since a
`Plan` only creates BackupJobs and never deletes a `Backup`) the driver
deletes the operator `PerconaServerMongoDBBackup` CR so that finalizer
frees the object — the finalizer's presence is the ownership marker
cleanup reads back. That reading is unambiguous because cleanup resolves
the CR by the name recorded in `driverMetadata` (the one the driver
minted), not by scanning: the psmdb operator does stamp
`percona.com/delete-backup` on its own `cron-*` task-minted backups, but
those are never in scope. A legacy backup keeps the pre-existing
hands-off contract, decided from the snapshot before any read of the
operator CR. A terminating namespace or the
`backups.cozystack.io/skip-artifact-cleanup: "true"` annotation strips
the driver's finalizer and releases; a `Get` of the operator CR that
fails with anything other than NotFound/NoMatchError still requeues, and
the operator's own `release-lock` finalizer can hold the CR while the
operator is down, so those two are what can keep a namespace in
`Terminating`. `schedule` and `retentionPolicy` are inert on this flow
(there is no engine-side retention and a `Plan` has no retention field
yet); platform-level retention is tracked in
[#4192](https://github.com/cozystack/cozystack/pull/4192) and teardown
GC of orphaned backups in
[#3966](https://github.com/cozystack/cozystack/issues/3966), so until
those land bound `cozy-backups` with an S3 lifecycle policy.
- **Bounds on the waits**: the injection race with the operator's
cluster cache (an `unable to get storage` error on a freshly injected
storage) is retried by deleting the errored CR and minting a fresh one,
with the retry count kept on the BackupJob
(`backups.cozystack.io/mongodb-storage-race-retries`) because the
deleted CR is reaped before the next poll; after three retries the
operator's error is terminal. A system-bucket dump that has started
streaming is not cancelled on the 30-minute deadline, but fails after 24
hours in `running` (`BackupAbandoned` warning naming the operator CR,
which is left in place as the handle to whatever pbm still writes). The
deadline cancel of a not-yet-started CR pins its delete to the
resourceVersion the live read answered for.
- **RBAC**: `patch` on `perconaservermongodbs` for the injection
(server-side apply), and `create`/`delete`/`patch` on
`perconaservermongodbbackups` — `delete` to prune the shared-bucket
archive on `Backup` deletion, `patch` to strip the driver's finalizer
when releasing a wedged cleanup.

Coverage: Go unit tests over `internal/backupcontroller` — the injection
gate keyed off the app flag (legacy cluster never patched) and off the
applied render (a cluster still carrying tasks or pitr is held,
untouched), the SSA-injection precondition with the whole entry
(per-application prefix included) asserted on the cluster, strategy
templating per application, the finalizer refused against a storage
without platform coordinates (strategy without `s3` over a legacy entry,
foreign coordinates on the merged entry), the retry budget driven with
the operator simulated reaping the deleted CR between polls, the restore
snapshot fallback and credential swap across all four legacy/system flow
combinations, the finalizer-owned cleanup with its escape hatches (skip
annotation before any read, `NoMatchError` treated as gone,
terminating-namespace release using the pre-read CR, a failed strip
surfaced as an Event, a legacy CR never deleted, a legacy snapshot
released without a read, a system-bucket snapshot reaching the prune),
the deadline cancelling a not-yet-running operator backup only on a
positive live read (including no live client at all, and a CR that moved
since the read), and the running ceiling. Plus helm-unittest on both
charts (chart skips the creds Secret and omits
`spec.backup.storages`/`tasks`/`pitr` on `useSystemBucket` and keeps
them on the legacy flow; the strategy renders the S3 coordinates and the
`insecureSkipTLSVerify` knob). A live `useSystemBucket` backup+restore
e2e against the platform bucket is a follow-up (needs the system bucket
wired in CI, same status as the Postgres/ClickHouse useSystemBucket
e2e). One item to confirm in e2e: the psmdb operator accepting
`spec.backup.enabled=true` with an empty `storages` between install and
the first BackupJob.

### Screenshots

Not applicable (no UI changes).

### Downstream repositories

The tenant-facing schema surface is one new field,
`spec.backup.useSystemBucket`, on the MongoDB `values.schema.json` / CRD
(regenerated), plus a validation rule on `spec.backup`.
`terraform-provider-cozystack` hand-models MongoDB and may want to
expose it, but it is a new optional boolean that defaults false (no
drift for existing users), and the chart `README.md` regenerates on
release. Leaving every box unticked per the trigger-map guidance so a
maintainer decides on the provider follow-up.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(mongodb): MongoDB backups now support the platform system bucket via backup.useSystemBucket, so the default cozy-default BackupClass works without per-app S3 credentials — the backup driver injects the system-bucket storage onto the cluster at backup time, once the release has rendered the flag. On this flow nothing prunes the shared bucket automatically yet (a Plan carries no retention field; tracked in #4192 and #3966), so an S3 lifecycle policy on cozy-backups is a prerequisite for enabling it, not a suggestion. useSystemBucket requires backup.enabled=true; the chart refuses the combination with backups disabled at render time, and the schema carries the same rule as a marker until the apiserver enforces CEL (#2657). A system-bucket dump still running after 24 hours fails its BackupJob with the operator backup left in place. A MongoDB restore now adopts the target cluster's S3 credentials only for a storage on the same bucket as the archive (it used to adopt them unconditionally); a legacy restore into a target on a different bucket that relied on one key pair covering both must keep or re-create the source release's -s3-creds Secret by name, and is held at Ready=False RestoreCredentialsMissing until it exists.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added opt-in system bucket mode for MongoDB backups.
* Automatically injects platform bucket storage without
application-managed S3 credentials.
* Supports endpoint, region, path prefix, addressing, and TLS settings.
* Manages shared-bucket backup artifacts when system bucket backups are
deleted.

* **Bug Fixes**
* Improved restore credential selection to avoid cross-bucket
mismatches.
  * Preserved backup metadata when storage details are unavailable.

* **Documentation**
* Documented configuration, scheduling, retention, PITR limitations, and
restore guidance.

* **Validation**
* Prevented incompatible bootstrap and system bucket restore
configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 18:10:56 +03:00

440 lines
71 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Backup Classes
Cozystack ships a single platform-managed `BackupClass` named `cozy-default`. It is provisioned automatically when the `backupstrategy-controller` package is installed and references the system-managed bucket provisioned through the `apps.cozystack.io/Bucket` CR `cozy-backups` in the `tenant-root` namespace (the real S3 bucket name is the COSI-assigned one from `BucketClaim.status.bucketName`).
Tenants reference `cozy-default` from `BackupJob`, `Plan`, and `RestoreJob` resources — they do **not** supply S3 credentials, endpoints, or paths. The platform projects the system-managed credentials Secret into the tenant namespace per BackupJob (or, for long-lived references like Velero's `BackupStorageLocation`, into a fixed list of system namespaces on a periodic tick), and the default strategy templates encode `<namespace>/<application>` into every S3 path so two tenants with the same application name never collide.
## Supported applications
### Bound by `cozy-default` (work out-of-the-box)
| Application Kind | Driver | Strategy CR |
|----------------------------------|--------------------------------------|----------------------------------------------------------------------------|
| `apps.cozystack.io/Postgres` | CloudNativePG (barman) | `strategy.backups.cozystack.io/CNPG` `cozy-default-cnpg` |
| `apps.cozystack.io/MariaDB` | mariadb-operator dump | `strategy.backups.cozystack.io/MariaDB` `cozy-default-mariadb` |
| `apps.cozystack.io/ClickHouse` | Altinity `clickhouse-backup` sidecar | `strategy.backups.cozystack.io/Altinity` `cozy-default-altinity` |
| `apps.cozystack.io/MongoDB` | Percona psmdb operator (pbm) dump | `strategy.backups.cozystack.io/MongoDB` `cozy-default-mongodb` |
| `apps.cozystack.io/Kafka` | Kafka Admin API (topic metadata) | `strategy.backups.cozystack.io/Kafka` `cozy-default-kafka` |
| `apps.cozystack.io/Etcd` | etcd-operator snapshot | `strategy.backups.cozystack.io/Etcd` `cozy-default-etcd` |
| `apps.cozystack.io/RabbitMQ` | RabbitMQ definitions (management API) | `strategy.backups.cozystack.io/Rabbitmq` `cozy-default-rabbitmq` |
| `apps.cozystack.io/Redis` | RDB dump Job (sentinel-discovered master) | `strategy.backups.cozystack.io/Redis` `cozy-default-redis` |
| `apps.cozystack.io/VMInstance` | Velero + kubevirt-velero-plugin | `strategy.backups.cozystack.io/Velero` `cozy-default-velero-vminstance` |
| `apps.cozystack.io/VMDisk` | Velero | `strategy.backups.cozystack.io/Velero` `cozy-default-velero-vmdisk` |
### Shipped but NOT bound (admin opt-in required)
| Application Kind | Driver | Strategy CR |
|----------------------------------|--------------------------------------|----------------------------------------------------------------------------|
| `apps.cozystack.io/FoundationDB` | FoundationDB operator backup_agent | `strategy.backups.cozystack.io/FoundationDB` `cozy-default-foundationdb` |
The FoundationDB strategy CR is rendered by the chart so admins can reference it from a custom BackupClass once the operator-side plumbing (mounting `cozy-backups-creds` into the `cozy-foundationdb-operator` Deployment) is wired manually. See "FoundationDB caveat" below.
### Endpoint format per driver
Different operators expect different endpoint shapes; the strategy templates rendered by `backupstrategy-controller` resolve one S3 endpoint (via the `backupstrategy-controller.endpoint` helper) and adapt it to each consumer's contract. For a provisioned bucket (`provisionBucket: true`, the default) the endpoint is **derived from the COSI bucket's system-credentials Secret** (`backupStorage.systemSecretName`) and forced to `https://` — the external S3 ingress with an ACME cert, the only endpoint the backup operators can verify. SeaweedFS's in-cluster S3 serves TLS on `:8333` behind the self-signed "SeaweedFS CA", and the Etcd Strategy S3 schema has no `caCert` field, so the in-cluster endpoint cannot be targeted directly. The `backupStorage.endpoint` chart value (a full URL like `http://seaweedfs-s3.tenant-root.svc:8333`) is the **fallback**, used for external S3 (`provisionBucket: false`) and for offline `helm template`/pre-reconcile renders where the Secret lookup returns nothing. The resolved endpoint is adapted per consumer:
| Driver | Strategy template field | Form |
|--------|-------------------------|------|
| CNPG (Postgres) | `barmanObjectStore.endpointURL` | full URL (scheme preserved) |
| Etcd | `destination.s3.endpoint` | full URL (scheme preserved) |
| MariaDB | `storage.s3.endpoint` | bare host:port (scheme stripped); `tls.enabled` derived from the scheme |
| MongoDB | `s3.endpointUrl` | full URL (scheme preserved); on the default flow the app's `backup.endpointURL`, on `useSystemBucket` the strategy coordinate injected by the driver |
| FoundationDB | `blobStoreConfiguration.accountName` + `urlParameters.secure_connection` | bare host:port + derived secure flag |
| Velero | `BackupStorageLocation.spec.config.s3Url` | full URL (scheme preserved) |
| ClickHouse sidecar | `S3_ENDPOINT` env | bare host:port (from projected Secret) |
| Redis (dump Job) | `S3_ENDPOINT` env | bare host:port (from projected Secret); `https://` prepended when unscheme'd |
| Kafka | `S3_ENDPOINT` env on the strategy Job | full URL (scheme preserved); the Job's `curl --aws-sigv4` prepends `https://` if the endpoint carries no scheme, honours `backupStorage.forcePathStyle`, and (for the image's curl 7.76.1) signs a portless URL with `--connect-to` to the real port so a ported endpoint still verifies |
The projected `cozy-backups-creds.endpoint` key is **stripped of scheme** so chart-emitted sidecars (ClickHouse) consume it directly. Drivers that need the full URL receive the resolved endpoint described above — derived from the COSI system Secret (forced `https://`) for a provisioned bucket, or the `backupStorage.endpoint` fallback for external S3.
VM-driven (Velero) backups land in the same `cozy-backups` bucket under the `velero/` prefix. A `BackupStorageLocation` named `cozy-default` is shipped by the `backupstrategy-controller` chart (`packages/system/backupstrategy-controller/templates/velero-bsl.yaml`) so endpoint/bucket/region come from the same `backupStorage` values block used by Strategy CRs and the projector.
### FoundationDB caveat
The strategy CR `cozy-default-foundationdb` is shipped, but it is **not** bound by `cozy-default` yet. Restore runs `fdbrestore` from inside the `cozy-foundationdb-operator` Deployment, which does not yet mount `cozy-backups-creds`. Until the operator deployment is updated to mount the projected Secret, FDB platform-default restore silently fails — admins who need it today should keep using a per-app `Bucket` plus a custom `BackupClass`, or wire the credentials file into the operator deployment themselves.
**Cleanup gotcha (zombie backup_agent).** Unlike CNPG/MariaDB/Altinity (one-shot operator-side Backup CRs), the FoundationDB driver creates an `apps.foundationdb.org/FoundationDBBackup` CR that drives a **long-lived** `backup_agent` Deployment streaming continuously to S3. Deleting a Cozystack `Backup` (e.g. via retention sweeping) does NOT stop that Deployment — the agent keeps writing until the next BackupJob's `stopOtherFoundationDBBackups` call swaps it out, until an admin invokes `examples/backups/foundationdb/cleanup.sh`, or until the operator-side CR is deleted by hand. If a tenant deletes their last Cozystack Backup and never submits another BackupJob, the agent pods will continue running indefinitely and accumulate S3 PUTs. This is intentional today (the driver has no RBAC verb to stop the operator-side CR on Cozystack-Backup deletion) but admins should be aware of it.
## Postgres and MariaDB: a restored copy converges its application passwords shortly after recovery
Managed Postgres no longer accepts a plaintext `users[].password`; every application-user password is chart-generated into `<release>-credentials` and read only from there. The CNPG backup snapshot therefore carries no passwords — deliberate, since a snapshot that did was the one place a tenant password sat in cleartext.
**Residue from before the upgrade — this is forward-looking, not retroactive.** The removal scrubs new renders and new snapshots; it does not rewrite copies made earlier, and upgrading preserves the live password (the chart reads the existing `<release>-credentials` through `lookup`), so those older copies still expose the *current* credential. Two sinks retain it: a Postgres `Backup` object created before this release snapshotted `app.spec.users` while that spec still carried a plaintext `password`, so its `status.underlyingResources` holds that password in cleartext — a field any tenant with `get backups` in the namespace can read (see the `underlyingResources` caution below); and every pre-upgrade Helm revision of `<release>-init-script` rendered `ALTER ROLE … WITH PASSWORD '<literal>'` into its manifest, retained for `MaxHistory` revisions. There is no in-band rotation to invalidate the exposed value (chart-based rotation was intentionally not added — a chart-rendered Secret cannot revoke a leaked credential; rotation is being reworked as a controller, cozystack/community#72). So after upgrading, treat any pre-upgrade Postgres backup as credential-bearing: delete or re-take the pre-upgrade `Backup` objects, and be aware that the old init-script Secret persists in Helm history until those revisions age out of `MaxHistory`.
The consequence for a **restore into a copy** (a differently-named target): the recovered database keeps the source roles with their source password hashes, but the target chart generates fresh random passwords into the target's `<release>-credentials`, and the role-reconciling init-job is skipped while `bootstrap.enabled` is set on the restored app. The restore driver clears `bootstrap.enabled` automatically once recovery is healthy, so the next HelmRelease reconcile runs the init-job, which applies `ALTER ROLE … WITH PASSWORD` for every `spec.users` entry and converges the recovered roles onto the target Secret. Convergence is therefore automatic but not instant: between the RestoreJob reaching `Succeeded` and that init-job completing, the passwords advertised in `<target>-credentials` do not yet match the recovered roles, so a client reading that Secret cannot log in as an application user for that brief window. The chart flags the Secret `postgres.cozystack.io/credentials-pending: "true"` and the NOTES print the same caution while bootstrap is still on, so a reader does not treat the advertised password as usable mid-restore; both clear on their own once the driver clears bootstrap. The data is intact and reachable as the CNPG superuser throughout. After a to-copy restore, wait for the application login to succeed rather than assuming the advertised password works the instant the RestoreJob succeeds. The driver clears `bootstrap.enabled` by writing it onto the Postgres app's own spec, so a GitOps source (Argo CD, Flux, Terraform) that pins `bootstrap.enabled: true` for that app will revert the clear on its next apply and the copy's credentials never converge — exclude the restored app from such a source until convergence completes, or clear `bootstrap.enabled` in the source itself. If the driver cannot clear the flag at all (the app was deleted or renamed mid-restore), the RestoreJob fails with reason `BootstrapDisableFailed` once its deadline elapses rather than sitting `Running` forever with no signal. An in-place restore is unaffected: the release keeps its existing `<release>-credentials`, whose passwords already match the roles the recovery brings back. One-time migration: an app restored to a copy under an earlier release — before this auto-clear existed — may still be sitting with `bootstrap.enabled` set and its init-job permanently skipped; the driver clears the flag only for a restore it drives to completion, never retroactively for one that already finished, so such an app keeps advertising unconverged credentials until you clear `bootstrap.enabled` on it once by hand.
Databases and users after a Postgres restore, in place or into a copy: the driver sets the app's `spec.databases` and `spec.users` to the snapshot taken when the Backup was created, while recovery replays WAL past that point (to the end of the archive when no `recoveryTime` is set), so the recovered cluster can hold databases and users the snapshot does not list. On the first init-job run after the restore, a database the chart manages but the restored spec does not declare is released instead of dropped: its data stays, and it and its `<db>_admin` and `<db>_readonly` roles lose the `managed by helm` comment, so the chart no longer grants on it or drops it. Add it to `databases` to manage it again. Users are not released: a recovered user the restored spec does not declare, including a source user the target of a copy does not declare, is dropped on that run and every object it owned is reassigned to `postgres`, so a client still logging in as that user loses the login. Declare the user in `users` to keep it; the chart then generates a new password for it into `<release>-credentials`.
MariaDB avoids the same problem at backup time: the driver sets `ignoreGlobalPriv` on every operator `Backup` it creates, so `mysql.global_priv` — the grant table holding every user's password hash, root included — is excluded from the logical dump and a restore into a copy never overwrites the target's chart-managed users. The same exclusion is also a limitation, and it applies to every restore, not only a to-copy one: because the grant table is never in the dump, only accounts Cozystack declares through `User` CRs (the chart's `users` map) are recreated on restore — an account created out of band, directly in MySQL rather than through `users`, is in no backup, so a restore brings back its data and its `mysql.db` grants but not the account that owns them, and that login fails until the account is recreated. Cozystack treats the `User` CRs as the source of truth for accounts precisely so the grant table need not be, so keep managed accounts in `users` rather than creating them by hand. That matters most for `root`, which the operator only applies at datadir bootstrap via `rootPasswordSecretKeyRef` and could never repair afterwards; keeping the source's grants out of the dump is what lets the target's advertised `root` password stay correct through a restore. This is a **backup-side** fix, so it only protects backups captured from this release onward. A restore from a backup taken **before** this change still replays the source's `mysql.global_priv` and overwrites the target's application users and `root` with the source's hashes; after restoring such an older backup, treat the target's advertised credentials as unreliable and reset them — application users converge on the next reconcile through their `User` CRs, but `root` must be reset out of band (`ALTER USER 'root' … IDENTIFIED BY …` against the restored server) since the operator applies `rootPasswordSecretKeyRef` only at datadir bootstrap. That once-only write is also why the Secret itself now matters: `root` lives solely in `<release>-credentials` with no plaintext escape hatch left in values, so if that Secret is lost after bootstrap — a tenant deletes it, or a partial Velero restore drops it — the advertised `root` password and the live server's diverge with no in-band way to reconcile them; back the Secret up alongside the data, and reset `root` out of band if it is ever lost.
## ClickHouse: opt-in to the system bucket
The `clickhouse-backup` sidecar runs inside the ClickHouse Pod itself, so the Helm chart is what wires its S3 credentials. Existing tenants on the legacy `backup.s3*` values continue to work unchanged. To switch a release onto the platform bucket, set:
```yaml
backup:
enabled: true
useSystemBucket: true
```
When `useSystemBucket: true`:
- The chart-emitted `<release>-backup-s3` Secret is no longer rendered.
- The sidecar consumes `cozy-backups-creds` (projected by the platform).
- `S3_PATH` is set to `<namespace>/<release>` so two tenants with the same ClickHouse release name never share a prefix.
`s3Region`, `s3Bucket`, `endpoint`, `s3AccessKey`, `s3SecretKey`, and `s3CredentialsSecret` are ignored in this mode.
## MongoDB: backup storage
MongoDB supports two storage flows. By default the S3 target lives on the app CR and the tenant supplies its own credentials; alternatively the tenant opts into the platform system bucket and the driver injects the storage onto the live cluster at backup time.
### Default: the application owns the backup storage
The Percona psmdb operator runs the percona-backup-mongodb (pbm) agents whenever the `PerconaServerMongoDB` cluster has `spec.backup.enabled: true` (a declared storage is not needed for the agents to start), and services a `PerconaServerMongoDBBackup` only once the storage it references by name is declared on the cluster. On the default flow the MongoDB application **opts into backups** and points at the bucket in its own chart values:
```yaml
apiVersion: apps.cozystack.io/v1alpha1
kind: MongoDB
metadata:
name: orders-db
spec:
backup:
enabled: true
destinationPath: "s3://<bucket>/orders-db/"
endpointURL: "https://seaweedfs-s3.tenant-root:8333"
insecureSkipTLSVerify: true # self-signed in-cluster seaweedfs; psmdb s3 storage has no CA-bundle field
s3AccessKey: "<key>"
s3SecretKey: "<secret>"
```
The chart declares that storage as `s3-storage`, which the `cozy-default-mongodb` strategy names (`storageName: s3-storage`, `type: logical`). The driver then drives on-demand backups and restores through the unified `BackupJob` / `RestoreJob` / `Plan` interface on top of the operator's storage config — adding restore-to-differently-named instances (via `PerconaServerMongoDBRestore.spec.backupSource`) and point-in-time recovery that the raw scheduled-task / `mongodump` paths do not offer. When a target cluster has backups disabled, the driver surfaces a clear `Ready=False` precondition on the BackupJob/RestoreJob rather than hanging until the deadline.
For a to-copy restore the operator reads the dump from the **source** backup's bucket/endpoint. The driver re-points the restore at the **target** cluster's own S3 credentials only when the target declares a storage on that same bucket (the common case where both share one bucket), since the source release's Secret dies with it in a DR scenario; a system-bucket source keeps its projected `cozy-backups-creds` reference, which outlives the source app. A target whose storages are all on a different bucket gets no credential swap: the restore keeps the source's Secret reference and is held at `Ready=False RestoreCredentialsMissing` naming that Secret until it exists in the restore namespace, so a cross-bucket restore never runs with a credential for the wrong bucket. Re-create the source release's `-s3-creds` Secret by name to let it proceed.
A full, scripted example (write a marker document, back up, restore to a copy, assert the round-trip while the source stays untouched) is in [`examples/backups/mongodb/`](../../examples/backups/mongodb/) — driven by `run-all.sh`.
### Opt-in to the system bucket
Unlike ClickHouse's sidecar, PBM takes the bucket/endpoint/prefix as static fields on the `PerconaServerMongoDB` CR, and the platform bucket name is only known at BackupJob time — so the chart cannot render them. Instead the `cozy-default-mongodb` driver injects the storage onto the live cluster. To back up to the platform bucket without supplying S3 credentials, set:
```yaml
backup:
enabled: true
useSystemBucket: true
```
When `useSystemBucket: true`:
- The chart-emitted `<release>-s3-creds` Secret is no longer rendered, and the chart leaves `spec.backup.storages`, `spec.backup.tasks` and `spec.backup.pitr` unset on the `PerconaServerMongoDB`.
- On every BackupJob the driver SSA-injects the `s3-storage` entry from the strategy coordinates (bucket/endpoint from the platform system bucket, `credentialsSecret: cozy-backups-creds`, prefix `<namespace>/<application>`) under its own field manager, so a Flux re-render never reverts it and a later change to the coordinates is picked up on the next backup rather than frozen at the first. The backup itself is not started until the operator reports that it has pushed that `spec.backup` into pbm's own configuration (`status.observedGeneration` caught up with the cluster's generation and the `PBMReady` condition `True`): pbm writes to whatever its config last said, not to the spec, so a backup started before that push would land at the previous coordinates while its `Backup` recorded the new ones. The job waits at `Ready=False PerconaServerMongoDBPBMConfigPending` meanwhile, and fails at the backup deadline if the operator never gets there.
- The injection waits for the release to actually render the flag. The app's `backup.useSystemBucket` is a desired value (it reads `true` as soon as it is written, before helm-controller renders the revision), while the tenant's own storage, scheduled tasks and PITR stream from the previous render stay on the `PerconaServerMongoDB` until it does. As long as the cluster still carries `spec.backup.tasks` or `spec.backup.pitr`, the driver does not inject and holds the BackupJob at `Ready=False` (`PerconaServerMongoDBLegacyRender`), failing it at the backup deadline; a release wedged on an old revision therefore never has its nightly dump and oplog redirected into the shared bucket. `useSystemBucket: true` together with `enabled: false` is refused by the chart at render time; the schema carries the same rule as an `x-kubernetes-validations` marker, which the aggregated apiserver does not evaluate yet ([#2657](https://github.com/cozystack/cozystack/issues/2657)), so until then the render-time refusal is what stops such a release, and a release wedged on it shows up as a failed HelmRelease rather than a redirected backup.
`destinationPath`, `endpointURL`, `insecureSkipTLSVerify`, `s3AccessKey` and `s3SecretKey` are ignored in this mode.
Because the chart drops `spec.backup.tasks` and `spec.backup.pitr`, the psmdb operator's own scheduled backups and oplog PITR do **not** run on this flow — `backup.schedule` and `backup.retentionPolicy` stay in the schema for the default flow but have no effect here, and the `recoveryTime` PITR restores described below are not available. Migrate scheduled backups to a `backups.cozystack.io/Plan` against `cozy-default` instead. **Flipping this flag on a running app** removes the storage, tasks and pitr the app had, so a release that was taking scheduled backups + PITR before the flip stops until a `Plan` takes over.
Dropping `spec.backup.tasks` also drops the psmdb `keep` retention it carried, so on this flow **the driver owns the archive** the way the Rabbitmq/Redis drivers own theirs (an on-demand `PerconaServerMongoDBBackup` is not covered by psmdb task retention). The driver stamps the operator's `percona.com/delete-backup` finalizer onto each backup it mints against a storage that carries the platform coordinates (the strategy's bucket and `credentialsSecret`, as read back from the live cluster after the injection; a strategy without an `s3` block, or a storage pointing anywhere else, holds the job at `Ready=False` instead of minting an unowned dump), and deleting a Cozystack `Backup` (by hand today; by platform retention once it lands) deletes that `PerconaServerMongoDBBackup` CR — whose finalizer removes the pbm object from the shared bucket before the `Backup` is released, so deleting `Backup` objects is what reclaims `cozy-backups`. A legacy backup writes to the tenant's own bucket and keeps the pre-existing hands-off contract (its lifecycle stays the tenant's / operator's; nothing is deleted on `Backup` removal). Note this prunes only when a `Backup` is deleted: unlike CNPG, whose shipped strategy sets a `retentionPolicy` the engine trims on, MongoDB has no engine-side scheduled trim on this flow, so keeping `cozy-backups` bounded depends on a `Plan`'s retention actually deleting old `Backup` objects. **A `Plan` has no retention field yet** (`PlanSpec` carries only `applicationRef`, `backupClassName` and `schedule`), so today a nightly `useSystemBucket` Plan grows the shared bucket unbounded — `schedule` and `retentionPolicy` on the app are inert here (see [Opt-in to the system bucket](#opt-in-to-the-system-bucket)), and platform-level retention is tracked in the [`BackupRetentionPolicy` design (#4192)](https://github.com/cozystack/cozystack/pull/4192). Teardown GC of orphaned `Backup`s / S3 data is [#3966](https://github.com/cozystack/cozystack/issues/3966). Until those land, bound `cozy-backups` with an S3 lifecycle policy on the object store. Two cases release the `Backup` without waiting for the object to be pruned, so a wedged prune never pins a namespace in `Terminating`: a terminating namespace (where the operator's delete-backup step cannot run) and the `backups.cozystack.io/skip-artifact-cleanup: "true"` annotation. In both the driver strips its own finalizer and leaves the object for a bucket lifecycle policy to reclaim. A down operator or unreachable storage in a live namespace is NOT released automatically — the `Backup` stays `Terminating` as a visible signal until the operator recovers or you set that annotation to force the release.
A dump that has started streaming on this flow is not cancelled on the 30-minute backup deadline (a real dataset routinely outruns it, and deleting the CR mid-stream would prune a partial), but it is not unbounded either: after 24 hours in `running` the `BackupJob` fails with a `BackupAbandoned` warning naming the `PerconaServerMongoDBBackup`, which is left in place. If that dump later completes, its archive is reachable only through that CR; delete the CR to prune it.
**Opting back out** (`useSystemBucket` true→false) is not automatic in reverse: the driver stops injecting, but the `s3-storage` entry it already applied stays on the live `PerconaServerMongoDB` under the driver's own field manager (`cozystack-psmdb-backup-driver`) — the chart does not own it and so never prunes it. It is inert once `backup.enabled=false`; to fully return a release to the legacy flow, delete that storage entry from the `PerconaServerMongoDB` by hand before re-declaring the tenant's own `backup.*` S3 values. Until it is removed, a new `BackupJob` on that release is refused rather than written into the platform bucket unowned: the driver recognises the injected entry by its `credentialsSecret` and holds the job at `Ready=False` (`PerconaServerMongoDBStorageStale`) naming that manual step, failing it at the backup deadline. A backup already streaming when the flag was flipped is unaffected — its flow is fixed by the finalizer on its own operator CR.
### Point-in-time recovery (MongoDB)
psmdb records an oplog stream between logical backups. A MongoDB `RestoreJob` recovers to a timestamp via `spec.options.recoveryTime` — the same option name and RFC3339 format the Postgres/CNPG driver uses, so the two are uniform:
```yaml
spec:
options:
recoveryTime: "2026-08-05T12:34:56Z" # RFC3339 (UTC), same as the CNPG driver
```
The driver converts `recoveryTime` to the psmdb oplog target internally. Unlike CNPG (whose empty `recoveryTime` replays WAL to the latest archived point), an empty `recoveryTime` here restores the backup snapshot as taken — a psmdb logical backup is already a consistent point, so "restore this backup" is the safe default. `spec.options.restoreTimeoutSeconds` caps how long the driver waits for the operator restore before failing (default 30m), matching the CNPG option. Any unrecognised key under `spec.options` (e.g. a `recoverytime` typo) is ignored but surfaced as a `UnknownRestoreOption` Warning event on the RestoreJob rather than silently dropped.
## Inspecting the defaults
```bash
kubectl get backupclasses
kubectl get backupclass cozy-default -o yaml
kubectl -n tenant-root get bucket cozy-backups
kubectl -n tenant-root get secret bucket-cozy-backups-system-credentials
kubectl -n cozy-velero get backupstoragelocation cozy-default
```
The bucket lives in `tenant-root` and is provisioned through the `apps.cozystack.io/Bucket` CR. The system-managed credentials Secret never leaves that namespace. The backupstrategy-controller projects a copy under the name `cozy-backups-creds` into a tenant namespace right before each BackupJob or RestoreJob runs, and refreshes the same Secret in `cozy-velero` (and any other namespace listed in `backupStorage.systemNamespaces`) on a 1-minute tick. The projected Secret carries multiple key formats so each driver finds what it needs in one place:
| Key | Consumer |
|-----------------------------------------------|-------------------------------------------|
| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | CNPG, MariaDB, Etcd, RabbitMQ, Redis |
| `accessKey` / `secretKey` (plus `bucketName`, `endpoint`, `region`) | ClickHouse sidecar |
| `cloud` | Velero (AWS credentials file format) |
| `blob_credentials.json` | FoundationDB backup_agent |
The Redis dump Job additionally reads the `endpoint`, `bucketName`, and `region` keys (alongside the `AWS_*` pair) via `secretKeyRef`, the same set the ClickHouse sidecar consumes.
The dump Job also speaks to the Redis app's own Sentinel and master, which is a separate TLS axis from the S3 endpoint above. When the `Redis` app has `tls.enabled`, the operator moves both to a TLS-only listener and publishes the CA at `redis-<app>.ca-cert`; the Job mounts that Secret optionally and connects with `--tls --cacert` exactly when its `ca.crt` is present, so a plaintext (non-TLS) app is unaffected. This covers the default `tls.authClients: no`. Mutual TLS (`authClients: yes`) is not supported for backup — it requires a client certificate signed by the release CA, which the platform does not issue.
### Bootstrap window
On a fresh-cluster install, the Velero `BackupStorageLocation` `cozy-default` is rendered before the credentials projector has had a chance to copy `cozy-backups-creds` into `cozy-velero`. The BSL reports `Unavailable` until the projector's first synchronous round completes (which runs as soon as the `backupstrategy-controller` acquires leadership — in practice moments after the Pod becomes Ready, typically tens of seconds after `helm install` returns, not minutes). Velero rejects new `Backup` AND `Restore` requests against `storageLocation: cozy-default` during that window. Plan VM backup automation accordingly, or wait for the BSL to become ready before submitting backups: `kubectl -n cozy-velero wait backupstoragelocation cozy-default --for=jsonpath='{.status.phase}'=Available --timeout=5m`.
**Note on controller restarts.** The BSL flickers `Unavailable` on every `backupstrategy-controller` pod restart while the projector replays its first synchronous round. The window is short (single-digit seconds) but operators who alert on BSL availability should suppress alerts during the controller's `kube_pod_container_status_restarts_total{container=backupstrategy-controller}` events or use a longer evaluation window than the projector tick (60s).
### Cozy-default Bucket bootstrap
`cozy-default` ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name (`bucket-<claim-UID>`, so it cannot be computed in advance) and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster.
**That skip does not repair itself on a reconcile.** helm-controller re-renders a release only when its chart or values change; the interval reconcile is a no-op for a healthy release, and drift detection is off on operator-generated HelmReleases. A cluster that lost the race at install time therefore keeps `BackupClass cozy-default` with **no** `Strategy` CRs and **no** Velero BSL indefinitely — which also fail-closes the pre-adoption snapshot in the v1.6.0 etcd migration.
The same trap sits one release earlier, on the Secret everything else depends on. `bucket-cozy-backups-system-credentials` is rendered by the `bucket-cozy-backups-system` release behind a `lookup` of the COSI Secret, and is skipped just as permanently when that lookup is empty. Without it the credentials projector has no source at all, so no strategy, no Velero and no v1.6.0 etcd migration can resolve — and the gate cannot even determine the bucket name, which it reads from that Secret.
Convergence is driven instead by the controller's default-objects gate (`backupStorage.reconcileDefaultObjects`, on by default), which repairs both, in order. While the credentials Secret is absent or carries no bucket name, the gate stamps `reconcile.fluxcd.io/forceAt` + `requestedAt` on the **bucket's** `bucket-<name>-system` HelmRelease. Once the bucket name resolves it checks that every object `cozy-default` routes to exists and stamps the same pair on the **`backupstrategy-controller`** HelmRelease, forcing the real Helm upgrade that re-runs the lookups. The two are throttled independently (one force per 5 minutes each), so the second is not delayed by the first. Expect the objects within a minute or two of the bucket becoming ready.
The gate does not force a suspended HelmRelease — helm-controller ignores both annotations while `spec.suspend` is true, so it logs the skip and leaves the force counter alone. A release left suspended (`cozyhr suspend`) is therefore never repaired until it is resumed.
Watch it with:
```bash
kubectl -n cozy-backup-controller logs deploy/backupstrategy-controller | grep default-objects-gate
```
#### Manual recovery on an affected cluster
Only needed on a cluster running a version without the gate (or with `reconcileDefaultObjects: false`). A plain `flux reconcile helmrelease` does **nothing** here — it does not re-render. You need a forced upgrade, and **both** annotations: `forceAt` is what makes helm-controller run a real Helm upgrade, and it is only honoured together with `requestedAt`.
First confirm the bucket name is actually resolvable — forcing before that just re-runs the same empty lookup:
```bash
kubectl -n tenant-root get bucketclaim bucket-cozy-backups -o jsonpath='{.status.bucketName}'
```
Then force the two releases, **in this order**:
```bash
# 1. The credentials Secret. It is rendered by the -system release, not by
# bucket-cozy-backups — easy to miss, and the projector (hence every
# strategy and Velero) has no source without it.
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
kubectl -n tenant-root annotate helmrelease bucket-cozy-backups-system \
reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite
kubectl -n tenant-root get secret bucket-cozy-backups-system-credentials
# 2. The Strategy CRs and the Velero BSL.
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
kubectl -n cozy-backup-controller annotate helmrelease backupstrategy-controller \
reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite
kubectl get $(kubectl get crd -o name | grep strategy.backups.cozystack.io) 2>/dev/null
kubectl -n cozy-velero get backupstoragelocation cozy-default
```
The race is nested: step 2's `lookup` resolves from the BucketClaim status, but the projector — and the v1.6.0 etcd migration — read the Secret from step 1, so a cluster missing both needs both.
### Observability
The credentials projector emits two Prometheus counters labelled by `namespace` (and `reason` for failures):
- `cozystack_backup_credentials_projection_successes_total`
- `cozystack_backup_credentials_projection_failures_total`
Alert on `rate(cozystack_backup_credentials_projection_failures_total[5m]) > 0` or `absent_over_time(cozystack_backup_credentials_projection_successes_total[10m])` to catch a stale BSL credential or a malformed source Secret without log scraping.
The default-objects gate emits three more:
- `cozystack_backup_default_objects_missing{backupclass="cozy-default"}` — how many of the objects the platform default backups depend on are absent: the credentials Secret, the Strategy CRs `cozy-default` routes to, and the Velero BSL. **This is the alert that would have caught the missing Strategy CRs**: it is non-zero whatever the HelmRelease's `Ready` condition says. Alert on `min_over_time(cozystack_backup_default_objects_missing[15m]) > 0`.
- `cozystack_backup_default_objects_check_errors_total{backupclass="cozy-default"}` — checks that could not reach a conclusion (an API error reading the source Secret, the BackupClass, or one of the routed objects). The gauge above is deliberately **not** written on those ticks, so that it does not flap on a transient API error — which means that while this counter climbs, the gauge is stale and a `0` on it proves nothing. Pair the two: `min_over_time(cozystack_backup_default_objects_missing[15m]) > 0 or rate(cozystack_backup_default_objects_check_errors_total[15m]) > 0`.
- `cozystack_backup_default_objects_force_reconciles_total{namespace,name}` — forced Helm upgrades issued, labelled by the release forced (this chart's own, or the bucket's `-system` release). A counter that keeps climbing means the forced render is not producing the objects (a missing CRD, for instance), which is a different problem from the install-time race. A suspended release is skipped before the patch and is **not** counted here, so a paused release cannot masquerade as a render that keeps failing — look for the `skipped forcing a suspended HelmRelease` log line instead.
## Admin overrides for `cozy-default`
`cozy-default` is rendered by the `backupstrategy-controller` chart and owned by Flux's helm-controller. **Direct `kubectl edit backupclass cozy-default` is overwritten on the next helm reconcile** — the same applies to its companion `strategy.backups.cozystack.io/*` CRs (`cozy-default-cnpg`, `cozy-default-etcd`, `cozy-default-mariadb`, `cozy-default-altinity`, `cozy-default-mongodb`, `cozy-default-foundationdb`, `cozy-default-rabbitmq`, `cozy-default-redis`, the two `cozy-default-velero-*`). The supported override path is the `backupStorage` block on the **`platform` component** of the `cozystack.cozystack-platform` Package CR:
```yaml
apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
name: cozystack.cozystack-platform
spec:
components:
platform:
values:
backupStorage:
provisionBucket: true # default; set false for external S3
bucketName: cozy-backups # apps.cozystack.io/Bucket release name
endpoint: http://seaweedfs-s3.tenant-root.svc.cozy.local:8333
region: us-east-1
forcePathStyle: true
systemSecretName: bucket-cozy-backups-system-credentials
systemNamespaces:
- cozy-velero
```
The platform chart forwards this block into the child `Package cozystack.backupstrategy-controller` as `components.backupstrategy-controller.values.backupStorage` (`packages/core/platform/templates/bundles/system.yaml`), from where the cozystack operator merges it into the `backupstrategy-controller` HelmRelease over the chart defaults. Two paths that look plausible do **not** work: `spec.components.backupstrategy-controller` on the `cozystack.cozystack-platform` Package is silently ignored (the only component under that PackageSource is `platform`), and patching the child `Package cozystack.backupstrategy-controller` directly is reverted whenever the platform helm-reconcile re-renders it.
A sibling `backupStrategyController` block on the same `platform` component is forwarded the same way, for the controller's own knobs rather than the bucket. The one an operator reaches for is `redisBackupResources`, which sizes the Redis strategy Pod — restore loads the whole dataset into a throwaway loader, so a large Redis needs more memory / ephemeral-storage than the defaults (the loader-not-ready log line points here):
```yaml
spec:
components:
platform:
values:
backupStrategyController:
redisBackupResources:
limits:
memory: 8Gi
ephemeral-storage: 32Gi
```
| Knob | Effect |
|---|---|
| `provisionBucket` | Toggle creation of the in-cluster `apps.cozystack.io/Bucket` CR. Set `false` for external S3 (see [Disabling the platform-managed bucket](#disabling-the-platform-managed-bucket)). |
| `bucketName` | Two modes. With `provisionBucket: true` (default): K8s name of the Bucket CR + lookup key for the COSI BucketClaim — the actual S3 bucket name is the COSI-assigned UUID, surfaced through `BucketClaim.status.bucketName`. With `provisionBucket: false`: taken **verbatim as the real S3 bucket name** and baked into every strategy CR + the Velero BSL. |
| `namespace` | Namespace the Bucket CR (and its system-credentials Secret) lives in — `tenant-root` by default. Must be a tenant namespace (`tenant-*`): the Bucket chart's RBAC helper fails the Helm render for any other prefix. |
| `bucketNameOverride` | Escape hatch for offline `helm template` renders — bypasses the live-cluster BucketClaim lookup. Leave empty in production. |
| `endpoint` | **Fallback** S3 endpoint. For a provisioned bucket the strategy CRs + Velero BSL derive the endpoint from the COSI system Secret (external ACME ingress, forced `https://`) instead; this value is used only for external S3 (`provisionBucket: false`) and offline renders. For external S3, switching it to `https://` enables TLS in the MariaDB/FoundationDB strategies, which derive TLS from the scheme. The Redis dump Job is not scheme-driven: the projector delivers a bare host and the script prepends `https://` unconditionally, so the Job always connects over TLS — point `endpointCASecretName` (below) at the private CA rather than relying on the scheme, and ensure that CA is reachable to the relevant Pods. |
| `endpointCASecretName` | Optional Secret (key `ca.crt`) in the app namespace the Redis Job trusts for a self-signed S3 endpoint. Empty by default: the projected endpoint is always `https://` and the platform bucket's ACME cert verifies against the image's system CA store unaided. Set it only for a private CA. The Job mounts it optionally (nothing projects this Secret automatically), so a name typo does not wedge the Pod on `FailedMount`: a missing `ca.crt` falls through to the system CA store and the Job fails fast at the TLS handshake with a legible error — it does not skip verification, which still needs the explicit `insecureSkipTLSVerify` opt-in. |
| `insecureSkipTLSVerify` | Disables S3 certificate verification for the Redis Job (`curl -k`). `false` by default and an explicit opt-in, never a fallback — an untrusted-cert endpoint fails closed unless this is set. Prefer `endpointCASecretName`. |
| `region` | Re-projected into `cozy-backups-creds` on the next reconcile. Pod-restart required for chart-emitted clients consuming the region via env (ClickHouse sidecar today). |
| `forcePathStyle` | Path-style addressing; SeaweedFS S3 requires it, AWS S3 typically doesn't. |
| `systemSecretName` | Name of the human-friendly Secret produced by the Bucket app (or pre-created manually for external S3). The projector also accepts the raw COSI Secret format. |
| `systemNamespaces` | Namespaces where the controller eagerly projects `cozy-backups-creds` (Velero BSL, FDB operator). Tenants are projected lazily during BackupJob reconcile. |
When the override needs to go beyond storage coordinates — different retention, different driver→Kind binding, multi-region split — create a **sibling BackupClass** with a unique name (anything but `cozy-default`). Sibling BackupClasses live outside the chart, are admin-owned, and Flux will not touch them. Tenants opt in by setting `backupClassName: <your-class>` on their `BackupJob`s.
## Tuning via a custom BackupClass
The defaults aim at a reasonable middle (gzip compression where applicable; engine-side retention only where the strategy carries it, which today is the CNPG `retentionPolicy: "30d"` and the two Velero `ttl: 720h`; every other shipped strategy, MongoDB included, sets none, so its archives on the shared bucket are bounded only by an S3 lifecycle policy or a manual `Backup` delete: a `Plan` only creates BackupJobs and never deletes a `Backup`). To override for a specific tenant or workload, create your own `BackupClass` pointing at the same strategy CRs but with tweaked `parameters`, or a fresh strategy CR. Common knobs:
- **CNPG strategy**: `barmanObjectStore.retentionPolicy`, `data.compression`, `wal.compression`.
- **MariaDB strategy**: `compression`, `maxRetention`, `databases[]`.
- **MongoDB strategy**: `storageName` (which `spec.backup.storages` entry on the psmdb cluster to use), `type` (`logical`), `compressionType` / `compressionLevel`, and — for the `useSystemBucket` flow — the `s3` coordinates the driver injects. On the default flow the S3 target is tuned via `backup.*` values on the MongoDB release instead (see [MongoDB: backup storage](#mongodb-backup-storage)).
- **Altinity strategy**: tune the `clickhouse-backup` sidecar via `backup.*` values on the ClickHouse release; the strategy Pod is a thin HTTP client. When the S3 endpoint's certificate is signed by a private CA rather than a publicly-trusted one — SeaweedFS's in-cluster `:8333` being the case in point — point `backup.endpointCA` at a Secret holding that CA bundle; the chart mounts it into the sidecar and adds it to the trust store via `SSL_CERT_DIR`, which supplements the system CA set rather than replacing it.
- **FoundationDB strategy**: `snapshotPeriodSeconds`, `agentCount`, `urlParameters[]`.
- **Rabbitmq strategy**: `artifactURITemplate` (the `<namespace>/<application>/<backup-name>/definitions.json` object-key layout) and the pod `template` (image, resources). The backup is a definitions export over the management API — vhosts, users, permissions, queues, exchanges, bindings, policies, parameters — so message payloads are out of scope and there is no point-in-time recovery; a full message-data backup is a Velero volume snapshot instead. **Restore is a merge, not a reset**: importing definitions (`POST /api/definitions`) creates or updates what the export contains and never deletes, so anything created after the backup survives the restore and the broker is not returned to its exact backup-time state. A to-copy restore imports the source's users (with password hashes) into the target, so those source credentials become valid logins there. Unlike the operator-backed strategies (whose engine owns archive retention), the Rabbitmq driver owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not normally accumulate objects. The delete is **best-effort**: when it cannot run from here the Backup is released without deleting the object (with a `Warning` Event naming what was left behind), so the Backup — and any namespace being torn down — never wedges. The named give-up conditions are: the namespace is terminating (`CREATE` is forbidden there), the strategy CR or the projected credentials are unavailable, the rendered Job name is invalid, or the skip annotation below is set. Otherwise a genuinely failing delete (object storage unreachable) keeps the `Backup` in `Terminating` (a visible signal to act on) and retries. To release such a stuck `Backup`, annotate it `backups.cozystack.io/skip-artifact-cleanup: "true"` — cleanup then skips the delete and lets the `Backup` go, leaving the object in the bucket to be reclaimed manually or by a bucket lifecycle policy (`kubectl annotate backup <name> -n <namespace> backups.cozystack.io/skip-artifact-cleanup=true`).
- **Velero strategy (VMInstance / VMDisk)**: `ttl`, `includedResources[]`, `excludedResources[]`.
- **Etcd strategy**: today the strategy is path-only, and a `Plan` carries no retention field yet (see [#4192](https://github.com/cozystack/cozystack/pull/4192)), so nothing trims etcd snapshots on the shared bucket automatically either.
The system-managed credentials Secret is the **only** way for in-cluster strategies to reach `cozy-backups`. Do not embed access keys in `BackupClass.parameters` — the security model relies on Secret references, and `parameters` end up in `Backup.status.underlyingResources`, which tenants can read.
## Disabling the platform-managed bucket
If a deployment runs against an external S3 (no SeaweedFS), set `backupStorage.provisionBucket: false` via the `platform` component override described above and create the source credentials Secret in `tenant-root` manually (flat-key format: `accessKey` / `secretKey` / `endpoint` / `bucketName`; or the raw COSI `BucketInfo` JSON). In the same `backupStorage` block, update `endpoint`, `region`, **and `bucketName`**: with `provisionBucket: false` the strategies and the Velero BSL take `bucketName` verbatim as the real S3 bucket name (no COSI lookup), so it must name the actual bucket on the external S3 — the `bucketName` key inside the Secret alone is not enough. The Velero `BackupStorageLocation` picks the same values up automatically (the chart renders it from the same `backupStorage` block), so no separate BSL configuration is needed. Note that disabling the cluster-default BSL itself (the chart's `velero.bslEnabled` value) is **not** carried by the `backupStorage` override path — the platform Package forwards only the `backupStorage` block.
## Upgrade notes from chart-managed backups
> **Postgres backups now use the CloudNativePG Barman Cloud plugin, not native `spec.backup.barmanObjectStore`.**
>
> Postgres backups were migrated off the deprecated native `barmanObjectStore` (removed in CNPG 1.29) onto the [Barman Cloud plugin](https://github.com/cloudnative-pg/plugin-barman-cloud). The `barman-cloud-*` binaries the native path needs are absent from the `standard` image variant `keycloak-db` pins; `apps/postgres` pins bare (system-flavor) tags that still ship them — its breakage was the operator/CRD skew below, not missing binaries — but the system flavor is deprecated upstream, so both move to the plugin. The chart now renders `spec.plugins` on the `cnpg.io/Cluster` plus a `barmancloud.cnpg.io/ObjectStore` CR that carries the S3 configuration, and the CNPG backup driver SSA-applies an `ObjectStore` and patches `spec.plugins` (instead of `spec.backup.barmanObjectStore`) for the platform flow; `Backup`/`ScheduledBackup` use `method: plugin`. On `helm upgrade` a legacy chart-managed Cluster's `spec.backup.barmanObjectStore` is replaced by `spec.plugins` + an `ObjectStore`. This only works once the `plugin-barman-cloud` operator is installed and co-located with the CNPG operator in `cozy-postgres-operator` (shipped by the `postgres-operator` chart) and its cert-manager dependency is present — otherwise the Cluster references a plugin that is not registered and backups / WAL archiving stall.
>
> **Mixed-version window: BackupJobs fail until the app re-renders with the new chart.** A `BackupJob` that fires while a `Cluster` still carries the pre-upgrade chart's native `spec.backup.barmanObjectStore` is rejected by the CNPG admission webhook (`Cannot enable a WAL archiver plugin when barmanObjectStore is configured`) — the driver cannot attach `spec.plugins` alongside the native config. This resolves itself once Flux re-renders the Postgres app with the new chart (which replaces `barmanObjectStore` with `spec.plugins`); re-run the BackupJob afterward. Upgrading only the operators/driver without the app charts leaves every `backup.enabled: true` Postgres in this state permanently.
>
> **Postgres `backup.enabled: true` with placeholder credentials no longer wires backups on upgrade.**
>
> The pre-v1.5 defaults for `backup.s3AccessKey` / `backup.s3SecretKey` in `packages/apps/postgres/values.yaml` were the literal `"<your-access-key>"` / `"<your-secret-key>"` placeholders, so the Postgres chart still rendered a backup block on the `cnpg.io/Cluster` (with junk credentials, WAL archiving failing at runtime). Starting with v1.5 those defaults are empty strings and the chart NO LONGER renders any backup wiring when the placeholders are unmodified. Tenants on the legacy chart-managed flow who relied on those placeholders see `spec.plugins` and the `ObjectStore` disappear from the live `Cluster` on `helm upgrade`. Action — pick one:
>
> - **Move to the platform flow (recommended).** Set `backup.useSystemBucket: true`; the chart leaves `spec.plugins` unset and the CNPG backup driver SSA-applies the `ObjectStore` and patches `spec.plugins` onto the live `Cluster` at first BackupJob time. No tenant-side keys required.
> - **Stay on the legacy chart-managed flow.** Supply real `backup.s3AccessKey` / `backup.s3SecretKey` (or a pre-existing `backup.s3CredentialsSecret.name`); the chart renders `spec.plugins` + the `ObjectStore` from those coordinates.
>
> **Two platform-level changes ride along with this migration (they are required for the plugin to run):**
>
> - **CloudNativePG operator bumped to 1.28.1** (`postgres-operator` chart). A minor operator upgrade for the whole fleet; the CNPG CRDs move with it. This also fixes a latent bug where the operator image was newer than its bundled CRDs (`status.instanceID.sessionID` pruned), which made the operator believe the instance manager restarted and fail **every** backup cluster-wide with `instance manager was restarted during backup`. The vendored chart ships the CNPG CRDs as ordinary release manifests (gated by `crds.create`), so a normal `helm upgrade` updates them together with the operator — the `helm.sh/resource-policy: keep` annotation only prevents their deletion, not updates. The historical skew came from the `image.tag` pin outrunning the vendored chart's CRDs, which this PR removes; no out-of-band CRD apply is needed unless the HelmRelease itself is held back.
> - **LINSTOR scheduler admission webhook must not strip native-sidecar fields.** Older `linstor-scheduler` extender builds (≤ v0.3.5's predecessors) decoded every pod through a stale vendored Kubernetes API and re-encoded it, stripping `initContainers[].restartPolicy` cluster-wide — which breaks the plugin's restartable barman sidecar ("startupProbe forbidden without restartPolicy=Always"). The platform already ships the fixed webhook (`linstor-scheduler` re-vendored to upstream chart 0.3.1 / extender v0.3.6, which stopped stripping unknown pod fields), so no action is needed on an up-to-date platform; just do not hold the `linstor-scheduler` package back on an older version when rolling out the plugin.
>
> The same `useSystemBucket` opt-in applies to ClickHouse — see [ClickHouse: opt-in to the system bucket](#clickhouse-opt-in-to-the-system-bucket). When `useSystemBucket: true` is set on ClickHouse, the legacy `<release>-backup` CronJob, credential Secret, and backup script are no longer rendered (they are mutually exclusive with the platform flow); migrate scheduled backups to a `backups.cozystack.io/Plan` against `cozy-default`.
## Tenant workflow
Tenants only ever see the BackupClass name. Typical apply:
```yaml
apiVersion: backups.cozystack.io/v1alpha1
kind: BackupJob
metadata:
name: ad-hoc
namespace: tenant-acme
spec:
backupClassName: cozy-default
applicationRef:
apiGroup: apps.cozystack.io
kind: Postgres
name: orders-db
```
## Point-in-time recovery (PostgreSQL)
A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). The CNPG barman-cloud plugin restores a base backup and replays archived WAL from it: up to `recoveryTime` when one is set, so the restored cluster reflects the database as of that instant and later writes are absent, and to the end of the archive otherwise.
The base backup a restore starts from is the one the `Backup` refers to: the driver passes its barman backup ID to CNPG as `bootstrap.recovery.recoveryTarget.backupID`. There are two exceptions, and for both the choice is left to the plugin. The first is a `recoveryTime` earlier than one second after that backup ended, which the backup cannot serve (recovery would have to stop before the backup is consistent); the plugin then takes the newest base backup that ended at or before `recoveryTime`. The second is a `Backup` whose `cnpg.io/Backup` no longer exists: retention (`barmanObjectStore.retentionPolicy`) deletes it together with the base backup it describes, so the archive can no longer start from that backup, and the `RestoreJob` records a `BaseBackupGone` warning event. A base backup removed from the bucket by anything else — a lifecycle rule, a manual delete — is still pinned for as long as its `cnpg.io/Backup` stays. The plugin deletes a `cnpg.io/Backup` whose base backup has left the catalog on its next maintenance pass (every `instanceSidecarConfiguration.retentionPolicyIntervalSeconds` of the `ObjectStore`, 30 minutes by default, retention policy or not), but only from the primary of the cluster that took it, running under the same UID. Until then a restore from it fails after the target has been purged, with `no backup found with ID` in the recovery log, and a retry fails the same way; deleting that `cnpg.io/Backup` by hand, which leaves the object store untouched, lets the next `RestoreJob` go through the plugin's choice. Left to itself for every restore, the plugin would start from the newest base backup in the archive whenever `recoveryTime` is empty, and from the newest one ending by `recoveryTime` otherwise, whichever `Backup` the `RestoreJob` names and on any timeline — see [Archives with several timelines](#archives-with-several-timelines) for why that matters.
```yaml
apiVersion: backups.cozystack.io/v1alpha1
kind: RestoreJob
metadata:
name: orders-db-pitr
namespace: tenant-acme
spec:
backupRef:
name: orders-db-adhoc # a completed Backup of the source
targetApplicationRef: # omit for a destructive in-place restore
apiGroup: apps.cozystack.io
kind: Postgres
name: orders-db-copy # a pre-deployed, empty Postgres app
options:
recoveryTime: "2026-07-21T09:30:00Z"
```
A full, scripted example (write a marker, capture the timestamp, restore to it, assert what survived) is in [`examples/backups/postgres/`](../../examples/backups/postgres/) — `45-restorejob-pitr.yaml`, driven by `run-all.sh`.
### The recoverable window
`recoveryTime` must fall inside the window the archive can reconstruct:
- **Earliest** — the completion of the oldest base backup still in the archive. You cannot recover to an instant before the first base backup: WAL replay always starts from a base backup, and retention (`barmanObjectStore.retentionPolicy`) eventually trims the oldest ones together with the WAL that predates them. Aim at least one second after a backup's `status.stoppedAt`: `stoppedAt` is truncated to the second, while the plugin compares `recoveryTime` with the end time barman records in `backup.info`, to the microsecond, and keeps only a backup that ended at or before it. A `recoveryTime` equal to `stoppedAt` therefore falls before that backup's actual end: an older backup is used instead, and when there is none the restore fails with `no target backup found`.
- **Latest** — the last transaction, commit or abort, contained in the WAL shipped to object storage. It is not the newest WAL segment: a time-targeted recovery ends just before the first transaction recorded after `recoveryTime`, and a segment switch, a checkpoint or the end of a base backup is not one. A `recoveryTime` at or after the last archived transaction leaves PostgreSQL nothing to stop on. On a busy database the last transaction trails "now" by the archive lag; on a database at rest it is its last write, however recent the WAL archive and however recent the latest base backup. Once the source is deleted, the window is frozen at the last transaction that made it to S3 before it went away.
A `recoveryTime` **at or after the last archived transaction cannot converge**: PostgreSQL replays every available WAL, never finds a transaction after the target, exits with `FATAL: recovery ended before configured recovery target was reached`, and CNPG re-creates the recovery instance in a loop. Such a wedged restore is failed by the restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) — and the driver reads the recovery pod's log at that point, so instead of a generic timeout the RestoreJob ends `status.phase: Failed` with conditions `RecoveryConverged=False` and `Ready=False`, both reason `RecoveryTargetUnreachable`, and a message naming the target. The driver deliberately does **not** fail earlier off that FATAL: a *reachable* near-now target hits the identical FATAL transiently — often repeatedly, on a slow archiver — before it converges, so an earlier trip would reject a recoverable restore. To reject an unreachable target quickly, set a short `restoreTimeoutSeconds`; otherwise pick a time before the last archived transaction. The state as of the last write needs no `recoveryTime` at all: omitting it replays the whole archive, and that is the state it recovers.
Restoring to a **very recent** instant is safe as long as the database keeps writing and WAL archiving is current: the segment covering the target, or the transaction after it, may not be in object storage yet and the same FATAL fires transiently, but because the driver only fails at the deadline (not on that FATAL), the recovery has the whole window to catch up — the next attempt promotes once the WAL ships and the cluster goes healthy, which completes the restore. Only a target that never becomes reachable within the deadline is failed: on a database that has stopped writing, that is every target after its last transaction. If archiving is badly behind (a stalled `archive_command`, an overloaded cluster) a near-now target can exceed even the default 30m window; restore to a point you can confirm is followed by an archived transaction (see the discovery section below).
A `recoveryTime` **before the earliest base backup** fails differently: PostgreSQL cannot begin replay before the base backup it restored, so recovery never reaches a consistent state and never emits the "recovery ended before …" FATAL the driver classifies on. That case also surfaces at the deadline, but with the generic reason `RestoreFailed` rather than `RecoveryTargetUnreachable`. Choosing a `recoveryTime` at least one second after the oldest base backup's `stoppedAt` (below) avoids it.
### Discovering the earliest / latest restorable time
CNPG's `Cluster.status.firstRecoverabilityPoint` exists in the status schema but is unreliable under the barman-cloud plugin — it was observed empty on every plugin-backed cluster on the dev7 test cluster (CNPG 1.28.1) — so read the window from the backup catalog instead. Each completed base backup records its WAL range and timestamps on the underlying `cnpg.io/Backup`:
```bash
# Completed base backups, oldest first: STOP plus one second is the earliest
# instant that backup alone can restore to; the oldest one bounds the window.
kubectl -n <ns> get backups.postgresql.cnpg.io \
--sort-by=.status.stoppedAt \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,ID:.status.backupId,START:.status.startedAt,STOP:.status.stoppedAt,BEGINWAL:.status.beginWal,ENDWAL:.status.endWal
# The cozystack Backup points at its underlying cnpg.io/Backup and S3 prefix.
kubectl -n <ns> get backup.backups.cozystack.io <name> -o jsonpath='{.spec.driverMetadata}'
```
The upper bound is the timestamp of the last `COMMIT` or `ABORT` record in the archived WAL, which nothing in the cluster status reports. `pg_waldump` prints it from the newest segments under `<destinationPath>/<serverName>/wals/`, working back from the newest segment until one holds such a record; on a database at rest that can be several segments back, because every segment switch and every base backup adds one with no transaction in it. Any `recoveryTime` before that timestamp is reachable; for the state as of that last transaction, omit `recoveryTime`.
### Archives with several timelines
A WAL archive can hold several PostgreSQL timelines under one `serverName`. Every failover starts one, and so did every in-place restore made before restored clusters archived to a fresh prefix (`bootstrap.newServerName`): those wrote the new timeline back into the prefix they had recovered from, often without its `.history` file, so the archive ends up with branches that are siblings rather than a single line of descent.
Recovery follows `recovery_target_timeline: latest`, which CNPG leaves as PostgreSQL's default: starting from the base backup's timeline, PostgreSQL looks for the history file of the next timeline number, then the next, stops at the first one missing, and targets the last one it found, replaying only the segments on that line of descent. Without any history file that is the base backup's own timeline alone. When the timeline it targets does not descend from the base backup — a sibling branch whose history file happens to carry the next number — recovery does not fall back to another timeline: it stops with `FATAL: requested timeline N is not a child of this server's history` (or `... is not in this server's history`). Two consequences:
- the last archived transaction that bounds a restore is the last one on the timeline that restore follows, not the newest one in the archive;
- a restore must start from a base backup on the branch it means to bring back. The plugin's own choice ignores timelines — CNPG passes no `targetTLI` — so the newest backup, or the newest one ending by `recoveryTime`, can lie on another branch and restore that branch's data instead. The driver pins the `Backup`'s own base backup with `recoveryTarget.backupID` for that reason; a `recoveryTime` earlier than that backup's end still goes through the plugin's time-based choice, and on such an archive can land on another branch.
### Idempotency under GitOps
An in-progress restore is safe to reconcile. The driver purges the target `Cluster` + PVCs exactly once per RestoreJob (guarded by the `TargetPurged` condition and a freshly-recovered check), suspends the target's HelmRelease across the purge so Flux cannot race the bootstrap swap, and resumes it once the recovery cluster is rendered. A Flux reconcile (or a controller restart) mid-restore therefore re-attaches to the recovering cluster rather than deleting it and starting over.
## Kafka: topic metadata only
The `cozy-default-kafka` strategy backs up **topic metadata**, not message data. Its Job talks to the Kafka Admin API (`kafka-topics --describe` on backup, `kafka-topics --create` + `kafka-configs --alter` on restore) and stores one small object per run at `s3://<bucket>/<namespace>/<application>/<backup-name>/kafka-metadata.txt`. It captures every non-internal topic's partition count, replication factor and non-default configs; message payloads, consumer-group offsets, ACLs, quotas and `KafkaUser`s are out of scope, and there is no point-in-time recovery. The driver gates each run on the Strimzi `Kafka` cluster reporting `Ready` and never mutates it.
This is the "reconstruct a cluster's topic topology" flow: an in-place restore recreates dropped topics into the source, and a to-copy restore applies the source's topics onto a freshly-bootstrapped empty `Kafka` (partitions and configs preserved). Restore is additive at the topic-set level — it never deletes topics that exist live but not in the backup — but it does not silently accept a divergent existing topic: it grows the partition count when the backup asks for more, and fails loudly (naming the topic and both values) when the backup asks for fewer partitions or a different replication factor, since Kafka cannot shrink partitions or change RF in place and a restore that cannot reach the recorded state must not report `Succeeded`. Restored topics are created directly through the Admin API, so on the target they are **unmanaged** (no `KafkaTopic` CR); that is the accepted tradeoff of an Admin-API-only driver. Restoring a topic whose replication factor exceeds the target cluster's broker count fails at `--create`. Topic configs are restored **additively** (`kafka-configs --add-config`): a config the backup recorded is re-applied, but a config added to a live topic after the backup is not removed, so an in-place restore does not return a topic's config set to its exact backup-time state (partition count and replication factor are the only shape the restore reconciles or fails loudly on).
One consequence of the Admin-API approach: if a topic is declared in the Kafka app's `spec.topics`, the Strimzi Topic Operator owns it and reconciles its config from the `KafkaTopic` CR, reverting any dynamic config an in-place restore sets that the CR does not list. So this strategy is durable for **out-of-band** topics (created directly, no CR); for CR-declared topics the `KafkaTopic` CR / GitOps is the source of truth. A to-copy restore onto a fresh empty cluster is unaffected. For message-data durability, use a volume-snapshot strategy instead. See `examples/backups/kafka-metadata/` for the end-to-end flow.
Restore is not transactional: it applies topics one at a time as it reads the object, so a failure part-way through (an RF mismatch, an unreachable broker) leaves the topics already created behind — re-running the restore is safe (existing topics are reconciled, not recreated). Like the Rabbitmq driver, this one owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not accumulate objects; the delete is **best-effort** with the same give-up conditions and `backups.cozystack.io/skip-artifact-cleanup` escape hatch documented for Rabbitmq above. One caveat specific to this driver: its image ships an older `curl` (7.76.1) whose `--aws-sigv4` omits a non-default port from the SigV4 canonical host, which every S3 backend then rejects with `403 SignatureDoesNotMatch` on a ported endpoint (fixed in curl 7.86.0). The strategy script works around it with `--connect-to` — it signs and sends a portless URL and redirects the connection to the real port — so a ported endpoint (the `backupStorage.endpoint` fallback `…:8333`, `provisionBucket: false`, or SeaweedFS with `s3.ingress.enabled: false`) works too; the sibling drivers run curl 8.x and need no such workaround. Export time scales with topic count — the backup spends two Kafka CLI invocations (each a fresh JVM, ~1.5–3s) per topic — so a cluster with many hundreds of topics takes tens of minutes; the run is bounded at the Job layer by `activeDeadlineSeconds` (default 30 minutes, from the same wait the readiness precondition uses), so a wedged or unschedulable run is failed by Kubernetes as `DeadlineExceeded` — the pod is killed before the script's final upload, so no object is left orphaned — rather than requeuing forever. A cluster whose export legitimately runs longer raises the bound with the `backupTimeout` BackupClass parameter (a Go duration, e.g. `2h`), which round-trips through the `Backup` so a restore is bounded by the same value.