mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 10:37:21 +09:00
Andrey KolkovandClaude Opus 4.8
6a1e2d0d06
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested capability for the new backups.cozystack.io API (issue #2774). The recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing already existed on the barman-cloud-plugin branch; this adds the reliability and verification layer the issue asked for: - Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set and the recovery Cluster is not healthy, the driver reads the CNPG full-recovery pod logs and, once several distinct attempts have exited with PostgreSQL's "recovery ended before configured recovery target was reached" FATAL, fails the RestoreJob with conditions RecoveryConverged=False / Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the restore deadline. Health is checked first, so a converged recovery always wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a transient archive-lag miss on a near-now target and ignores transient recovery-pod crashes. Adds the pods/log RBAC grant the log read requires. - e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures the server timestamp, writes an after marker, restores to that timestamp and asserts the before row survived and the after row did not; step 46 submits a recoveryTime an hour ahead and asserts it fails fast with reason RecoveryTargetUnreachable. - Working, uncommented example (45-restorejob-pitr.yaml) and a docs section (docs/operations/backup-classes.md) covering the recoverable window, how to discover the earliest/latest restorable time from the backup catalog, the archive-lag tolerance, and idempotency under GitOps. Refs #2774 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com>
39 lines
4.0 KiB
Markdown
39 lines
4.0 KiB
Markdown
# PostgreSQL backup & restore (barman-cloud plugin)
|
|
|
|
End-to-end demo of backing up a cozystack `Postgres` application to S3 and restoring it, using the CloudNativePG **barman-cloud plugin** (the native `spec.backup.barmanObjectStore` path is deprecated in CNPG 1.27 and removed in 1.29). Backups flow through the platform `BackupClass` + `CNPG` strategy so one strategy CR serves every `Postgres` instance in the tenant.
|
|
|
|
## What it provisions
|
|
|
|
The numbered manifests are the documented flow; a human can read and apply them by hand, and `run-all.sh` drives the same files for a smoke test.
|
|
|
|
- `00-bucket.yaml` — a cozystack-managed S3 `Bucket` for the backups, with a `backup` user.
|
|
- `05-postgres-src.yaml` — the source `Postgres`, with `backup.enabled: true` so `archive_command` streams WALs from helm-install onwards.
|
|
- `10-cnpg-strategy.yaml` — the `CNPG` strategy templated against the live application.
|
|
- `15-backupclass.yaml` — maps `Kind=Postgres` to that strategy.
|
|
- `20-plan.yaml` — an optional cron `Plan` for recurring backups.
|
|
- `25-backupjob-adhoc.yaml` — an ad-hoc `BackupJob`.
|
|
- `30-postgres-target.yaml` + `40-restorejob-to-copy.yaml` — restore into a fresh copy, leaving the source running.
|
|
- `35-restorejob-in-place.yaml` — the destructive in-place restore variant.
|
|
- `45-restorejob-pitr.yaml` — point-in-time recovery: restore into the copy as of a specific `spec.options.recoveryTime` (RFC3339).
|
|
|
|
## Placeholders and derived Secrets
|
|
|
|
`05-postgres-src.yaml` and `10-cnpg-strategy.yaml` carry `REPLACE_WITH_COSI_BUCKET_NAME`, `REPLACE_WITH_S3_ENDPOINT` and `REPLACE_WITH_PASSWORD`. They also reference two Secrets, `<app>-cnpg-backup-creds` (the S3 credentials the barman-cloud sidecar reads) and `<app>-cnpg-backup-ca` (the CA it trusts for a self-signed endpoint). The strategy template renders those names against whichever application it drives, so the restore target needs its own pair too — a restore against `pg-target` looks up `pg-target-cnpg-backup-*`. `run-all.sh` resolves the placeholders from the provisioned `Bucket`'s `BucketInfo` Secret and materialises the pair for both the source and the target; editing by hand, you copy the coordinates from `kubectl -n <ns> get secret bucket-<name>-backup -o jsonpath='{.data.BucketInfo}'` and the CA from the seaweedfs CA secret in `tenant-root` (`seaweedfs-ca-cert` by default; `run-all.sh` auto-discovers it from the cert-manager Certificate if the name differs).
|
|
|
|
Drop the `endpointCA` block from `05`/`10` (and set `S3_CA_SECRET=""` for `run-all.sh`) when the S3 endpoint is signed by a publicly-trusted CA.
|
|
|
|
## Run it
|
|
|
|
```sh
|
|
# Defaults to NAMESPACE=tenant-root; override for a tenant namespace.
|
|
NAMESPACE=tenant-root examples/backups/postgres/run-all.sh
|
|
# Tear everything down afterwards (idempotent).
|
|
NAMESPACE=tenant-root examples/backups/postgres/cleanup.sh
|
|
```
|
|
|
|
`run-all.sh` writes a sentinel row into the source, waits for the `BackupJob` to reach `Succeeded`, restores to a copy, and asserts the sentinel round-tripped through S3 into the restored copy. It then runs a **point-in-time recovery**: it writes a `before` marker, captures the server timestamp, writes an `after` marker, and restores the copy to that timestamp (`45-restorejob-pitr.yaml`), asserting the `before` row survived and the `after` row did not. Finally it submits a RestoreJob with a `recoveryTime` an hour in the future and asserts it **fails fast** with `status.phase: Failed` and reason `RecoveryTargetUnreachable` (rather than hanging to the restore deadline). Set `SKIP_RESTORE=1` to stop after a successful backup, or `SKIP_PITR=1` to stop after the latest-point restore.
|
|
|
|
## Automated e2e
|
|
|
|
The Chainsaw suite at `hack/e2e-chainsaw/postgres/` drives this same `run-all.sh` as a second test (`postgres-2-backup-roundtrip`), selected by CI whenever the postgres app (or these scripts) change and on every release cut. It runs in `tenant-root` against the in-cluster seaweedfs endpoint — the isolated e2e tenant cannot reach it across the Cilium egress policy, and the external ingress endpoint is an unroutable placeholder in the sandbox.
|