## What this PR does
Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported,
tested capability for the new `backups.cozystack.io` API. Fixes#2774.
Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the
`recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime`
plumbing; #3313 has since merged and this branch is rebased on `main`.
This PR adds the reliability + verification layer #2774 asked for.
### Changes
- **Precise failure reason for an unreachable target.** The restore
deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the
sole authority for "this restore is stuck" — the client waits the full
window either way. What changes is the *diagnosis*: once the deadline
has elapsed, if `recoveryTime` was set and the recovery Cluster never
went healthy, the CNPG driver reads the `full-recovery` pod logs and —
when several distinct attempts exited with PostgreSQL's `FATAL: recovery
ended before configured recovery target was reached` — fails the
RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason
`RecoveryTargetUnreachable`) and an actionable message, instead of a
bare generic timeout. Timing is unchanged; only the reason/message
differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.)
Design points:
- **We deliberately do NOT trip early off the FATAL.** A valid near-now
target emits that same FATAL transiently — repeatedly, for many
attempts, on a slow archiver — before it converges, so any earlier trip
would wrongly reject a recoverable restore (observed twice in CI). The
FATAL is used only to *explain* a failure the deadline has already
declared.
- **Health wins unconditionally** — the deadline/classification check
precedes the healthy check, but a recovery that converges within the
window reaches success and is never classified as failed (a live dev7
restore did exactly this).
- **Counting distinct FATAL-ing attempts** (not pod failures) tolerates
a one-off archive-lag miss on a near-now target and ignores transient
recovery-pod crashes (a node blip / brief API-server unreachability
never emits that FATAL).
- Adds the `pods/log` RBAC grant the log read needs (distinct
subresource from `pods`); a `Forbidden` is logged loudly and surfaced in
the failure message rather than silently disabling the classification.
- **e2e** (`examples/backups/postgres/run-all.sh`, wired into the
Chainsaw `postgres-2-backup-roundtrip` suite):
- Step 45 — write `before`, capture server timestamp (µs precision),
write `after`, restore to that timestamp, assert `before` present and
`after` absent.
- Step 46 — a `recoveryTime` an hour ahead must fail with reason
`RecoveryTargetUnreachable` once the deadline elapses (asserts the
*reason*, so it proves the classification fired, not just that a timeout
occurred).
- Chainsaw budget raised 25m → 40m to cover three sequential restores +
the post-deadline classification.
- **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a
`docs/operations/backup-classes.md` PITR section (recoverable window,
discovering earliest/latest restorable time from the backup catalog
since `firstRecoverabilityPoint` is unreliable under the plugin,
archive-lag tolerance, idempotency under GitOps).
### Verification
- `go build` / `go vet` clean; unit tests pass
(`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`,
`recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level
success-path test asserting `Succeeded` + `RecoveryConverged=True`, and
— added per review —
`TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`,
a reconcile-level test driving the deadline branch through all three
sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL
absent → generic timeout, `Forbidden` log read → generic timeout +
missing-`pods/log`-RBAC note. It fails if the call-site reason is
reverted to the generic one).
- Reviewed via adversarial branch-review; fixes include a critical
missing-RBAC dead-on-arrival, a classification false-positive on
transient recovery-pod crashes, and a sub-second e2e flake.
- **dev7**: backup + WAL archiving + recovery convergence verified live
on the #3313 build. The load-bearing assumption of the classification —
CNPG **accumulates** failed `full-recovery` pods rather than reusing one
— was confirmed (observed 7 simultaneous failed recovery pods). A
fully-green live step-46 run was blocked by intermittent COSI
bucket-provisioning flakiness on dev7; the negative path is covered by
the CI Chainsaw e2e.
### Downstream repositories
Walked the trigger map in `docs/agents/contributing.md` against the
diff. The only user-facing surface is the new
`docs/operations/backup-classes.md` PITR section; the website keeps a
hand-mirrored copy per docs version.
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: PITR docs mirror warranted (not yet opened; same pending
mirror as #3313's backup-classes changes)
- [x] No other downstream repository is affected by this change
```release-note
feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added PostgreSQL point-in-time recovery (PITR) support via RFC3339
`recoveryTime`.
* Introduced a PITR RestoreJob example, plus helper variables and an
unreachable-target scenario.
* **Bug Fixes**
* Improved restore deadline handling with deterministic, log-based
fail-fast classification for unreachable recovery targets, including
clearer conditions.
* **Documentation**
* Added detailed PITR operational documentation (recovery windows,
failure behavior, and GitOps/idempotent behavior).
* Expanded README and example comments for PITR verification and failure
expectations.
* **Tests**
* Extended unit tests and end-to-end chainsaw workflow to cover PITR
success and unreachable failures.
* **Chores**
* Updated RBAC to allow pod log reads; refined cleanup and run scripts
for additional RestoreJob artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- 00-bucket.yaml: correct the Secret-name comment to include the
"bucket-" release prefix (bucket-<bucket>-<user>), matching the
README and 00-helpers.sh.
- README: note that S3_CA_NAMESPACE defaults to tenant-root
independently of NAMESPACE, so a non-root tenant run must override
it (and S3_CA_SECRET) too.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Reworks the never-merged backup-mariadb-e2e branch onto the modern
Chainsaw approach (docs/agents/e2e-testing.md). The bats port that
branch carried (hack/e2e-apps/backup-mariadb.bats) is dropped — the
whole hack/e2e-apps/*.bats app suite has been replaced by
hack/e2e-chainsaw/, and the mariadb strategy controller + example
manifests it depended on have since landed on main independently. Only
the e2e coverage was still missing.
Convert examples/backups/mariadb/ from static numbered YAML into the
same executable harness the other backup demos use (00-helpers.sh,
run-all.sh, cleanup.sh), keeping the numbered manifests as the
human-readable flow that run-all.sh drives — so the documented flow and
the automated test can never drift. run-all.sh provisions a Bucket,
derives the two per-app Secrets the strategy references from BucketInfo
+ the seaweedfs CA, deploys a source MariaDB with a sentinel row, fires
an ad-hoc BackupJob, restores to a copy with a to-copy RestoreJob, and
asserts the sentinel round-tripped through S3 into the copy while the
source is left untouched.
Add a second Chainsaw Test (mariadb-2-backup-roundtrip) to
hack/e2e-chainsaw/mariadb/ that drives run-all.sh + cleanup.sh, mirroring
postgres-2-backup-roundtrip: runs in tenant-root (the isolated e2e tenant
cannot reach tenant-root's seaweedfs across the Cilium egress policy),
skipDelete since the harness (not Chainsaw) owns the resources, fail-fast
on terminal BackupJob/RestoreJob phases, and scoped catch diagnostics.
Cleanup also prunes the operator-side k8s.mariadb.com Backup/Restore CRs
the Cozystack jobs own only by label.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested
capability for the new backups.cozystack.io API (issue #2774). The
recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing
already existed on the barman-cloud-plugin branch; this adds the reliability
and verification layer the issue asked for:
- Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set
and the recovery Cluster is not healthy, the driver reads the CNPG
full-recovery pod logs and, once several distinct attempts have exited with
PostgreSQL's "recovery ended before configured recovery target was reached"
FATAL, fails the RestoreJob with conditions RecoveryConverged=False /
Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the
restore deadline. Health is checked first, so a converged recovery always
wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a
transient archive-lag miss on a near-now target and ignores transient
recovery-pod crashes. Adds the pods/log RBAC grant the log read requires.
- e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw
postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures
the server timestamp, writes an after marker, restores to that timestamp and
asserts the before row survived and the after row did not; step 46 submits a
recoveryTime an hour ahead and asserts it fails fast with reason
RecoveryTargetUnreachable.
- Working, uncommented example (45-restorejob-pitr.yaml) and a docs section
(docs/operations/backup-classes.md) covering the recoverable window, how to
discover the earliest/latest restorable time from the backup catalog, the
archive-lag tolerance, and idempotency under GitOps.
Refs #2774
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The backup roundtrip e2e copied the seaweedfs S3 CA from a Secret named
seaweedfs-system-ca-cert in tenant-root, a name captured off a dev
cluster. On current main the seaweedfs chart pins
fullnameOverride: seaweedfs, so the cert-manager CA Certificate's secret
is seaweedfs-ca-cert; the old name is absent and the first CI run of the
e2e failed at step 05a with 'secrets "seaweedfs-system-ca-cert" not
found'.
Default S3_CA_SECRET to seaweedfs-ca-cert and, when that Secret is
absent, auto-discover the seaweedfs CA Certificate (the seaweedfs-labelled
one with spec.isCA=true) and read its secretName, so a future
fullnameOverride change does not silently break the copy again. Fail with
a clear message when neither resolves. README and chainsaw comment
updated to match.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Ungate postgres-2-backup-roundtrip: instead of an opt-in env flag it now
runs whenever TIA selects the postgres suite (postgres changes, and every
release cut), like the other chainsaw tests. The blocker was network
placement, not capability: the S3 client is the barman-cloud sidecar
inside the Postgres Pod, and the isolated e2e tenant's Cilium egress
allowlist blocks it from tenant-root's seaweedfs — so the Test runs in
tenant-root (same-tenant egress) against the in-cluster S3 endpoint,
trusting its self-signed cert via the copied seaweedfs-system-ca-cert.
This is exactly the recipe validated end to end on a real cluster.
Also map examples/backups/<app>/ edits to that app's suite in
hack/select-e2e.sh (the round-trip executes those scripts as its
harness; previously such edits were silently ignored), with bats
coverage for both the mapped and the no-suite case.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The CNPG strategy template renders <app>-cnpg-backup-creds / -ca against
whichever application it drives, so a to-copy restore looks the pair up
under the TARGET's name. The harness only created the source's pair; on a
cluster with stale target Secrets from an earlier run the restore failed
TLS verification against the wrong CA. Materialise the pair for the
target too (kubectl apply, so stale content is corrected) and delete both
pairs in cleanup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Two defects of the same class — the WAL-archive S3 prefix (serverName)
changing or defaulting differently between components:
Recovery default regression (review blocker): native CNPG defaulted an
absent externalClusters serverName to the entry name (= bootstrap.oldName),
but the barman-cloud plugin defaults it to the NEW cluster's name, so a
legacy-flow restore with bootstrap.serverName unset would read the wrong
S3 prefix (hard failure, or silently restoring another cluster's data).
The chart now always emits serverName, defaulting to bootstrap.oldName.
Live-cluster serverName flip (found by the gated postgres backup e2e on a
real cluster): the chart renders spec.plugins without serverName
(effective prefix = cluster name) while a strategy template may name a
different one; the driver's ForceOwnership patch then flipped the prefix
mid-stream, splitting the WAL archive across two prefixes — the WAL
holding the backup's start was lost between them and the restore failed
with "WAL not found". The driver now preserves the live Cluster's
effective serverName (explicit parameter, or the cluster name when the
plugin is attached without one) and only applies the strategy's value to
a Cluster with no plugin attached; both charts pin serverName explicitly,
and the example strategy no longer sets one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Port examples/backups/postgres into an automated e2e, following the etcd
backup precedent: the numbered example manifests become a runnable harness
(00-helpers.sh, run-all.sh, cleanup.sh) that a human reads and the test
drives, so the documented flow and the test can never drift.
run-all.sh provisions a Bucket, deploys the source Postgres (writing a
sentinel row), derives the two Secrets the barman-cloud ObjectStore
references from the Bucket's BucketInfo and seaweedfs' CA, submits an
ad-hoc BackupJob and waits for Succeeded, then restores to a copy and
asserts the sentinel round-tripped through S3 — the in-cluster witness
that a plugin backup is both taken and restorable end to end.
A second Test document (postgres-2-backup-roundtrip) in the existing
hack/e2e-chainsaw/postgres suite drives run-all.sh, gated out of CI behind
POSTGRES_E2E_S3_ROUNDTRIP=1 (kind has no reachable, CA-trusted S3 endpoint,
and a tenant Pod cannot cross the Cilium egress policy to tenant-root's
seaweedfs). Placing it in the postgres suite (rather than a new dir) keeps
it triggered by packages/apps/postgres changes under Test Impact Analysis.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The backup round-trip test could wedge for 20 min and then report a bare
EtcdSnapshot phase=Started with no root cause: dump_diagnostics dumped the
(healthy) EtcdCluster and the operator log, but the snapshot agent runs in
its own Job Pod, so the etcd-read / S3-upload error lives only in that Pod's
log and in the BackupJob/EtcdSnapshot status.
- dump_diagnostics now dumps the whole backup chain: BackupJob/Backup and
EtcdSnapshot describe, each snapshot Job's describe + Pod logs (selected by
the operator's etcd-operator.cozystack.io/cluster label), and recent
namespace events (the primary signal when a hung Pod logs nothing).
- wait_for_field gains an optional terminal-failure value; steps 04/05 pass
Failed so the BackupJob/RestoreJob waits return immediately on a terminal
phase instead of polling to the timeout — which also keeps the snapshot
Job Pod within its 600s TTL so its log is still readable at dump time.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Rewrite docs/agents/e2e-testing.md for the Chainsaw suite: declarative
asserts replace the until-get/wait backstops, Chainsaw cleanup replaces
test-level EXIT traps, catch blocks replace the `|| { ...; false; }` idiom,
and the TIA/diagnostics sections describe suites + `make test-chainsaw
CHAINSAW_SUITES`. Move the BATS->Chainsaw bullet out of 'in-flight' since it
is now the standard.
Fix references the migration left stale: hack/e2e-apps/ -> hack/e2e-chainsaw/
in the gemini styleguide and the nats/etcd backup READMEs. The
backup-postgres.bats / backup-foundationdb.bats files cited by the postgres
examples, the postgres README, and the foundationdb controller comment never
existed -- drop the false 'CI e2e drives this' claims and describe those
backup flows as the manual reference flows they are.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What
Implements the `strategy.backups.cozystack.io/Job` backup strategy — a
generic, application-agnostic driver that runs a user-supplied
Kubernetes Job to create (and restore) a backup.
## Background
This PR began as a draft written against an earlier shape of the backups
API. The API has since moved to a `BackupClass`-based dispatch model,
which broke the original draft on three counts:
- `BackupJob.Spec.StrategyRef` and `BackupJob.Spec.StorageRef` were
removed; the strategy and its parameters are now resolved from a
`BackupClass` into a `ResolvedBackupConfig`.
- Strategy handling moved from standalone per-strategy reconcilers to a
dispatch model: `BackupJobReconciler.Reconcile` /
`RestoreJobReconciler.Reconcile` resolve the strategy and switch into
per-strategy methods (`reconcileVelero`, `reconcileCNPG`,
`reconcileAltinity`, …).
- The draft's template context (`.Values`, `.Storage`) diverged from the
convention the other strategies adopted, and its `.Storage` half
targeted a storage API that does not exist.
## What changed
- **`reconcileJob` / `reconcileJobRestore`**
(`internal/backupcontroller/jobstrategy_controller.go`): replaces the
standalone `BackupJobStrategyReconciler` with the two dispatch methods
the `BackupJob` / `RestoreJob` reconcilers call. The Job strategy is the
generic counterpart of the Altinity driver — same lifecycle (render
`PodTemplateSpec` → run a one-shot `batch/v1.Job` → poll its terminal
condition → emit a `Backup`), minus the ClickHouse-specific
`applicationRef` gate, so it runs for any application. Shared package
helpers (`renderJobTemplate`, `jobConditionState`,
`getApplicationUnstructured`, status helpers, deterministic Job naming,
controller refs) are reused.
- **Template context** follows the in-tree convention used by the other
drivers: `.Application` (live application object), `.Release.Name` /
`.Release.Namespace`, `.Mode` (`backup` / `restore`), `.Parameters`
(resolved `BackupClass` parameters), and `.Backup` (restore runs only).
Strategy parameters round-trip through the `Backup`'s `driverMetadata`
so a later restore re-renders against the same values.
- **Restore** re-renders the same template with `.Mode == "restore"`.
Cross-namespace restore is unsupported (the RestoreJob target reference
is namespace-local), matching the Altinity driver.
- **`JobSpec.Template` doc + regenerated CRD**: the field doc previously
advertised `.Values`; it now documents the real context, and
`strategy.backups.cozystack.io_jobs.yaml` is regenerated to match (the
change is limited to that field's description).
- **Tests** (`jobstrategy_controller_test.go`): backup-Job creation /
args / labels / status, completion → `Backup` artifact with parameter
round-trip, Job-failure surfacing, missing-strategy terminal failure,
restore into the target namespace, restore completion, and the
parameter-prefix unit. A deliberately generic application kind pins that
the driver never gates on `applicationRef.kind`.
No RBAC changes are required: the controller already holds
`strategy.backups.cozystack.io/*`, `batch/jobs`, `apps.cozystack.io/*`,
and `backups` permissions.
## Notes for reviewers
- Backup and restore share one template, so a strategy author's
`PodTemplateSpec` must branch on `.Mode` to do anything
restore-specific. This matches the existing convention; worth confirming
it's the intended contract for the Job strategy's docs/examples.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Implemented full backup/restore job strategy with deterministic Job
creation, idempotent execution, and status/condition updates; added demo
example scripts and reusable shell helpers for a NATS JetStream
backup/restore workflow.
* **Documentation**
* Clarified Job template rendering context and listed supported template
variables; added README and step-by-step scenario docs for the NATS
example.
* **Tests**
* Added comprehensive unit tests covering job creation,
completion/failure handling, parameter extraction, and unmappable-target
failure cases.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/1721?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add examples/backups/nats/, the first worked example of the generic Job
backup strategy. It backs up a NATS JetStream stream that has no dedicated
driver, using a stock natsio/nats-box image: 'nats stream backup' -> tar ->
curl --aws-sigv4 PUT to the COSI/seaweedfs S3 endpoint, and the reverse for
restore. One PodTemplateSpec serves both directions, branching at runtime on
{{ .Mode }} (the engine templates each string leaf independently and cannot
restructure the Pod per mode).
Mirrors the clickhouse example's structure: numbered scripts (bucket -> NATS
src + seeded stream -> strategy -> BackupClass -> BackupJob -> in-place and
to-copy RestoreJob), shared helpers, run-all.sh, README, and scenario docs.
Like every examples/backups/* dir it is a runnable demo, not CI-gated.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Andrey Kolkov
