A to-copy restore replays the source cluster's system users into the
target, so the target's own generated credentials stop authenticating
and the harness cannot read the sentinel back out of the restored copy.
The suite has never passed for that reason, holding the in-tree e2e run
red on every branch that selects it.
Drive run-all.sh with SKIP_RESTORE=1 so the suite still proves the
backup half end to end, and pin the skip by asserting the skip branch's
own log line: exit zero alone is also what a completed round-trip
returns, so without that assertion the step could not tell a skipped
flow from a full one, nor notice SKIP_RESTORE failing to reach the
harness.
The to-copy manifest and the example README both stated that CI covers
the restore, which stops being true here, so both now say where
coverage ends. The asserted log line is annotated at its source, since
rewording it would red the suite with a bare match failure.
Coverage returns by dropping SKIP_RESTORE together with the assertion
on that line, once the credential handling behind the restore is
fixed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
Address the two blocking findings in PR #3562:
- Restore into a differently-named target now authenticates with the
TARGET cluster's own S3 credentials instead of the source's. The
backupSource carried the source release's -s3-creds Secret, which is
deleted together with the source app — the disaster-recovery case
backups exist for — leaving a to-copy restore unable to reach S3. The
restore replays into the target (gated on backup.enabled just above),
whose own -s3-creds Secret is present and grants access to the same
bucket; only the credential reference is swapped, bucket/prefix/endpoint
still come from the source archive. Falls back to the source reference
when the target declares no discoverable s3 credentials (in-place
restore resolves to the same Secret, so it is a no-op there).
- A non-string recoveryTime (spec.options is free-form runtime.RawExtension,
so {"recoveryTime": 20260805} passes admission) no longer silently
degrades a PITR request to a plain snapshot restore reported Succeeded.
When the typed decode fails and the blob carries a recoveryTime key, the
RestoreJob now fails terminally — the same guarantee pitrSpec already
makes for a malformed RFC3339 string.
Minor findings also addressed:
- BackupJob tolerates a not-yet-present MongoDB app CR with the same
bounded grace the psmdb-CR existence gate uses, instead of failing
terminally on a GitOps single-commit ordering race.
- examples: the to-copy target writes into its OWN S3 prefix
(mongodb-target/) rather than the source's, so the two clusters' oplog
and backup metadata no longer interleave under shared keys (both run
replset rs0).
- examples: wait_psmdb_ready no longer treats a transient state=error as
terminal — the cluster state is recomputed each reconcile and can blip
through error during replset init before converging to ready.
Unit tests cover the target-credential selector (preferred/default/sole
storage) and the load-bearing recoveryTime-key detection.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
PITR: replace the bespoke spec.options.pitr{type,date} blob with a single
spec.options.recoveryTime RFC3339 string, matching the CNPG strategy's option
exactly (CNPGRestoreOptions.RecoveryTime). The driver converts recoveryTime to
the psmdb Restore's pitr{type:date,date} in the operator's "YYYY-MM-DD HH:MM:SS"
UTC format. An empty recoveryTime restores the snapshot as-taken (documented
difference from CNPG, whose empty value means latest WAL — a psmdb logical
backup is already consistent). A non-RFC3339 recoveryTime fails terminally.
Unknown options: parseMongoDBRestoreOptions now also returns the set of
spec.options keys the driver does not recognise; reconcileMongoDBRestore emits a
UnknownRestoreOption Warning event per unknown key (and logs it) instead of
silently ignoring it — so a "recoverytime" typo no longer silently degrades a
PITR restore to a snapshot restore. (BackupJob carries no spec.options blob, so
there is no backup-path equivalent; the only backup-time input is the free-form
BackupClass parameters used as template vars.)
Docs + examples updated to recoveryTime; tests cover the RFC3339→UTC mapping,
empty/invalid recoveryTime, and unknown-key reporting.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so
`apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and
ClickHouse native drivers — logical backups, restore-to-differently-named
instances, and point-in-time recovery through the unified
BackupJob/RestoreJob/BackupClass/Plan interface (closes#3545).
The driver orchestrates the operator's PerconaServerMongoDBBackup /
PerconaServerMongoDBRestore CRDs:
- BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls
status.state, and persists the S3 destination + storage descriptor
(by-reference, no credentials) onto the Cozystack Backup artifact.
- RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back
from the source backup status, snapshot fallback) so the same artifact
restores in-place or into a differently-named instance; PITR via
spec.options.pitr (date/latest).
Because psmdb resolves the backup storage by name against the cluster's
spec.backup.storages, the driver names the chart's storage (s3-storage) via the
cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear
Ready=False precondition when a target has backups disabled.
Wiring:
- api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy.
- internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp
partial-schema mirrors; dispatch, supported-kinds and scheme registration.
- backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb
Strategy CR, and psmdb.percona.com RBAC.
- apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no
CA-bundle field) so backups can target Cozystack's self-signed seaweedfs.
Tests + e2e:
- unit tests for the driver (validation, ensure-idempotency, precondition,
backupSource resolution, snapshot round-trip, PITR option parsing).
- examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that
drives run-all.sh end-to-end (sentinel document round-trips through S3 into a
restored copy while the source stays untouched), per docs/agents/e2e-testing.md.
- docs/operations/backup-classes.md updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review feedback on #3385:
- Move the endpointCA rationale out of the Helm template (lllamnyp): the
SSL_CERT_DIR and CA-volume blocks were multi-paragraph essays. Trim each to
a few lines stating only the non-obvious decision (SSL_CERT_DIR vs
AWS_CA_BUNDLE; the deliberate non-optional mount) and point at the
values.yaml endpointCA field for the full description.
- wait_deleted (00-helpers.sh): poll with 'get --ignore-not-found' and treat
only an empty result as deletion, so an RBAC/API-server/transport failure is
no longer read as success and cleanup can't settle on a false positive.
- Backup demo CA Secret: scope the default name to CLICKHOUSE_NAME, stamp an
ownership label on the copy, refuse to overwrite a Secret the demo doesn't
own, and delete on cleanup only when that label is present — so a foreign
Secret sharing the name is never clobbered or removed.
- README: document CH_BACKUP_CA_SECRET and the release-scoped, demo-owned CA
Secret; spell out the admin-copy path (export CH_BACKUP_CA_SECRET before
steps 04 and 07).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review notes from IvanHunters on #3385 (LGTM with non-blocking findings).
1. The backup-s3-ca volume carries no `optional: true` while the adjacent
useSystemBucket credential refs deliberately do, and the asymmetry was
undocumented — an operator naming a Secret that does not exist gets the whole
ClickHouse Pod stuck in ContainerCreating, not just the sidecar. Kept the
fail-closed behaviour and wrote down why the two cases differ: credentials are
projected by the platform after install, so a strict ref would wedge every
fresh release until the first BackupJob, whereas this CA is named by the
tenant, must already exist, and is trust material — optional would mount an
empty dir, the Pod would look healthy, and the first backup would fail deep
inside clickhouse-backup with an x509 error naming neither the Secret nor the
chart. The comment also states the blast radius plainly.
2. The CA probe swallowed stderr, so an RBAC denial on the cross-namespace read
was reported as "CA not found" followed by advice (set S3_CA_SECRET) that
cannot fix a permissions problem. It now separates present / absent /
not-allowed-to-look and says so, and the README no longer labels step 03
simply "tenant": copying the CA reads a Secret in tenant-root, which is a
platform capability. Added a section spelling out the three ways to satisfy
that (admin credentials, a one-off admin copy, or skipping the CA entirely on
a publicly-trusted endpoint).
3. Caveat, and a usability regression this branch introduced: on a cluster with
a publicly-trusted endpoint and no seaweedfs, step 03 hard-exited on a default
the user never chose. It now distinguishes an explicitly named Secret (still
an error when missing — a typo should not be guessed past) from the unset
default, where it warns and continues with backup.endpointCA unset. Verified
all four paths against a fake kubectl: Forbidden, absent+default,
absent+explicit, and present.
4. Caveat: endpointCA reaches the clickhouse-backup sidecar only. The legacy
`schedule` CronJob writes to S3 through restic and does not consume it, so a
private-CA endpoint combined with `schedule` still fails TLS. Documented on
the field itself, so it lands in values.schema.json, the README and the
cozyrds openAPISchema.
The PR body's claim about select-e2e.sh/select-install.sh mappings was stale
after the restructure that removed them; corrected there rather than in code.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Recommendations from the third review pass (which returned LGTM; none of these
blocked, all three are cheap and make the failure paths honest).
1. The round-trip's catch named nothing. An unnamed `describe HelmRelease` in
tenant-root dumps every release on the root tenant of a live cluster and
buries the two that matter, so name them (mariadb-2-backup-roundtrip names
its release for the same reason). Raised podLogs tail 100 -> 200, since the
sidecar logs every action and an S3 error sits well back from the tail by the
time a step fails, and added the Keeper Pods: chkeeper.yaml labels them
`app: <release>-keeper`, so the existing app.kubernetes.io/instance selectors
missed them entirely — and a source ClickHouse that never reaches a ready
replica is usually Keeper failing to form a quorum, which explains itself
only there.
2. SSL_CERT_DIR is now `/etc/ssl/cozy-s3-ca:/etc/ssl/certs`. The single-directory
form works — Go reads the default bundle FILE regardless, which is what the
earlier measurement showed — but that left the public-CA half of the contract
resting on a file this image happens to ship. Naming the system directory
costs nothing and survives a base-image bump that moves the bundle; verified
against public AWS S3 that both forms still verify (403 auth, no x509). The
mount path stays the bare directory, and the unit test now pins both halves
separately — it caught the over-broad edit that had briefly made mountPath a
colon list.
3. cleanup.sh returned while the uninstalls it triggered were still running, so
the next suite (etcd) could start with helm-controller still draining this
one, and the pre-clean could hand step 04 an application whose predecessor
was still going away. It now waits for both ClickHouse releases and the
Bucket to actually disappear, and — since the script deliberately runs
without `set -e` so one absent resource cannot abandon the rest — records
those failures and exits non-zero, rather than reporting a teardown that did
not settle as success. wait_deleted polls `get` instead of using
`kubectl wait --for=delete`, which errors on an already-absent resource: the
normal case here, and indistinguishable from a real failure.
Sized the two cleanup step budgets to 12m for the same reason the run step is
60m: cleanup.sh's waits now sum to 540s, which the previous 8m did not clear.
The Keeper-sized-from-the-server-resources bug the review found outside this
diff is filed as #3487.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Second review pass on this branch.
1. The op timeout equalled the inner sum instead of exceeding it: the comment
enumerated 3900s of inner waits and then set 65m, and 3900s IS 65m — so there
was zero headroom for the applies and SQL calls between those waits, and the
claim of "room" was false. Sized the inner budgets against what CI actually
takes instead (the bucket suite completes in ~37s and a backup-enabled
ClickHouse plus Keeper reaches Ready in ~98s, measured on run 30353894332):
bucket claim/access 300->180 and the three job waits 600->480, summing to
3300s, under a 60m op timeout. Shrinking the inner budgets rather than
raising the ceiling also keeps this suite closer to its postgres and mariadb
siblings on a 180-minute job cap.
2. The strategy Pod's securityContext comment still justified root with "Pod
runs apk add at startup", which the previous commit made conditional — the
default path now runs the platform's pre-baked image and installs nothing.
Scoped the note to the alpine fallback and said explicitly that the field set
matches cozy-default-altinity, runAsNonRoot deliberately unset in both.
3. clickhouse-backup.yaml still referenced clickhouse-backup-{1-contracts,
2-roundtrip}: the file moved unchanged, so the Test rename never reached its
own header and a reader following either name found nothing.
4. wait_sts_ready's timeout diagnostic could not print anything. It selected
clickhouse.altinity.com/chi=${name#chi-}, which yields
clickhouse-<app>-clickhouse-0-0 while the label carries the CHI name
(clickhouse-<app>), and 2>/dev/null then swallowed kubectl's "No resources
found". Ask for the StatefulSet's own first replica by name instead and let
both streams through; verified it now surfaces the Pod events that explain
why the replica never came up.
Also dropped the clickhouse bats case added in the previous commit: it asserted
what the generic examples/backups/<app> -> <app> mapping already guarantees, so
it did not lock the invariant its comment claimed. That invariant is now
recorded where it belongs — in the mapping's own comment in select-e2e.sh.
The dedicated ch-backup-client image this example now inherits is tracked in
#3420; noted there that the command: override is load-bearing in two files.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review findings on the previous two commits.
1. TIA no longer selected the suite that runs the harness. select-e2e.sh maps
examples/backups/<app>/** to suite <app>, so with the round-trip living in a
separate clickhouse-backup/ dir an edit to the harness selected `clickhouse`
— which did not execute those scripts — while the suite that does was left
out. Fixed the way postgres/mariadb/etcd already do it: both Tests move into
hack/e2e-chainsaw/clickhouse/ as clickhouse-1-backup-contracts and
clickhouse-2-backup-roundtrip. That also deletes every special case the
earlier commits added (src_to_suites, suite_to_source, and the bats case that
guarded them — both scripts are now byte-identical to main), and the suite
list in hack/e2e-chainsaw/README.md stays correct because no dir is added.
The bats case is replaced by one asserting the invariant that actually
matters: an examples/backups/clickhouse edit selects the suite that runs it.
2. The docs claimed a fail-fast the harness did not have, and the budget was
sized as if it did. wait_for_field polled to the full timeout with no
terminal-value escape, and the HelmRelease/StatefulSet waits were bare
`kubectl wait` with no existence backstop and no Stalled fail-fast. Ported
wait_for_field's fail_value parameter and wait_hr_ready from the postgres
helpers, added wait_sts_ready, and routed all eight waits through them.
Verified against a fake kubectl: a Failed BackupJob now bails in 3s instead
of 600s, a Stalled HelmRelease immediately. The inner budgets sum to 3900s,
which the old 30m op timeout did not cover — a hang would have been SIGKILLed
with no diagnostics, the failure mode .chainsaw.yaml calls strictly worse — so
the timeout is now 65m with the arithmetic written out.
3. The strategy Pod ran `apk add --no-cache curl jq` on every backup and
restore. Harmless while nothing in CI ran it; a per-PR dependency on the
Alpine CDN once it does, and broken outright air-gapped — which is exactly
why the shipped strategy bakes curl+jq into its image. Step 01 now reads that
image off cozy-default-altinity and drops the apk add, falling back to
alpine only when the platform strategy is absent. Also bounded the two
polling curls with --max-time; a hung connection previously stalled the loop
until the outer timeout.
4. 91-scenario-user-backup.md still documented the old tenant-test default.
5. .chainsaw.yaml claimed all app tests run in tenant-test, which the postgres
and mariadb round-trips already contradicted before this branch.
The round-trip header now also states its scope plainly: it exercises the
example's strategy, not the shipped cozy-default-altinity.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The round-trip Test was skipping: it exited 0 behind CLICKHOUSE_E2E_S3_ROUNDTRIP,
so "PASS (1.01s)" meant the gate fired, not that backup and restore worked. Only
the contracts Test ran, which proves the chart still exposes the sidecar and its
auth Secret — not that a byte ever reaches S3 and comes back.
The gate existed because the sidecar could not verify the in-cluster SeaweedFS
certificate. backup.endpointCA closes that, so adopt the shape mariadb and
postgres already use to run their round-trips un-gated:
- examples: resolve the S3 endpoint from S3_ENDPOINT when set (BucketInfo
advertises the external ingress URL, an unroutable placeholder in CI), copy the
seaweedfs CA (auto-discovered from its cert-manager Certificate when the
fullname default is absent) into a per-release Secret, and reference it from
backup.endpointCA on both the source and the to-copy target. NAMESPACE now
defaults to tenant-root, matching the other backup demos.
- suite: drop the gate, run in tenant-root — an isolated tenant's Cilium egress
allowlist blocks its Pods from reaching tenant-root's seaweedfs — and pre-clean
with cleanup.sh, because tenant-root is shared and a leftover Succeeded
BackupJob would satisfy the harness's wait without a backup running.
cleanup.sh also drops the copied CA Secret; the platform's own is never touched.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
## What this PR does
Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported,
tested capability for the new `backups.cozystack.io` API. Fixes#2774.
Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the
`recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime`
plumbing; #3313 has since merged and this branch is rebased on `main`.
This PR adds the reliability + verification layer #2774 asked for.
### Changes
- **Precise failure reason for an unreachable target.** The restore
deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the
sole authority for "this restore is stuck" — the client waits the full
window either way. What changes is the *diagnosis*: once the deadline
has elapsed, if `recoveryTime` was set and the recovery Cluster never
went healthy, the CNPG driver reads the `full-recovery` pod logs and —
when several distinct attempts exited with PostgreSQL's `FATAL: recovery
ended before configured recovery target was reached` — fails the
RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason
`RecoveryTargetUnreachable`) and an actionable message, instead of a
bare generic timeout. Timing is unchanged; only the reason/message
differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.)
Design points:
- **We deliberately do NOT trip early off the FATAL.** A valid near-now
target emits that same FATAL transiently — repeatedly, for many
attempts, on a slow archiver — before it converges, so any earlier trip
would wrongly reject a recoverable restore (observed twice in CI). The
FATAL is used only to *explain* a failure the deadline has already
declared.
- **Health wins unconditionally** — the deadline/classification check
precedes the healthy check, but a recovery that converges within the
window reaches success and is never classified as failed (a live dev7
restore did exactly this).
- **Counting distinct FATAL-ing attempts** (not pod failures) tolerates
a one-off archive-lag miss on a near-now target and ignores transient
recovery-pod crashes (a node blip / brief API-server unreachability
never emits that FATAL).
- Adds the `pods/log` RBAC grant the log read needs (distinct
subresource from `pods`); a `Forbidden` is logged loudly and surfaced in
the failure message rather than silently disabling the classification.
- **e2e** (`examples/backups/postgres/run-all.sh`, wired into the
Chainsaw `postgres-2-backup-roundtrip` suite):
- Step 45 — write `before`, capture server timestamp (µs precision),
write `after`, restore to that timestamp, assert `before` present and
`after` absent.
- Step 46 — a `recoveryTime` an hour ahead must fail with reason
`RecoveryTargetUnreachable` once the deadline elapses (asserts the
*reason*, so it proves the classification fired, not just that a timeout
occurred).
- Chainsaw budget raised 25m → 40m to cover three sequential restores +
the post-deadline classification.
- **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a
`docs/operations/backup-classes.md` PITR section (recoverable window,
discovering earliest/latest restorable time from the backup catalog
since `firstRecoverabilityPoint` is unreliable under the plugin,
archive-lag tolerance, idempotency under GitOps).
### Verification
- `go build` / `go vet` clean; unit tests pass
(`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`,
`recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level
success-path test asserting `Succeeded` + `RecoveryConverged=True`, and
— added per review —
`TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`,
a reconcile-level test driving the deadline branch through all three
sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL
absent → generic timeout, `Forbidden` log read → generic timeout +
missing-`pods/log`-RBAC note. It fails if the call-site reason is
reverted to the generic one).
- Reviewed via adversarial branch-review; fixes include a critical
missing-RBAC dead-on-arrival, a classification false-positive on
transient recovery-pod crashes, and a sub-second e2e flake.
- **dev7**: backup + WAL archiving + recovery convergence verified live
on the #3313 build. The load-bearing assumption of the classification —
CNPG **accumulates** failed `full-recovery` pods rather than reusing one
— was confirmed (observed 7 simultaneous failed recovery pods). A
fully-green live step-46 run was blocked by intermittent COSI
bucket-provisioning flakiness on dev7; the negative path is covered by
the CI Chainsaw e2e.
### Downstream repositories
Walked the trigger map in `docs/agents/contributing.md` against the
diff. The only user-facing surface is the new
`docs/operations/backup-classes.md` PITR section; the website keeps a
hand-mirrored copy per docs version.
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: PITR docs mirror warranted (not yet opened; same pending
mirror as #3313's backup-classes changes)
- [x] No other downstream repository is affected by this change
```release-note
feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added PostgreSQL point-in-time recovery (PITR) support via RFC3339
`recoveryTime`.
* Introduced a PITR RestoreJob example, plus helper variables and an
unreachable-target scenario.
* **Bug Fixes**
* Improved restore deadline handling with deterministic, log-based
fail-fast classification for unreachable recovery targets, including
clearer conditions.
* **Documentation**
* Added detailed PITR operational documentation (recovery windows,
failure behavior, and GitOps/idempotent behavior).
* Expanded README and example comments for PITR verification and failure
expectations.
* **Tests**
* Extended unit tests and end-to-end chainsaw workflow to cover PITR
success and unreachable failures.
* **Chores**
* Updated RBAC to allow pod log reads; refined cleanup and run scripts
for additional RestoreJob artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- 00-bucket.yaml: correct the Secret-name comment to include the
"bucket-" release prefix (bucket-<bucket>-<user>), matching the
README and 00-helpers.sh.
- README: note that S3_CA_NAMESPACE defaults to tenant-root
independently of NAMESPACE, so a non-root tenant run must override
it (and S3_CA_SECRET) too.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Reworks the never-merged backup-mariadb-e2e branch onto the modern
Chainsaw approach (docs/agents/e2e-testing.md). The bats port that
branch carried (hack/e2e-apps/backup-mariadb.bats) is dropped — the
whole hack/e2e-apps/*.bats app suite has been replaced by
hack/e2e-chainsaw/, and the mariadb strategy controller + example
manifests it depended on have since landed on main independently. Only
the e2e coverage was still missing.
Convert examples/backups/mariadb/ from static numbered YAML into the
same executable harness the other backup demos use (00-helpers.sh,
run-all.sh, cleanup.sh), keeping the numbered manifests as the
human-readable flow that run-all.sh drives — so the documented flow and
the automated test can never drift. run-all.sh provisions a Bucket,
derives the two per-app Secrets the strategy references from BucketInfo
+ the seaweedfs CA, deploys a source MariaDB with a sentinel row, fires
an ad-hoc BackupJob, restores to a copy with a to-copy RestoreJob, and
asserts the sentinel round-tripped through S3 into the copy while the
source is left untouched.
Add a second Chainsaw Test (mariadb-2-backup-roundtrip) to
hack/e2e-chainsaw/mariadb/ that drives run-all.sh + cleanup.sh, mirroring
postgres-2-backup-roundtrip: runs in tenant-root (the isolated e2e tenant
cannot reach tenant-root's seaweedfs across the Cilium egress policy),
skipDelete since the harness (not Chainsaw) owns the resources, fail-fast
on terminal BackupJob/RestoreJob phases, and scoped catch diagnostics.
Cleanup also prunes the operator-side k8s.mariadb.com Backup/Restore CRs
the Cozystack jobs own only by label.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested
capability for the new backups.cozystack.io API (issue #2774). The
recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing
already existed on the barman-cloud-plugin branch; this adds the reliability
and verification layer the issue asked for:
- Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set
and the recovery Cluster is not healthy, the driver reads the CNPG
full-recovery pod logs and, once several distinct attempts have exited with
PostgreSQL's "recovery ended before configured recovery target was reached"
FATAL, fails the RestoreJob with conditions RecoveryConverged=False /
Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the
restore deadline. Health is checked first, so a converged recovery always
wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a
transient archive-lag miss on a near-now target and ignores transient
recovery-pod crashes. Adds the pods/log RBAC grant the log read requires.
- e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw
postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures
the server timestamp, writes an after marker, restores to that timestamp and
asserts the before row survived and the after row did not; step 46 submits a
recoveryTime an hour ahead and asserts it fails fast with reason
RecoveryTargetUnreachable.
- Working, uncommented example (45-restorejob-pitr.yaml) and a docs section
(docs/operations/backup-classes.md) covering the recoverable window, how to
discover the earliest/latest restorable time from the backup catalog, the
archive-lag tolerance, and idempotency under GitOps.
Refs #2774
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The backup roundtrip e2e copied the seaweedfs S3 CA from a Secret named
seaweedfs-system-ca-cert in tenant-root, a name captured off a dev
cluster. On current main the seaweedfs chart pins
fullnameOverride: seaweedfs, so the cert-manager CA Certificate's secret
is seaweedfs-ca-cert; the old name is absent and the first CI run of the
e2e failed at step 05a with 'secrets "seaweedfs-system-ca-cert" not
found'.
Default S3_CA_SECRET to seaweedfs-ca-cert and, when that Secret is
absent, auto-discover the seaweedfs CA Certificate (the seaweedfs-labelled
one with spec.isCA=true) and read its secretName, so a future
fullnameOverride change does not silently break the copy again. Fail with
a clear message when neither resolves. README and chainsaw comment
updated to match.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Ungate postgres-2-backup-roundtrip: instead of an opt-in env flag it now
runs whenever TIA selects the postgres suite (postgres changes, and every
release cut), like the other chainsaw tests. The blocker was network
placement, not capability: the S3 client is the barman-cloud sidecar
inside the Postgres Pod, and the isolated e2e tenant's Cilium egress
allowlist blocks it from tenant-root's seaweedfs — so the Test runs in
tenant-root (same-tenant egress) against the in-cluster S3 endpoint,
trusting its self-signed cert via the copied seaweedfs-system-ca-cert.
This is exactly the recipe validated end to end on a real cluster.
Also map examples/backups/<app>/ edits to that app's suite in
hack/select-e2e.sh (the round-trip executes those scripts as its
harness; previously such edits were silently ignored), with bats
coverage for both the mapped and the no-suite case.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The CNPG strategy template renders <app>-cnpg-backup-creds / -ca against
whichever application it drives, so a to-copy restore looks the pair up
under the TARGET's name. The harness only created the source's pair; on a
cluster with stale target Secrets from an earlier run the restore failed
TLS verification against the wrong CA. Materialise the pair for the
target too (kubectl apply, so stale content is corrected) and delete both
pairs in cleanup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Two defects of the same class — the WAL-archive S3 prefix (serverName)
changing or defaulting differently between components:
Recovery default regression (review blocker): native CNPG defaulted an
absent externalClusters serverName to the entry name (= bootstrap.oldName),
but the barman-cloud plugin defaults it to the NEW cluster's name, so a
legacy-flow restore with bootstrap.serverName unset would read the wrong
S3 prefix (hard failure, or silently restoring another cluster's data).
The chart now always emits serverName, defaulting to bootstrap.oldName.
Live-cluster serverName flip (found by the gated postgres backup e2e on a
real cluster): the chart renders spec.plugins without serverName
(effective prefix = cluster name) while a strategy template may name a
different one; the driver's ForceOwnership patch then flipped the prefix
mid-stream, splitting the WAL archive across two prefixes — the WAL
holding the backup's start was lost between them and the restore failed
with "WAL not found". The driver now preserves the live Cluster's
effective serverName (explicit parameter, or the cluster name when the
plugin is attached without one) and only applies the strategy's value to
a Cluster with no plugin attached; both charts pin serverName explicitly,
and the example strategy no longer sets one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Port examples/backups/postgres into an automated e2e, following the etcd
backup precedent: the numbered example manifests become a runnable harness
(00-helpers.sh, run-all.sh, cleanup.sh) that a human reads and the test
drives, so the documented flow and the test can never drift.
run-all.sh provisions a Bucket, deploys the source Postgres (writing a
sentinel row), derives the two Secrets the barman-cloud ObjectStore
references from the Bucket's BucketInfo and seaweedfs' CA, submits an
ad-hoc BackupJob and waits for Succeeded, then restores to a copy and
asserts the sentinel round-tripped through S3 — the in-cluster witness
that a plugin backup is both taken and restorable end to end.
A second Test document (postgres-2-backup-roundtrip) in the existing
hack/e2e-chainsaw/postgres suite drives run-all.sh, gated out of CI behind
POSTGRES_E2E_S3_ROUNDTRIP=1 (kind has no reachable, CA-trusted S3 endpoint,
and a tenant Pod cannot cross the Cilium egress policy to tenant-root's
seaweedfs). Placing it in the postgres suite (rather than a new dir) keeps
it triggered by packages/apps/postgres changes under Test Impact Analysis.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The backup round-trip test could wedge for 20 min and then report a bare
EtcdSnapshot phase=Started with no root cause: dump_diagnostics dumped the
(healthy) EtcdCluster and the operator log, but the snapshot agent runs in
its own Job Pod, so the etcd-read / S3-upload error lives only in that Pod's
log and in the BackupJob/EtcdSnapshot status.
- dump_diagnostics now dumps the whole backup chain: BackupJob/Backup and
EtcdSnapshot describe, each snapshot Job's describe + Pod logs (selected by
the operator's etcd-operator.cozystack.io/cluster label), and recent
namespace events (the primary signal when a hung Pod logs nothing).
- wait_for_field gains an optional terminal-failure value; steps 04/05 pass
Failed so the BackupJob/RestoreJob waits return immediately on a terminal
phase instead of polling to the timeout — which also keeps the snapshot
Job Pod within its 600s TTL so its log is still readable at dump time.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Rewrite docs/agents/e2e-testing.md for the Chainsaw suite: declarative
asserts replace the until-get/wait backstops, Chainsaw cleanup replaces
test-level EXIT traps, catch blocks replace the `|| { ...; false; }` idiom,
and the TIA/diagnostics sections describe suites + `make test-chainsaw
CHAINSAW_SUITES`. Move the BATS->Chainsaw bullet out of 'in-flight' since it
is now the standard.
Fix references the migration left stale: hack/e2e-apps/ -> hack/e2e-chainsaw/
in the gemini styleguide and the nats/etcd backup READMEs. The
backup-postgres.bats / backup-foundationdb.bats files cited by the postgres
examples, the postgres README, and the foundationdb controller comment never
existed -- drop the false 'CI e2e drives this' claims and describe those
backup flows as the manual reference flows they are.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What
Implements the `strategy.backups.cozystack.io/Job` backup strategy — a
generic, application-agnostic driver that runs a user-supplied
Kubernetes Job to create (and restore) a backup.
## Background
This PR began as a draft written against an earlier shape of the backups
API. The API has since moved to a `BackupClass`-based dispatch model,
which broke the original draft on three counts:
- `BackupJob.Spec.StrategyRef` and `BackupJob.Spec.StorageRef` were
removed; the strategy and its parameters are now resolved from a
`BackupClass` into a `ResolvedBackupConfig`.
- Strategy handling moved from standalone per-strategy reconcilers to a
dispatch model: `BackupJobReconciler.Reconcile` /
`RestoreJobReconciler.Reconcile` resolve the strategy and switch into
per-strategy methods (`reconcileVelero`, `reconcileCNPG`,
`reconcileAltinity`, …).
- The draft's template context (`.Values`, `.Storage`) diverged from the
convention the other strategies adopted, and its `.Storage` half
targeted a storage API that does not exist.
## What changed
- **`reconcileJob` / `reconcileJobRestore`**
(`internal/backupcontroller/jobstrategy_controller.go`): replaces the
standalone `BackupJobStrategyReconciler` with the two dispatch methods
the `BackupJob` / `RestoreJob` reconcilers call. The Job strategy is the
generic counterpart of the Altinity driver — same lifecycle (render
`PodTemplateSpec` → run a one-shot `batch/v1.Job` → poll its terminal
condition → emit a `Backup`), minus the ClickHouse-specific
`applicationRef` gate, so it runs for any application. Shared package
helpers (`renderJobTemplate`, `jobConditionState`,
`getApplicationUnstructured`, status helpers, deterministic Job naming,
controller refs) are reused.
- **Template context** follows the in-tree convention used by the other
drivers: `.Application` (live application object), `.Release.Name` /
`.Release.Namespace`, `.Mode` (`backup` / `restore`), `.Parameters`
(resolved `BackupClass` parameters), and `.Backup` (restore runs only).
Strategy parameters round-trip through the `Backup`'s `driverMetadata`
so a later restore re-renders against the same values.
- **Restore** re-renders the same template with `.Mode == "restore"`.
Cross-namespace restore is unsupported (the RestoreJob target reference
is namespace-local), matching the Altinity driver.
- **`JobSpec.Template` doc + regenerated CRD**: the field doc previously
advertised `.Values`; it now documents the real context, and
`strategy.backups.cozystack.io_jobs.yaml` is regenerated to match (the
change is limited to that field's description).
- **Tests** (`jobstrategy_controller_test.go`): backup-Job creation /
args / labels / status, completion → `Backup` artifact with parameter
round-trip, Job-failure surfacing, missing-strategy terminal failure,
restore into the target namespace, restore completion, and the
parameter-prefix unit. A deliberately generic application kind pins that
the driver never gates on `applicationRef.kind`.
No RBAC changes are required: the controller already holds
`strategy.backups.cozystack.io/*`, `batch/jobs`, `apps.cozystack.io/*`,
and `backups` permissions.
## Notes for reviewers
- Backup and restore share one template, so a strategy author's
`PodTemplateSpec` must branch on `.Mode` to do anything
restore-specific. This matches the existing convention; worth confirming
it's the intended contract for the Job strategy's docs/examples.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Implemented full backup/restore job strategy with deterministic Job
creation, idempotent execution, and status/condition updates; added demo
example scripts and reusable shell helpers for a NATS JetStream
backup/restore workflow.
* **Documentation**
* Clarified Job template rendering context and listed supported template
variables; added README and step-by-step scenario docs for the NATS
example.
* **Tests**
* Added comprehensive unit tests covering job creation,
completion/failure handling, parameter extraction, and unmappable-target
failure cases.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/1721?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add examples/backups/nats/, the first worked example of the generic Job
backup strategy. It backs up a NATS JetStream stream that has no dedicated
driver, using a stock natsio/nats-box image: 'nats stream backup' -> tar ->
curl --aws-sigv4 PUT to the COSI/seaweedfs S3 endpoint, and the reverse for
restore. One PodTemplateSpec serves both directions, branching at runtime on
{{ .Mode }} (the engine templates each string leaf independently and cannot
restructure the Pod per mode).
Mirrors the clickhouse example's structure: numbered scripts (bucket -> NATS
src + seeded stream -> strategy -> BackupClass -> BackupJob -> in-place and
to-copy RestoreJob), shared helpers, run-all.sh, README, and scenario docs.
Like every examples/backups/* dir it is a runnable demo, not CI-gated.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
This patch implements the Reconcile function for BackupJobs with a
Velero strategy ref.
### Release note
```release-note
[backups] Implement the Velero backup strategy controller.
```
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Aleksei Sviridkin
