An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.
1. Archive-vs-recovery serverName collision. The driver set the restored
cluster's WAL-archive serverName equal to the recovery source
(bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
chart pinned the archiver serverName to the release name. In place those
coincide, so the cluster archived onto the source's own WAL prefix and
barman-cloud-check-wal-archive refused to start ("Expected empty
archive"). Fixed by giving the restored cluster a fresh
bootstrap.newServerName, keyed by the RestoreJob UID (unique per
restore, stable across reconciles); the chart archives there while
externalClusters still recovers from bootstrap.serverName. This is what
the manual CNPG strategies hit (serverName defaults to the cluster
name); the cozy-default flow dodged it (serverName is <ns>-<app>).
2. useSystemBucket guard. The restore populates explicit S3 coordinates
(destinationPath / s3CredentialsSecret / endpointCA) but left
backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
incompatible with useSystemBucket" guard failed the HelmRelease upgrade
and the cluster never re-rendered. This is what the cozy-default
(system-bucket) flow hits on an in-place restore. Fixed by clearing
useSystemBucket in the restore patch, since the app is now on the
explicit-creds path. The driver's app view (postgresapp.Backup) gains
the field, without omitempty so a false survives the merge patch.
Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 5bcdb8cfb8)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
The backup roundtrip e2e copied the seaweedfs S3 CA from a Secret named
seaweedfs-system-ca-cert in tenant-root, a name captured off a dev
cluster. On current main the seaweedfs chart pins
fullnameOverride: seaweedfs, so the cert-manager CA Certificate's secret
is seaweedfs-ca-cert; the old name is absent and the first CI run of the
e2e failed at step 05a with 'secrets "seaweedfs-system-ca-cert" not
found'.
Default S3_CA_SECRET to seaweedfs-ca-cert and, when that Secret is
absent, auto-discover the seaweedfs CA Certificate (the seaweedfs-labelled
one with spec.isCA=true) and read its secretName, so a future
fullnameOverride change does not silently break the copy again. Fail with
a clear message when neither resolves. README and chainsaw comment
updated to match.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Ungate postgres-2-backup-roundtrip: instead of an opt-in env flag it now
runs whenever TIA selects the postgres suite (postgres changes, and every
release cut), like the other chainsaw tests. The blocker was network
placement, not capability: the S3 client is the barman-cloud sidecar
inside the Postgres Pod, and the isolated e2e tenant's Cilium egress
allowlist blocks it from tenant-root's seaweedfs — so the Test runs in
tenant-root (same-tenant egress) against the in-cluster S3 endpoint,
trusting its self-signed cert via the copied seaweedfs-system-ca-cert.
This is exactly the recipe validated end to end on a real cluster.
Also map examples/backups/<app>/ edits to that app's suite in
hack/select-e2e.sh (the round-trip executes those scripts as its
harness; previously such edits were silently ignored), with bats
coverage for both the mapped and the no-suite case.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The CNPG strategy template renders <app>-cnpg-backup-creds / -ca against
whichever application it drives, so a to-copy restore looks the pair up
under the TARGET's name. The harness only created the source's pair; on a
cluster with stale target Secrets from an earlier run the restore failed
TLS verification against the wrong CA. Materialise the pair for the
target too (kubectl apply, so stale content is corrected) and delete both
pairs in cleanup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Two defects of the same class — the WAL-archive S3 prefix (serverName)
changing or defaulting differently between components:
Recovery default regression (review blocker): native CNPG defaulted an
absent externalClusters serverName to the entry name (= bootstrap.oldName),
but the barman-cloud plugin defaults it to the NEW cluster's name, so a
legacy-flow restore with bootstrap.serverName unset would read the wrong
S3 prefix (hard failure, or silently restoring another cluster's data).
The chart now always emits serverName, defaulting to bootstrap.oldName.
Live-cluster serverName flip (found by the gated postgres backup e2e on a
real cluster): the chart renders spec.plugins without serverName
(effective prefix = cluster name) while a strategy template may name a
different one; the driver's ForceOwnership patch then flipped the prefix
mid-stream, splitting the WAL archive across two prefixes — the WAL
holding the backup's start was lost between them and the restore failed
with "WAL not found". The driver now preserves the live Cluster's
effective serverName (explicit parameter, or the cluster name when the
plugin is attached without one) and only applies the strategy's value to
a Cluster with no plugin attached; both charts pin serverName explicitly,
and the example strategy no longer sets one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Port examples/backups/postgres into an automated e2e, following the etcd
backup precedent: the numbered example manifests become a runnable harness
(00-helpers.sh, run-all.sh, cleanup.sh) that a human reads and the test
drives, so the documented flow and the test can never drift.
run-all.sh provisions a Bucket, deploys the source Postgres (writing a
sentinel row), derives the two Secrets the barman-cloud ObjectStore
references from the Bucket's BucketInfo and seaweedfs' CA, submits an
ad-hoc BackupJob and waits for Succeeded, then restores to a copy and
asserts the sentinel round-tripped through S3 — the in-cluster witness
that a plugin backup is both taken and restorable end to end.
A second Test document (postgres-2-backup-roundtrip) in the existing
hack/e2e-chainsaw/postgres suite drives run-all.sh, gated out of CI behind
POSTGRES_E2E_S3_ROUNDTRIP=1 (kind has no reachable, CA-trusted S3 endpoint,
and a tenant Pod cannot cross the Cilium egress policy to tenant-root's
seaweedfs). Placing it in the postgres suite (rather than a new dir) keeps
it triggered by packages/apps/postgres changes under Test Impact Analysis.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Rewrite docs/agents/e2e-testing.md for the Chainsaw suite: declarative
asserts replace the until-get/wait backstops, Chainsaw cleanup replaces
test-level EXIT traps, catch blocks replace the `|| { ...; false; }` idiom,
and the TIA/diagnostics sections describe suites + `make test-chainsaw
CHAINSAW_SUITES`. Move the BATS->Chainsaw bullet out of 'in-flight' since it
is now the standard.
Fix references the migration left stale: hack/e2e-apps/ -> hack/e2e-chainsaw/
in the gemini styleguide and the nats/etcd backup READMEs. The
backup-postgres.bats / backup-foundationdb.bats files cited by the postgres
examples, the postgres README, and the foundationdb controller comment never
existed -- drop the false 'CI e2e drives this' claims and describe those
backup flows as the manual reference flows they are.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Andrey Kolkov