An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.
1. Archive-vs-recovery serverName collision. The driver set the restored
cluster's WAL-archive serverName equal to the recovery source
(bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
chart pinned the archiver serverName to the release name. In place those
coincide, so the cluster archived onto the source's own WAL prefix and
barman-cloud-check-wal-archive refused to start ("Expected empty
archive"). Fixed by giving the restored cluster a fresh
bootstrap.newServerName, keyed by the RestoreJob UID (unique per
restore, stable across reconciles); the chart archives there while
externalClusters still recovers from bootstrap.serverName. This is what
the manual CNPG strategies hit (serverName defaults to the cluster
name); the cozy-default flow dodged it (serverName is <ns>-<app>).
2. useSystemBucket guard. The restore populates explicit S3 coordinates
(destinationPath / s3CredentialsSecret / endpointCA) but left
backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
incompatible with useSystemBucket" guard failed the HelmRelease upgrade
and the cluster never re-rendered. This is what the cozy-default
(system-bucket) flow hits on an in-place restore. Fixed by clearing
useSystemBucket in the restore patch, since the app is now on the
explicit-creds path. The driver's app view (postgresapp.Backup) gains
the field, without omitempty so a false survives the merge patch.
Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 5bcdb8cfb8)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
The backup roundtrip e2e copied the seaweedfs S3 CA from a Secret named
seaweedfs-system-ca-cert in tenant-root, a name captured off a dev
cluster. On current main the seaweedfs chart pins
fullnameOverride: seaweedfs, so the cert-manager CA Certificate's secret
is seaweedfs-ca-cert; the old name is absent and the first CI run of the
e2e failed at step 05a with 'secrets "seaweedfs-system-ca-cert" not
found'.
Default S3_CA_SECRET to seaweedfs-ca-cert and, when that Secret is
absent, auto-discover the seaweedfs CA Certificate (the seaweedfs-labelled
one with spec.isCA=true) and read its secretName, so a future
fullnameOverride change does not silently break the copy again. Fail with
a clear message when neither resolves. README and chainsaw comment
updated to match.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Ungate postgres-2-backup-roundtrip: instead of an opt-in env flag it now
runs whenever TIA selects the postgres suite (postgres changes, and every
release cut), like the other chainsaw tests. The blocker was network
placement, not capability: the S3 client is the barman-cloud sidecar
inside the Postgres Pod, and the isolated e2e tenant's Cilium egress
allowlist blocks it from tenant-root's seaweedfs — so the Test runs in
tenant-root (same-tenant egress) against the in-cluster S3 endpoint,
trusting its self-signed cert via the copied seaweedfs-system-ca-cert.
This is exactly the recipe validated end to end on a real cluster.
Also map examples/backups/<app>/ edits to that app's suite in
hack/select-e2e.sh (the round-trip executes those scripts as its
harness; previously such edits were silently ignored), with bats
coverage for both the mapped and the no-suite case.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Port examples/backups/postgres into an automated e2e, following the etcd
backup precedent: the numbered example manifests become a runnable harness
(00-helpers.sh, run-all.sh, cleanup.sh) that a human reads and the test
drives, so the documented flow and the test can never drift.
run-all.sh provisions a Bucket, deploys the source Postgres (writing a
sentinel row), derives the two Secrets the barman-cloud ObjectStore
references from the Bucket's BucketInfo and seaweedfs' CA, submits an
ad-hoc BackupJob and waits for Succeeded, then restores to a copy and
asserts the sentinel round-tripped through S3 — the in-cluster witness
that a plugin backup is both taken and restorable end to end.
A second Test document (postgres-2-backup-roundtrip) in the existing
hack/e2e-chainsaw/postgres suite drives run-all.sh, gated out of CI behind
POSTGRES_E2E_S3_ROUNDTRIP=1 (kind has no reachable, CA-trusted S3 endpoint,
and a tenant Pod cannot cross the Cilium egress policy to tenant-root's
seaweedfs). Placing it in the postgres suite (rather than a new dir) keeps
it triggered by packages/apps/postgres changes under Test Impact Analysis.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-up (cozystack#3300):
- docs/operations/backup-classes.md: add an operator-facing upgrade note for the
two platform-wide changes this migration carries — the CloudNativePG operator
bump to 1.28.1 (with the CRD-skew fix and the Helm-CRD-upgrade caveat) and the
LINSTOR scheduler admission-webhook removal, which makes data-locality
scheduling advisory cluster-wide (spec.schedulerName no longer auto-assigned;
opt in via explicit schedulerName). These affect more than Postgres backups and
were previously only noted in a chart values.yaml comment.
- hack/e2e-chainsaw/postgres/postgres.yaml: drop stale restic-era backup fields
(s3Region, s3Bucket, cleanupStrategy, resticPassword) that no longer exist in
the Postgres backup schema; use current fields (destinationPath, endpointURL,
retentionPolicy, 6-field schedule).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Non-blocking review notes from #2826:
- Add `# shellcheck shell=bash` to the four sourced `_lib/*.sh` helpers
(etcd-cleanup, remediation-guard, run-kubernetes, talos-image-cache) so
shellcheck stops flagging SC2148 (no shell directive) — they are only ever
`. sourced`, so there is no runtime interpreter to infer from a shebang.
- Add a `# TODO(chainsaw>0.2.15)` marker at the postgres Service asserts
explaining the `(ports[0].port)`/`(ports[*].port)` projection is a v0.2.15
number-literal-filter workaround (and that `ports[0]` checks only the first
port), so it gets reverted on a chainsaw bump; cross-references the same
workaround in kafka/mariadb/harbor and the e2e-testing.md gotchas section.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The (conditions[?type == 'Ready'])[0] indexed-filter idiom fails in
chainsaw v0.2.15 with 'field not found in the input object'. Use the
filter-as-list form instead. Verified: both tests pass end-to-end
against a live cluster.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Port two representative BATS E2E tests (postgres, bucket) to Kyverno
Chainsaw as a pilot for replacing the bash-based suite. Declarative
asserts replace the timeout/until polling boilerplate, catch blocks
add automatic failure diagnostics, and script steps cover the
imperative probes (psql, mc S3 access checks).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Andrey Kolkov