Two correctness gaps on the useSystemBucket flow.
Retention. Gating spec.backup.tasks out of the chart also dropped the keep
that psmdb task retention used, and an on-demand PerconaServerMongoDBBackup
is not covered by task retention, so nothing pruned the shared cozy-backups
bucket — a retention-swept Plan grew it unboundedly for every tenant at
once. Own the archive the way the Redis/Rabbitmq drivers own theirs: stamp
the operator's percona.com/delete-backup finalizer onto the backups minted
on this flow, and on Cozystack Backup deletion delete that operator CR and
wait for the finalizer to free the object before releasing. A legacy backup
(tenant's own bucket) keeps the pre-existing hands-off no-op. Adds the
delete verb on perconaservermongodbbackups and a skip-artifact-cleanup
escape hatch for a wedged operator, matching the sibling drivers. Also drops
the unused update verb from the psmdb cluster RBAC (server-side apply is a
patch, not an update).
Restore. resolveMongoDBBackupSource returned early off the live operator
Backup CR whenever it reported a destination, copying status.s3 as-is. On
this flow the operator can echo a destination with status.s3 empty (the
storage was injected, not declared), and the target has no declared storage
to resolve from either, so the restore stranded with no coordinates.
Backfill a missing live S3 block from the snapshot the BackupJob recorded.
Also correct the precondition godoc: the pbm agents start on
spec.backup.enabled alone; the declared storage is required by the backup
CR's resolution, not by the agents.
Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
- Drop `-k`: mirror the RabbitMQ S3 layer (ARTIFACT_URI as the single
source of truth, forcePathStyle honoured, whole base URL kept) so
curl --aws-sigv4 verifies TLS against the ACME endpoint the helper
returns. The derived endpoint is default-port, so the curl 7.76.1
--connect-to port workaround is gone too; this also fixes the IPv6
and path-prefix mangling in the old host-splitting.
- Own the artifact: a cleanup mode in the script plus cleanupKafkaBackup
wired into BackupReconciler.cleanupOnDelete delete a deleted Backup's
object (best-effort, waited on) instead of orphaning it, matching the
RabbitMQ precedent. Kafka has the identical own-your-object property.
- Restore probes existence with --list (fail closed), so an unreachable
broker aborts instead of being misread as "topic absent" and created.
- Hold the fixes with tests at the right level: a colliding topic pair
(audit.events / audit-events, distinct partition counts) in the e2e so
a revert of the \Q...\E literal-match breaks the round-trip, and Go
tests pinning the scope stamp and the cleanup Job.
- Correct the docs/comments: endpoint-shape row, the "cached on nodes"
claim, and the narrower curl 7.76.1 S3 envelope (empty-body payload
hash) that makes a small modern-curl client image the right follow-up.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
Delete the backup object when its Backup is removed. The Redis driver
owns its S3 object (one per BackupJob, no engine retention prunes it), so
it now spawns a cleanup-mode Job that deletes the object and holds the
Backup's finalizer until it completes, mirroring the Rabbitmq driver.
Without this a retention-pruned Plan grew the bucket unboundedly and the
key was lost with the CR.
Size the strategy Pod from backupStrategyController.redisBackupResources
so an admin can raise the restore ceiling (the throwaway loader holds the
whole dataset in memory) via chart values, which a direct edit of the
cluster-scoped, Helm-managed Strategy CR could not survive. The
loader-not-ready diagnostic now names the memory limit, the likeliest
cause, instead of only the image version.
Make the endpoint CA volume optional: nothing projects that Secret into
the app namespace, so a required mount wedged every backup Pod on
FailedMount; a missing ca.crt now falls through to the system CA store
and fails the handshake fast, which is not the same as skipping verify.
Promote the Pod-level activeDeadlineSeconds to the Job so a wedged run is
bounded once, not (backoffLimit+1) times. Use find, not ls, for the base
AOF so its no-match diagnostic is not dead code under pipefail.
Pin the object key reaching the rendered Job: both fixtures now render
--key={{ .ObjectKey }} and assert it, so zeroing the key (which left the
package green) is caught. Document .ObjectKey and the cleanup mode in the
CRD render-context, list cozy-default-redis among the Helm-managed CRs,
and mirror the CA/TLS and find fixes into the example strategy.
Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.
Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):
- reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
--aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
- A Go-read precondition gates each run on the RabbitmqCluster reporting
AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
precise Ready=False wait instead of a Job that starts and fails.
- The produced Backup records the object location on status.artifact.uri
(rendered from spec.artifactURITemplate; the Backup CRD has no status
subresource, so it persists on Create, as in the Etcd driver).
Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.
Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.
The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Address branch-review recommendation: give the MongoDB strategy an explicit
case in BackupReconciler.cleanupOnDelete, matching CNPG / Altinity / MariaDB /
FoundationDB. A deleted MongoDB Backup artifact previously fell through to the
Velero default (a safe no-op — MongoDB driverMetadata never carries
velero.io/backup-name), but the explicit branch documents the "we do not own
the psmdb S3 archive" contract and guards the seam against a future shared
helper incidentally stamping velero.io/backup-name onto MongoDB metadata.
Ships with TestBackupCleanup_MongoDB_DoesNotFallThroughToVelero, mirroring the
one-test-per-branch convention its three siblings follow: it seeds the velero
metadata keys + a matching velero.io/Backup and asserts no DeleteBackupRequest
is synthesised and the Velero Backup survives — so deleting the branch makes
the test fail, which is what makes the guard self-verifying.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Andrey Kolkov