Commit Graph
157 Commits
Author SHA1 Message Date
Andrey Kolkov 88073bb209 fix(backups): fail Velero RestoreJob rename for non-VMInstance kinds
A RestoreJob that set targetApplicationRef.name to a new name and
options.targetNamespace to another namespace reported Succeeded for any
Velero-backed application, but only a VMInstance was actually renamed.
postRestoreRename looks up the HelmRelease under the vm-instance- prefix,
gets NotFound for every other kind and returns nil, so a Postgres,
MariaDB, ClickHouse, MongoDB or standalone VMDisk copy came up under the
source name with no error surfaced.

Renaming those kinds through the Velero strategy cannot work: DataUpload
restores volumes under their original names, and only a VMInstance keeps
using them after its HelmRelease is renamed, because its disks keep their
names and CDI adopts the restored PVCs. A renamed database Cluster or
VMDisk renders fresh volumes and comes up empty.

Reject the rename up front for every kind other than VMInstance, before
the RestoreJob enters Running, with a message that points at omitting
targetApplicationRef.name or using the engine-native strategy. The
existing same-namespace guard keeps precedence, so its message is
unchanged.

Closes #3537

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-23 21:30:12 +04:00
Andrey Kolkov 8849aa13de fix(mongodb): keep a completed archive reachable and a legacy Backup unblocked
Once the operator CR is ready the archive exists, and the Backup object
is the only thing that can ever reach it again. A transient failure
creating that object (etcd timeout, quota, webhook) failed the job
terminally with no artifact, and the terminal-phase guard stopped any
retry, so the dump sat in the shared bucket with nothing to reclaim it.
The failure is now retried with backoff for a window after the dump
completed, and when it does give up the job names the operator CR that
still holds the archive.

Cleanup read the operator CR before deciding whether the Backup owned
anything, so a legacy Backup — an unconditional no-op before this driver
— could be held Terminating by a throttle or a revoked verb on a CR the
driver never owned. The snapshot records the flow, so a legacy Backup
now releases before any apiserver read; a missing or undecodable
snapshot still falls through to the read.

The opt-out refusal recognised the lingering injected entry only by the
strategy's current credentialsSecret and only while the strategy still
carried s3, so a platform change to the strategy after a tenant opted
out let a legacy CR be minted onto the platform bucket. The platform
default Secret name is now recognised as well, independent of the
strategy's current block.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 17:24:32 +04:00
Andrey Kolkov f004b35fb4 fix(mongodb): record the CR's flow on the artifact and bound the two open loops
Reading a backup's flow off its CR's finalizer had reached the state
switch but not the artifact: the snapshot was still written from the
app flag, re-read on the completing reconcile. A flag flipped to false
while a system-bucket dump streamed produced a Backup whose snapshot
called a cozy-backups archive legacy, so the point-in-time refusal and
the S3 backfill both switched off for it. The artifact now takes the
same flow the switch does, pinned by a test that flips the flag between
mint and completion.

Two waits had no upper bound. A live-state read that kept failing at the
deadline deferred every poll behind a debug line with no condition, so an
unreadable CR pinned the job Running with nothing to act on; it is now a
named wait and fails after a grace, with the CR left in place rather than
deleted blind. The storage-race retry minted a fresh operator CR per poll
while the previous ones terminated, for the whole deadline; it now stops
after a small budget and surfaces the operator's error.

On useSystemBucket the chart already guarantees backup.enabled, so a
strategy without s3 coordinates failed with a remedy the tenant had in
place; the wait names the missing coordinates instead. The
missing-credentials message no longer blames useSystemBucket on a release
that never enabled it.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 17:24:25 +04:00
Andrey Kolkov 5e3b8c217b fix(mongodb): refuse useSystemBucket without backup.enabled and pin the retry's flow
The pbm agents run only when backup.enabled is true, so useSystemBucket
with backups disabled renders a cluster that can never service a
BackupJob, and the tenant learned that thirty minutes into the first job
when the driver's precondition gave up. The chart already fails closed on
the bootstrap clash; do the same here so the combination is a render-time
error with the remedy in it.

The storage-race retry keyed on the app flag while everything else in the
state switch had moved to the flow recorded on the CR's own finalizer, and
that flag term was the only thing keeping a legacy tenant's operator CR
from being deleted and retried on the same operator error — with no test
holding it. Key it on the CR's flow like its neighbours and pin the legacy
side: the error is terminal and the CR survives.

The retry matches the operator's own error wording, which is correct at
the pinned crVersion and nothing else ties together: a bump that rewords
it turns the retry off with the suite still green. Name the coupling at
the pin and at the matcher so a bump re-checks it.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 4233e90792 fix(mongodb): decide a backup's flow from its own CR, not the app flag
The opt-out refusal ran before the lookup of the job's existing operator
CR, so it was a per-poll gate rather than a mint-time one: flipping
backup.useSystemBucket to false while a dump streamed routed every later
reconcile into the refusal, failed the job at the deadline and left the
CR minted under the old flag streaming into the platform bucket with a
finalizer no Backup would ever resolve — the stranded archive the running
exemption exists to prevent. The refusal now applies only while the job
has no CR, and everything after the mint reads the flow off the CR's
delete-backup finalizer, which is fixed at mint time, instead of the
mutable app flag. A late dump on the legacy flow is likewise handled by
its CR's flow: the deadline fails the job and leaves the operator CR, as
before this driver existed.

The refusal also identified the injected entry by bucket name and told
the operator to wait for the chart to re-render it, which the docs say
never happens. The entry is now recognised by its credentialsSecret, the
one field on it the driver chose (bucket names collide with an admin's
external bucket and drift when the platform bucket is re-provisioned),
and the message names the manual removal the docs prescribe.

At the pinned operator `requested` is set right after the command is
sent to pbm, so deleting a CR in that state cannot recall it and a late
dump would land with nothing to own it. Only an unobserved or queued CR
is cancelled; a requested one is left as the handle to whatever pbm
writes, and the job fails.

The credential pre-check failed at once where every sibling precondition
waits for the restore deadline, and it re-ran against a restore already
replaying, so a Secret pruned mid-replay marked an in-progress restore
Failed. It now waits like its neighbours and is skipped once the operator
restore exists. The status-write suppression on the waiting helper is
pinned by counting status updates across repeated reconciles.

The legacy deadline's justification claimed psmdb task retention covers
the archive; the operator prunes only its own scheduled backups, so the
object is the tenant's to reclaim, and the comments say that instead. The
bootstrap refusal names the value that actually clears it
(bootstrap.enabled is not reset after a bootstrap restore).

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 42a29e3597 fix(mongodb): keep the legacy deadline on a running dump and name the wait
Exempting state=running from the wall-clock deadline was justified by the
shared bucket: on the useSystemBucket flow a failed BackupJob leaves pbm
streaming an archive no Backup object represents. The exemption was
applied to both flows, though, so a legacy dump whose operator or node
died mid-run went from failing on the deadline to polling forever with no
condition and no Event. Gate the exemption on the flow: a legacy archive
lands in the tenant's own bucket under psmdb task retention and the
pre-existing contract failed the job while leaving the operator CR alone,
so that is restored. On the system-bucket flow the wait is now named on
the BackupJob (Ready=False with the start time) instead of being silent;
the waiting helper writes status only when the condition changes so a
dump that runs for hours does not cost a status write per poll.

The point-in-time refusal read the snapshot flag with the decode error
discarded, so an undecodable snapshot fell open and a recoveryTime restore
of a system-bucket archive reached the operator to die on a pbm-level
message. Treat a decode failure as "flow unknown" and refuse.

Adopting a target credential only for a storage on the same bucket left
the source's own reference in place when nothing matched, and opting an
app into the system bucket prunes exactly that Secret while older Backups
still name it. Check the reference resolves before the operator sees it
and fail with a named reason instead of a missing-Secret error deep in
the restore.

Opting back out stops the injection but leaves the driver's storage on the
live cluster until the chart re-renders; a BackupJob in that window passed
the presence-only precondition, minted a CR without the delete-backup
finalizer and wrote an object nothing owns into the platform bucket.
Refuse to mint onto a storage whose bucket is the platform's while the
flag reads false, bounded by the deadline like the other waits.

Also realign the snapshot literal for gofmt, and fix two doc slips: the
pbm agents start on backup.enabled alone (a declared storage is needed
only to service a backup), and the opt-in cross-reference targeted the
parent heading instead of its subsection.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 69f80dbdc2 fix(mongodb): drop the false-positive stall detector and harden the cancel
The pinned psmdb operator persists a backup's status only when the state
string or error changes, so status.lastTransition freezes at the instant a
dump enters running and never advances while it streams. A stall detector
keyed on that field fires on every dump that outruns its window and raises
the same signal a genuine wedge would, so it cannot be acted on either
way. Drop it: a running dump is polled quietly and never cancelled, since
this CR carries no field a driver-side timer can use to tell a slow dump
from a wedged one.

The deadline cancel re-read the live state but could not tell "could not
read" from "not started": the helper returned "" for a nil client, a
throttle, a 403 and a real absence alike, and a live ready or error state
also fell through, so an unanswerable read or a just-completed CR was
deleted, taking a finished archive with it via the delete-backup
finalizer. The helper now returns an error, and the cancel deletes only on
a positive not-started answer, deferring on anything else.

The restore's point-in-time refusal keyed on the resolved credentialsSecret
name, which a BackupClass can set to any Secret; record the useSystemBucket
flag on the backup snapshot and key the refusal on that instead.

The storage-race retry deleted the errored CR, but the lookup kept
returning it while it lingered Terminating, so a fresh CR was never minted;
skip a CR carrying a deletion timestamp in the lookup.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 2293c0c073 fix(mongodb): bound and surface the useSystemBucket backup race-recovery waits
The useSystemBucket driver injects storage onto the live cluster at BackupJob
time and works around the psmdb operator resolving that storage from a cache
that has not yet observed the apply. Several of those waits were unbounded or
silent, and a PITR restore of a system-bucket backup reached the operator
despite no oplog ever being captured on that flow:

- the first-injection requeue now fails terminally once the backup deadline
  passes, instead of polling a cluster whose cache never reflects the storage
- a state=error the operator raises because it has not yet seen the injected
  storage is retried (delete the CR, requeue) within the deadline rather than
  failing the BackupJob on the driver's own race
- a backup wedged in state=running past the stall window surfaces Ready=False
  and a Warning once its status stops transitioning, so it no longer parks
  later BackupJobs in silence, and is never cancelled (that would strand the
  partial archive pbm keeps writing to the shared bucket)
- the timeout-cancel path re-reads the CR live before deleting, so a waiting CR
  whose live state just turned running at the boundary is not deleted mid-dump
- a failed delete-backup finalizer strip now emits a Warning naming the stuck
  object instead of the success-case event
- a recoveryTime restore over a system-bucket backup fails with a named error
  rather than handing the operator a point-in-time target it cannot serve

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov d091c9951c fix(mongodb): requeue after a fresh storage injection so the operator's cache can observe it
The first BackupJob of a useSystemBucket app SSA-injected the storage and minted
the operator PerconaServerMongoDBBackup in one reconcile. The psmdb operator
resolves spec.backup.storages from a cached cluster read when it services that
CR, and a cache that has not yet observed the apply takes the not-found branch
and latches the CR at State=error — which the operator never re-drives (it
returns for a terminal-state CR and watches only the Backup CR and Pods) and the
driver reads as a terminal failure. Only the first job per app is exposed; later
ones re-apply an already-declared storage.

When the pre-apply cluster did not already declare the storage, requeue after
the apply instead of minting in the same pass. The next reconcile finds the
storage in this driver's own cache (a good proxy that the operator's has caught
up too) and only then mints the CR — one 5s poll per app, well inside the
deadline. A fake client has no informer lag, so this is pinned by asserting the
first pass requeues and mints nothing rather than by reproducing the timing.

Also correct the RBAC delete-grant comment: the 30m timeout branch cancels a
not-yet-started operator CR on either flow, not just the useSystemBucket
cleanup, so a Failed BackupJob never leaves a CR the operator would run later.
And pin the wantBucket=="" restore-credential short-circuit with a no-bucket
source case.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov e653a4fa5d fix(mongodb): correct the finalizer-prune claims, pin the pre-read release CR, and fix the retention field text
The delete-backup finalizer prunes storage only for a `ready` backup at
v1.22.0; a timed-out backup never reaches `ready`, so the deadline-cancel
comment no longer claims the finalizer "takes anything partially written" —
nothing is written until the backup runs, which the cancel prevents. The
psmdbBackupTimedOut godoc no longer says the operator "has not started" the
backup, which is wrong for `requested` (the operator sets it as it dispatches);
it now describes the not-yet-running states it actually gates.

The pre-read CR handed to releaseMongoDBCleanup on the terminating path had no
test that failed when the argument was dropped, because the existing cases
either call the function directly or let the second Get succeed. Added a
cleanupMongoDBBackup case whose operator-CR Get fails after the first: the
release must strip from the passed object, not re-read, or the teardown wedges.

The useSystemBucket field said "retention move[s] to the Plan", but a Plan has
no retention field; corrected to "scheduling moves to a Plan; this flow has no
retention yet" and regenerated the schema, README and mongodb-rd. Softened the
"never pins a namespace in Terminating" wording in the cleanup godoc and RBAC
comment: the driver only bounds its own finalizer, the operator's release-lock
finalizer can still hold the CR while the operator is down.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov a89321fa8c fix(mongodb): keep retentionPolicy a genuine no-op, and cancel a timed-out backup so it cannot orphan an archive
Every prose surface (the useSystemBucket field, the retentionPolicy field, and
backup-classes.md) tells a tenant that schedule and retentionPolicy are ignored
on this flow, and cozyvalues-gen copies that into the schema, README and the
dashboard. A render-time reject of a non-default retentionPolicy contradicted
all of them: a tenant who followed the documented opt-in with a retentionPolicy
still set got a failed render and a HelmRelease that stopped converging, pointed
at the very doc that says the field does nothing. Drop the reject so
retentionPolicy is inert exactly as documented, handled the same way as schedule
(gated out, no error). The absence of retention on this flow is real but belongs
to a follow-up, so the docs now point at the BackupRetentionPolicy design (#4192)
and the teardown-GC issue (#3966), and recommend an S3 lifecycle policy until
they land.

Cancel a timed-out operator backup instead of only failing the BackupJob. The
deadline fires for a backup the operator has not begun streaming; but `waiting`
is the state pbm reports for a backup queued behind another, and the operator
starts it once the slot frees — after the BackupJob is Failed and never
reconciled again, writing an archive into the shared bucket that no Backup
object represents. The deadline branch now deletes the operator CR before
failing, which prevents that late run entirely: at v1.22.0 the operator returns
for a CR with a deletion timestamp before it dispatches to pbm, so nothing is
written. The delete is best-effort so it never blocks the failure.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov f8554d289d fix(mongodb): stop the system-bucket flow stranding archives, and reject an inert retentionPolicy
Two ways the shared cozy-backups bucket could accumulate objects nothing prunes.

A dump that outran the 30-minute deadline was declared failed while pbm kept
writing. The BackupJob's terminal-phase guard then never runs the branch that
mints the Cozystack Backup, so the archive lands with no Backup object to
represent it and no retention can reach it (the operator CR carries no ownerRef
either). Leave a running dump to finish — a real dataset routinely outruns 30m,
and a broken one still terminates through the operator's error/rejected state.
The deadline keeps bounding the case it was written for: the operator never
starts the backup (state stays "", requested or waiting), where nothing has
been written and failing strands nothing.

retentionPolicy is inert on this flow (the chart's tasks[].keep is gated out and
no Plan-level retention exists yet), so a value a tenant set expecting it to work
would grow the bucket silently. Reject a retentionPolicy changed from the chart
default when useSystemBucket is on, rather than ignore it; the default still
renders so an untouched install is unaffected.

Also on the cleanup path: hand the terminating-namespace release the operator CR
the caller already read and confirmed owned, so the finalizer strip is not gated
on a second Get whose transient failure would silently drop it and leave the CR
finalized in a namespace being torn down. Correct the ownership comments — the
delete-backup finalizer is not one the operator never sets (it finalizes
task-minted backups on the legacy flow); what keeps it unambiguous is that
cleanup only ever resolves the one CR named in driverMetadata. Add the missing
isolation test that a legacy app's own s3-storage is never overwritten by the
injection call site, and move the strategy's storage comment into a Go-template
comment so it no longer ships into every rendered Strategy CR.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov cc51e040ae fix(mongodb): keep the cleanup escape hatch off legacy operator CRs, and surface a stuck finalizer
The skip-artifact-cleanup annotation reaches releaseMongoDBCleanup before the
ownership check, and the operator-CR delete sat outside the finalizer gate, so
a legacy backup (no delete-backup finalizer) lost its PerconaServerMongoDBBackup
record when a tenant set the annotation to release a stuck Backup or applied it
across a namespace during teardown. That record is what a psmdb restore-by-name
resolves against, and both the RBAC comment and the docs promise it is never
touched. Gate every write in the release on the finalizer being present: an
unowned (legacy) CR or one that cannot be read at all is left intact, and the
Cozystack Backup still releases.

Surface a failed finalizer strip. The release must proceed even against an
unreachable CR, but the swallowed patch error left the operator CR holding
percona.com/delete-backup — pinning its namespace in Terminating, the outcome
the branch exists to prevent — while the Event reported the success case. Emit
a FinalizerNotStripped Warning naming the object so an operator can find it.

Mark the sibling backup fields (schedule, retentionPolicy, destinationPath,
endpointURL, insecureSkipTLSVerify, s3AccessKey, s3SecretKey) as ignored under
useSystemBucket, so a tenant editing one instead of the flag itself is not
misled by a description that describes only the legacy flow; regenerate the
schema, README and mongodb-rd. Pair the insecureSkipTLSVerify rendering
assertion with a case that flips the platform knob on, so the two together
discriminate the rendered value from a hardcoded default.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov d58322e253 fix(mongodb): let the system-bucket cleanup escape hatch survive an unreadable operator CR
The two give-up paths sat behind the operator-CR Get, so a Get that hard-fails
the way a removed CRD (NoKindMatchError) or a revoked verb (Forbidden) does
returned the error and never reached the release — leaving a Backup that keeps
its cleanup finalizer and requeues forever, exactly in the situation someone
sets the escape-hatch annotation to get out of.

Read the skip annotation before any apiserver call (as the Redis cleanup does)
so it always releases, and classify NoMatchError alongside NotFound (as the
Job/Redis cleanups do) so a missing CRD reads as "gone". releaseMongoDBCleanup
now takes the CR name and treats every apiserver step as best-effort, so the
release is never blocked by the same unreachable CR.

Also correct the docs, which promised an automatic release when "the operator
is down": only a terminating namespace or the annotation release. A down
operator in a live namespace keeps the Backup in Terminating as a visible
signal, which is what the code does.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 542f42ef67 fix(mongodb): make the system-bucket cleanup teardown-safe and single-source its ownership
Four issues in the system-bucket cleanup path.

Teardown wedge. The cleanup requeued forever waiting for the operator to
prune, and the escape-hatch annotation released the Cozystack Backup without
touching the delete-backup finalizer the driver stamps on the operator CR.
So a namespace teardown, where the operator's delete-backup step cannot run
(it execs against a pbm agent that is going away), left the operator CR — and
the namespace — stuck in Terminating with no way out but hand-patching
finalizers. Cleanup now detects a terminating namespace (as Redis/Rabbitmq
do) and, on that or the annotation, strips its own finalizer and releases,
leaving the object for a bucket lifecycle policy. Needs patch on
perconaservermongodbbackups for the finalizer strip.

Ownership was read from two places written at different times: the finalizer
(stamped when the operator CR is created) and a driverMetadata flag (written
when the artifact is created, a later reconcile). If the app's useSystemBucket
flag flipped between them they disagreed. Read ownership from the finalizer on
the one CR named in driverMetadata (the one the driver minted): resolving it by
name, never by scan, keeps the operator's own cron-* task-minted backups (which
do carry percona.com/delete-backup at v1.22.0) out of scope, so the finalizer on
that named CR is an unambiguous single marker. Drops the driverMetadata flag.

Injection error was terminal. A transient server-side apply — an apiserver
hiccup, or a conflict while the operator writes the same cluster — failed the
BackupJob outright. Requeue with backoff instead, bounded by the existing
deadline, matching the cluster/app read errors alongside it.

Injection ran before the enabled check, mutating a cluster that can never
back up. Gate it on backup.enabled so a disabled cluster is left untouched.

Also: reframe the restore S3 backfill as the defensive guard it is (the
pinned operator always echoes status.s3, so the live path normally carries
coordinates), and correct the psmdbtypes doc that still claimed the driver
never patches the operator CRs.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 1d05dafb2e fix(mongodb): own and prune the system-bucket archive, and heal the restore read path
Two correctness gaps on the useSystemBucket flow.

Retention. Gating spec.backup.tasks out of the chart also dropped the keep
that psmdb task retention used, and an on-demand PerconaServerMongoDBBackup
is not covered by task retention, so nothing pruned the shared cozy-backups
bucket — a retention-swept Plan grew it unboundedly for every tenant at
once. Own the archive the way the Redis/Rabbitmq drivers own theirs: stamp
the operator's percona.com/delete-backup finalizer onto the backups minted
on this flow, and on Cozystack Backup deletion delete that operator CR and
wait for the finalizer to free the object before releasing. A legacy backup
(tenant's own bucket) keeps the pre-existing hands-off no-op. Adds the
delete verb on perconaservermongodbbackups and a skip-artifact-cleanup
escape hatch for a wedged operator, matching the sibling drivers. Also drops
the unused update verb from the psmdb cluster RBAC (server-side apply is a
patch, not an update).

Restore. resolveMongoDBBackupSource returned early off the live operator
Backup CR whenever it reported a destination, copying status.s3 as-is. On
this flow the operator can echo a destination with status.s3 empty (the
storage was injected, not declared), and the target has no declared storage
to resolve from either, so the restore stranded with no coordinates.
Backfill a missing live S3 block from the snapshot the BackupJob recorded.

Also correct the precondition godoc: the pbm agents start on
spec.backup.enabled alone; the declared storage is required by the backup
CR's resolution, not by the agents.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-22 12:03:58 +04:00
Andrey Kolkov 47fe7c9630 test(mongodb): pin the restore credential swap and document the opt-out
The restore-time decision to re-point a backup's credentialsSecret at the
target cluster lived inline in reconcileMongoDBRestore with no test, so the
same-bucket guard that keeps a cross-flow restore from authenticating
against the wrong bucket could be dropped unnoticed. Extract it into
mongodbRestoreCredentialsSecret and cover all four
{legacy,system}x{legacy,system} combinations; neutralising the bucket
match now reddens the suite. The projected-secret skip is kept as well: a
target storage sharing the platform bucket but naming a different secret is
better restored with the platform-projected cozy-backups-creds than with
that secret.

Document the reverse toggle: flipping useSystemBucket true->false leaves
the driver-injected s3-storage on the live PerconaServerMongoDB under the
driver's field manager, which the chart does not own and never prunes. It
is inert once backups are disabled; the operator must delete it by hand to
return the release to the legacy flow.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-22 12:03:58 +04:00
Andrey Kolkov 9545bee233 fix(mongodb): correct system-bucket prose, pin strategy rendering, make restore credential swap flow-aware
Several comments and the useSystemBucket field description had drifted from
what the code does.

The strategy header still said the $bucketName gate was "not because it
needs the value" and that the sibling strategies needed the bucket name
while this one only asked for ordering, but the strategy now renders
bucket: {{ $bucketName }} for the injected coordinates, so it needs the
value; the precondition comment claimed the s3-storage entry exists on
backup.enabled=true, which is only the legacy flow. The rendering-test
comment carried the same "needs no bucket coordinates" claim. The
useSystemBucket field description said injection happens "at first BackupJob
time" (it now runs on every BackupJob) and that backups scope to
<namespace>/<release> (the prefix templates the application name, not the
release), and it ran ~700 chars of implementation detail into the CRD that
backup-classes.md already carries; trimmed and corrected, then regenerated
schema/README/types/cozyrds. Same <release> slip fixed in the driver
comment and the docs.

Nothing asserted the new strategy rendering. Added the MongoDB row to the
endpoint-form suite (s3.endpointUrl, scheme preserved both ways) and field
assertions to the rendering suite — bucket, endpointUrl, region,
credentialsSecret, forcePathStyle, and prefix, which must reach the CR as a
literal go-template string so the driver can template it per-application;
a stray brace would empty it with every doc-count test still green.

The restore credential swap broke across flows. It replaced the source
credentialsSecret with whatever the target declared, comparing no buckets,
so a system-bucket backup restored into a legacy target kept the platform
bucket against the tenant's own -s3-creds, and a legacy backup restored
into a system-bucket target got cozy-backups-creds against the tenant
bucket - both AccessDenied. Skip the swap when the source already names the
platform-projected cozy-backups-creds (it outlives the source app, nothing
to repair), and only adopt a target credential whose storage is on the
same bucket as the archive. psmdbStorageS3 now reads .s3.bucket alongside
.s3.credentialsSecret; TestPsmdbTargetCredentialsSecret covers the match,
the cross-bucket refusal, and the empty-bucket no-op.

Also drop the post-apply re-Get: the typed SSA Patch decodes the server's
merged object back into the passed struct, so the precondition reads the
injected storage from it instead of a follow-up Get that hit the informer
cache and briefly reported Ready=False on the happy path.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-22 12:03:58 +04:00
Andrey Kolkov 4ea4888685 fix(mongodb): key system-bucket injection off the app flag, not live storage
The MongoDB backup driver decided what to inject and what to record from
the live cluster's storage set rather than from the app's opt-in flag. That
had three consequences.

The snapshot fallback fired for legacy apps too. Its gate was
`snap.S3 == nil && rendered.S3 != nil`, but the coordinates moved onto the
shared cozy-default strategy, so rendered.S3 is now non-nil for every
MongoDB app. A legacy backup written to the tenant's own bucket got its
snapshot rewritten to the platform bucket, sending restore to the wrong
place. Gate the fallback on the app's backup.useSystemBucket flag instead,
threaded through createMongoDBBackupArtifact, and pin the legacy case.

The injected storage never caught up with the strategy. Injection was
gated on the named storage being absent, so it ran once per cluster and
froze the coordinates at the first backup; a later endpoint/region change
or bucket re-provision was a silent no-op, and Helm could not repair it
because the useSystemBucket chart renders no storages. Key injection off
the app flag and re-apply on every BackupJob (the prefix is deterministic,
so this never splits the archive the way CNPG's serverName would). Add
UseSystemBucket to mongodbapp.MongoDBSpec so a legacy app is protected by
the flag, not by whatever happens to be on the cluster.

The docs said the opposite of what the driver does. backup-classes.md
claimed "the MongoDB driver cannot inject its S3 target per-backup";
correct it and add an opt-in-to-the-system-bucket section that names which
values stop being read and that scheduled backups + PITR must move to a
Plan. Fix the endpoint table row, the tuning reference, the RBAC comment
that said the driver never patches, the strategy header/inline comments,
and the example comment. Also assert notExists on spec.backup.pitr in the
helm test, and lift the duplicated cozy-backups-creds default into a const.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-22 12:03:58 +04:00
Andrey Kolkov c036efaa5a test(mongodb): cover system-bucket storage injection gate and builder
Extract the two untested contracts from applyMongoDBSystemStorage into
pure functions and unit-test both, mirroring the CNPG driver's
buildBarmanObjectStore split:

- shouldInjectMongoDBSystemStorage decides whether the driver injects the
  system-bucket storage; the tests pin both branches, including the one
  that leaves a legacy app's manual bucket untouched.
- buildMongoDBSystemStorageEntry builds the {type:s3,s3:{...}} entry,
  defaulting credentialsSecret and omitting forcePathStyle when nil.

Also add a helm-unittest for the restore.yaml guard rejecting
bootstrap.enabled together with backup.useSystemBucket.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-22 12:03:58 +04:00
Andrey Kolkov 730592f323 feat(mongodb): support the system-bucket backup flow (useSystemBucket)
MongoDB was the only cozy-default app with no useSystemBucket flow: its
backup storage was always static on the app CR (bucket/endpoint/creds
from backup.* values + a chart-emitted <release>-s3-creds Secret), so a
tenant could not back up to the platform system bucket out of the box the
way postgres/clickhouse/etc. do.

PBM takes bucket/endpointUrl/region/prefix as static CR fields (not
secret refs) and the platform bucket name/endpoint are only known at
BackupJob time, so - unlike ClickHouse's runtime env secretKeyRef - the
chart cannot render them. The fix mirrors the CNPG driver: the chart adds
a useSystemBucket flag that keeps spec.backup.enabled (so the pbm agents
run) but omits the static storages/tasks/pitr and the creds Secret; the
cozy-default MongoDB strategy carries the coordinates (endpointUrl already
scheme'd by the endpoint helper, credentialsSecret cozy-backups-creds);
and the psmdb driver SSA-injects spec.backup.storages[s3-storage] onto the
live PerconaServerMongoDB at BackupJob time under its own field manager
with ForceOwnership, so a Flux re-render never reverts it. Injection is
gated on the storage being absent, so a legacy app that ships its own
static storage is left untouched. The backup snapshot falls back to the
strategy coordinates when the operator status echo is empty, so the
restore path rebuilds backupSource (credentialsSecret cozy-backups-creds
is projected into the restore namespace before dispatch).

RBAC grows patch/update on perconaservermongodbs for the injection. A
bootstrap.enabled + useSystemBucket render guard mirrors postgres.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-22 12:03:58 +04:00
Yan Bondarenko 4dc6065ec9 fix(backups): give the sidecar sizing its reason per caller
The Keycloak comment and the helper doc claimed the tenant LimitRange as the
reason for the resources, but cozy-keycloak has no LimitRange and neither
does a tenant that leaves resourceQuotas empty: there the sidecar had no
requests and no limit at all, and the change gives it a 1Gi ceiling where
there was none. The comments and both doc blocks now say what applies to
each caller. No rendered object changes.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-18 17:30:20 +04:00
Yan Bondarenko 9214430bd8 fix(backups): give the barman-cloud sidecar its own resources
The barman-cloud plugin injects its sidecar without resource requests or
limits. Tenant namespaces ship a LimitRange that defaults containers to
128Mi, so the sidecar inherits that default and is OOMKilled mid-backup:
the ObjectStore stays healthy, the Cluster stays Ready, and only the
backup fails. A cgroup high-water mark of 254 MiB on a 38 MB database
shows the default cannot hold it.

The shared cozy-lib helper and the Go path that builds the platform's own
ObjectStore now request 100m/256Mi and cap memory at 1Gi. No CPU limit is
set, matching the entityOperator precedent in the kafka chart: a
throttled sidecar stalls WAL archiving instead of failing it.

The helper is renamed from checksumSidecarConfiguration to
sidecarConfiguration, since it no longer carries only the checksum pin.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-18 17:30:20 +04:00
Andrey Kolkov ca2a6d6429 fix(backups): warn when a Kafka backupTimeout is rejected
kafkaRunDeadline silently substituted the 30m default for a backupTimeout
that time.ParseDuration rejects or that falls below the one-minute floor,
with no Event, condition or log line. An operator who set an unparseable
or sub-minute value, then raised it, saw every run keep dying at
DeadlineExceeded with nothing naming the discarded parameter.

Return the applied deadline together with a message when the configured
value is rejected, and record it as a BackupTimeoutInvalid Warning Event
on the BackupJob and RestoreJob (the reconcilers already hold a Recorder).
An honoured or unset value stays silent. TestKafkaRunDeadline now asserts
the warning fires for the rejected cases and not otherwise.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-14 17:55:48 +04:00
Andrey Kolkov cca3b08733 test(backups): pin the Kafka restore path and fail fast on a missing artifact
The restore half of the driver - the half
that mutates a live Kafka cluster - had no test reaching its happy path,
so three reversions of it left the suite green.

- Add restore reconcile tests mirroring the Rabbitmq driver this was
  copied from: ToCopy (pins the targetApplicationRef.Name merge - a
  to-copy restore must resolve the TARGET, not write the source's topics
  back into the source), NoRecordedArtifactURI, TargetClusterNotReady and
  TargetNotFound. Verified non-vacuous: neutralising the name merge reddens
  ToCopy, and reverting the fail-fast ordering below reddens
  NoRecordedArtifactURI.

- Fail fast on a Backup with no status.artifact.uri: move the check ahead
  of the readiness gate so it terminates immediately instead of waiting out
  the whole deadline and then reporting a misleading "timed out waiting for
  Kafka cluster to become Ready" (reconcileRabbitmqRestore checks it first;
  same order here).

- Add bats cases for the fail-closed guards that no test reddened: MODE
  validation, the s3:// URI prefix, cleanup treating 404 as success, an
  unparsable backup/live describe, a kafka-configs describe error, and an
  unrecognised record kind (plus a CFG_FAIL knob in the configs stub). The
  describe-parse guards abort via `set -euo pipefail` at the
  `X=$(describe_field ...)` assignment, so these pin the fail-closed
  behaviour; the explicit guards remain as belt-and-braces.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:53 +04:00
Andrey Kolkov 38d6c4cf58 fix(backups): bound Kafka metadata run at the Job layer, not a controller wall clock
A controller-side running-Job deadline was enforced in reconcileKafka /
reconcileKafkaRestore: at kafkaDefaultBackupDeadline from StartedAt the
BackupJob/RestoreJob was marked Failed. But the batch Job carried no
activeDeadlineSeconds and markBackupJobFailed only writes status, so the
Job kept running, finished, and its final `s3 PUT` uploaded the object —
while the terminal phase meant no Backup was ever created and cleanup
(keyed on Backup deletion) never ran. Every timed-out run thus orphaned a
uniquely-keyed object, and a cluster whose export legitimately exceeds the
deadline could never produce a usable backup. The shipped docs also
claimed the run "keeps the BackupJob Running ... rather than failing on a
wall clock", which the deadline made false.

Move the bound onto the Job as activeDeadlineSeconds (setKafkaJobDeadline):
Kubernetes fails an unschedulable or wedged Job as DeadlineExceeded and
kills its pod before the upload, so a timed-out run lands in the JobFailed
branch cleanly and leaves no object behind. Make it operator-tunable with
the `backupTimeout` BackupClass parameter (a Go duration), which
round-trips through the Backup so a restore is bounded by the same value;
the readiness-precondition wait honours it too. Correct the docs.

Also requeue on a transient broker-image List error instead of failing the
whole run: resolveKafkaBrokerImage now wraps a sentinel (errNoKafkaBroker)
so only a Ready cluster with no listable broker is terminal.

Tests: pin the deadline on the built Job, the backupTimeout override, a
JobFailed run marking the owner Failed, and the transient-List requeue.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:52 +04:00
Andrey Kolkov 4925a84c26 fix(backups): pin the restore-side guards and bound a running Kafka Job
- delete_topic waited with `grep -qx` (BRE), so deleting `audit.events`
  matched the live `audit-events` and spun to its timeout — the e2e
  failed at step 25 on every run that reached it. Use `grep -qxF`.
- The two -k/--connect-to guards were written as `! grep`, which cannot
  fail under hack/cozytest.sh (it wraps each @test in `set -e` + a
  trailing `return 0`, and bash exempts a `!`-negated command from
  errexit). Rewrite them as positive `if grep ...; then return 1; fi`;
  verified the -k reinstatement now reddens the suite under cozytest.
- Cover the restore-side literal-topic guard that was unpinned at every
  level: a bats case runs the existing-topic branch with a colliding
  sibling and asserts the `--alter` argv carries `\Q…\E`, and run-all.sh
  now leaves `audit.events` live so the e2e exercises that branch instead
  of only `--create`. Add bats cases for the partition-shrink refusal, the
  `-f` upload guard, and `--exclude-internal`.
- Bound a running Job: reconcileKafka / reconcileKafkaRestore now fail the
  BackupJob/RestoreJob once kafkaDefaultBackupDeadline elapses in the
  Job-running branch (buildJobStrategyBatchJob sets no
  activeDeadlineSeconds), so an unschedulable pod no longer pins it in
  Running forever. Pinned by TestReconcileKafka_FailsWhenJobNeverCompletes.
- Fail closed on an unknown record kind and an unknown MODE, and validate
  the live partition/RF parsed on restore (an empty value silently skipped
  reconciliation). Correct the stale "image pinned in the strategy" comment
  in the example helpers and the chainsaw test (the controller now resolves
  the target broker's image as .ClientImage).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:52 +04:00
Andrey Kolkov 0d918e1f91 fix(backups): resolve Kafka backup client image from the target broker
The Kafka metadata strategy pinned kafkaBackupClientImage to an upstream
Strimzi tag with no producer: nothing stamped it and Renovate does not
manage helm-values, so it drifted from the broker version the operator
provisions. A dedicated small client image (the sibling curl-only drivers'
follow-up) is not viable here because the driver runs the Kafka Admin-API
CLI, which only the broker image ships.

Resolve the client image from a live broker pod of the target Strimzi Kafka
at reconcile time instead, so the CLI always matches the broker (any version
the operator picks, not only the chart default), and record it on the Backup
so cleanup can reuse it once the source cluster is gone. The chart no longer
pins the image; the strategy template renders it as the .ClientImage
placeholder the controller fills.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:52 +04:00
Andrey Kolkov 14c6af122e fix(backups): restore Kafka S3 port workaround and hold the script with a test
- Regression fix: an earlier change dropped the curl --connect-to port workaround along
  with -k, but curl 7.76.1 (the image's) 403s on ANY ported endpoint,
  which is the shape backupStorage.endpoint ships (:8333) and what
  provisionBucket:false and s3.ingress.enabled:false produce. Restore
  --connect-to (sign a portless URL, redirect to the real port), keeping
  -k off so TLS is still verified. IPv6-safe host/port split, path prefix
  preserved. The port-less ingress passed; the ported case is the one that regressed.
- Correct the S3 envelope claim in values.yaml and backup-classes.md: the
  403 is decided by the port, not a body-hash the backend recomputes, and
  --connect-to handles it on SeaweedFS and MinIO alike.
- Hold the script fixes at the layer the contract lives at. Externalise
  the driver to files/kafka-backup.sh (embedded verbatim via .Files.Get,
  render byte-identical) and add hack/kafka-backup-script.bats, which
  EXECUTES it against stubbed kafka CLIs + curl and asserts behaviour: \Q
  literal match on the colliding pair, --connect-to with no -k on a ported
  endpoint, fail-closed --list, comma-value bracket grouping, RF-divergence
  abort, and the cleanup delete's HTTP handling. Not a template grep.
- Pin the Backup-delete dispatch and the readiness deadline in Go:
  TestCleanupOnDelete_Kafka_RoutesToKafkaCleanup and
  TestReconcileKafka_FailsAfterReadinessDeadline.
- Docs: config restore is additive (no --delete-config); export time
  scales with topic count and the Job has no wall clock (as its siblings).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:32 +04:00
Andrey Kolkov ed3cb34cbf fix(backups): adopt-guard Kafka Backup artifact and correct URI-template doc
- createKafkaBackupArtifact now rejects adopting a same-named Backup that
  describes a different application, so a retained Backup reused for a
  different app of the same name is not reported as this run's success.
  Mirrors the RabbitMQ driver's guard; pinned by a Go test.
- ArtifactURITemplate godoc said empty leaves status.artifact unset, but
  the driver hard-fails an empty template. Reword to state it is required,
  and regenerate the CRD description to match.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:07 +04:00
Andrey Kolkov 0270650237 fix(backups): harden Kafka metadata S3 handling and own the artifact
- Drop `-k`: mirror the RabbitMQ S3 layer (ARTIFACT_URI as the single
  source of truth, forcePathStyle honoured, whole base URL kept) so
  curl --aws-sigv4 verifies TLS against the ACME endpoint the helper
  returns. The derived endpoint is default-port, so the curl 7.76.1
  --connect-to port workaround is gone too; this also fixes the IPv6
  and path-prefix mangling in the old host-splitting.
- Own the artifact: a cleanup mode in the script plus cleanupKafkaBackup
  wired into BackupReconciler.cleanupOnDelete delete a deleted Backup's
  object (best-effort, waited on) instead of orphaning it, matching the
  RabbitMQ precedent. Kafka has the identical own-your-object property.
- Restore probes existence with --list (fail closed), so an unreachable
  broker aborts instead of being misread as "topic absent" and created.
- Hold the fixes with tests at the right level: a colliding topic pair
  (audit.events / audit-events, distinct partition counts) in the e2e so
  a revert of the \Q...\E literal-match breaks the round-trip, and Go
  tests pinning the scope stamp and the cleanup Job.
- Correct the docs/comments: endpoint-shape row, the "cached on nodes"
  claim, and the narrower curl 7.76.1 S3 envelope (empty-body payload
  hash) that makes a small modern-curl client image the right follow-up.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:06 +04:00
Andrey Kolkov 537aadf389 fix(backups): fix Kafka backup S3 signing and literal topic names
Reproduced end-to-end against a real Kafka 3.9.1 broker:

- [CRITICAL] The Strimzi image ships curl 7.76.1, whose --aws-sigv4
  omits a non-default port from the SigV4 canonical Host while sending it
  in the Host header, so every S3 PUT/GET 403s and the driver never
  uploads or downloads. Sign/send the bare host and redirect the
  connection to the real port with --connect-to (the same fix the
  topic-data example already carries), and accept the in-cluster
  self-signed cert with -k.
- [CRITICAL] kafka-topics --topic is a Java regex, not a literal: a
  backup entry for "audit.events" also matched a live "audit-events", so
  restore repartitioned an unrelated topic irreversibly and backup
  recorded the wrong partition/replication values. Wrap the name in
  \Q...\E on every filter call (--describe on both paths, --alter on
  restore) so it matches exactly one topic; --create and
  kafka-configs --entity-name are already literal. Applied in the example
  helpers too.
- [MAJOR] cozy-default's Kafka route looked like a data backup at
  runtime. Stamp scope=topic-metadata on the produced Backup's
  DriverMetadata and say "topic definitions and configs only; no message
  data" in the Ready condition, so kubectl describe backup is honest.
- [MAJOR] Add reconcile tests for the two Go behaviours a mutation could
  silently drop: the Strimzi-Ready precondition on backup (BackupJob
  stays out of Succeeded and creates no Job) and the applicationRef Kind
  gate on the restore target.
- [MINOR] Fail closed on the config-capture grep (exit != 1 aborts) and
  clarify the image pin has no automated producer.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:21:25 +04:00
Andrey Kolkov 405eb25095 fix(backups): harden Kafka metadata backup
- Fail closed on topic listing: set -o pipefail, capture kafka-topics
  --list --exclude-internal exit separately, and abort on failure so an
  unreachable broker can no longer produce an empty backup reported as
  Succeeded. An empty list after a successful call stays legitimate.
- Check each kafka-topics --describe exit and validate the parsed
  partition count / replication factor before writing a record, so a
  timed-out describe cannot write a corrupt orders\t\t\t line.
- Store topic configs one per line with key and value in separate
  tab-delimited fields, sourced from kafka-configs --describe (one config
  per line), so a value containing a comma (cleanup.policy=compact,delete)
  round-trips; on restore each config is applied with the value bracketed
  only when it contains a comma.
- Detect topic divergence on restore instead of a silent --if-not-exists
  no-op: grow partitions with --alter when the backup asks for more, and
  fail loudly (naming the topic and both values) when it asks for fewer
  partitions or a different replication factor. A restore that cannot
  reach the recorded state no longer reports Succeeded.
- Add KafkaStrategyKind to the RestoreJob cleanup-dispatch no-op group so
  deleting a Kafka RestoreJob does not run a pointless Velero cleanup; add
  a regression test alongside the siblings, plus a reconcile Kind-gate
  test and a rendering test that actually renders the strategy template.
- Wire the kafka-metadata Chainsaw suite into both selectors
  (select-install suite_to_source, select-e2e src_to_suites) so editing
  the kafka app or the suite selects it and the bats unit tests pass.
- Pin the client image to the operator's default broker version
  (kafka 3.9.1) with a digest, and align the example default; fix the
  values comment. Point the RabbitMQ comment references (which shipped in
  the CRD description) at the Job strategy actually mirrored, and trim the
  in-pod comments that rendered into the manifest.
- Document the Strimzi Topic Operator caveat: for a topic owned by a
  KafkaTopic CR the operator reconciles its config from the CR, so an
  in-place config restore is durable only for out-of-band topics.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:21:24 +04:00
Andrey Kolkov a410a3f48b feat(backups): Kafka topic-metadata backup via a dedicated strategy driver
Add a strategy.backups.cozystack.io/Kafka driver that backs up the topic
metadata of a Cozystack-managed Kafka application — partition counts,
replication factors and non-default topic configs — through the Kafka
Admin API, and restores it with kafka-topics --create + kafka-configs
--alter. Modelled on the Job/Rabbitmq strategies: the driver renders a
one-shot batch Job, gates on the Strimzi Kafka cluster reporting Ready,
and keys each backup by .BackupName so runs do not overwrite each other.

Scope is deliberately narrow: message payloads, consumer-group offsets,
ACLs, quotas and KafkaUsers are out of scope, and there is no PITR.
Restored topics are created directly (unmanaged on the target) — the
accepted tradeoff of an Admin-API-only driver. Both restore flows are
implemented: in-place recreates dropped topics into the source, to-copy
applies the source's topics onto a freshly-bootstrapped empty cluster.

Wire it into cozy-default (strategy-kafka-default.yaml gated on the
bucket name, a BackupClass binding, read RBAC on kafka.strimzi.io/kafkas
for the readiness gate, and the Strimzi Kafka client image), document it
in backup-classes.md, and add an honest round-trip example plus Chainsaw
e2e (examples/backups/kafka-metadata, kafka-3-metadata-roundtrip) that
proves a per-run retention.ms sentinel survives both restores.

Unit tests cover the driver's pure helpers (application-ref validation,
readiness verdict, parameter round-trip, per-run artifact-URI keying).

Not yet verified on a live cluster.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:20:27 +04:00
Andrey Kolkov b4d621f4f7 fix(backups): delete Redis backup objects on cleanup and size the restore Pod
Delete the backup object when its Backup is removed. The Redis driver
owns its S3 object (one per BackupJob, no engine retention prunes it), so
it now spawns a cleanup-mode Job that deletes the object and holds the
Backup's finalizer until it completes, mirroring the Rabbitmq driver.
Without this a retention-pruned Plan grew the bucket unboundedly and the
key was lost with the CR.

Size the strategy Pod from backupStrategyController.redisBackupResources
so an admin can raise the restore ceiling (the throwaway loader holds the
whole dataset in memory) via chart values, which a direct edit of the
cluster-scoped, Helm-managed Strategy CR could not survive. The
loader-not-ready diagnostic now names the memory limit, the likeliest
cause, instead of only the image version.

Make the endpoint CA volume optional: nothing projects that Secret into
the app namespace, so a required mount wedged every backup Pod on
FailedMount; a missing ca.crt now falls through to the system CA store
and fails the handshake fast, which is not the same as skipping verify.

Promote the Pod-level activeDeadlineSeconds to the Job so a wedged run is
bounded once, not (backoffLimit+1) times. Use find, not ls, for the base
AOF so its no-match diagnostic is not dead code under pipefail.

Pin the object key reaching the rendered Job: both fixtures now render
--key={{ .ObjectKey }} and assert it, so zeroing the key (which left the
package green) is caught. Document .ObjectKey and the cleanup mode in the
CRD render-context, list cozy-default-redis among the Helm-managed CRs,
and mirror the CA/TLS and find fixes into the example strategy.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 12:19:02 +04:00
Andrey Kolkov 919efa044f fix(backups): branch the Redis Job-existence check on IsNotFound
The readiness re-gating guard read `r.Get(...) == nil` to mean "the Job does
not exist yet", folding any non-NotFound error (an apiserver hiccup, an RBAC
denial) into that branch. On such a transient error the driver would re-enter
the readiness precondition even though the Job may already exist and have
completed, reintroducing the spurious terminal failure this guard removes.

Switch on the Get error: nil means the Job exists (observe it), NotFound
means run the pre-launch readiness gate, and any other error requeues without
deciding. Same on the backup and restore paths.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-08 17:57:06 +04:00
Andrey Kolkov ff061a8eea fix(backups): stop re-gating Redis readiness after the backup Job exists
The Redis backup/restore reconcilers checked the application's Ready
condition on every pass, ahead of the Job-state switch. Once the Job has
been created that re-check is wrong: if the app's Ready flips False after
launch (an unrelated HelmRelease blip) and stays so past the deadline, the
driver marks a run Failed whose Job may already have completed, a spurious
terminal failure of a successful backup or restore.

Gate the readiness precondition on the Job not yet existing; once it does,
observe the Job directly. The deadline still bounds the pre-launch wait.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-08 17:56:20 +04:00
Andrey Kolkov 632f7b31a0 test(backups): cover Redis backup-path unmappable kind
The restore path already asserts a terminal failure for an applicationRef
that passes the Redis gate but has no REST mapping; add the symmetric
backup-path case so both dispatch sides are covered.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-08 17:56:20 +04:00
Andrey Kolkov b0eb104276 feat(backups): add Redis backup/restore strategy driver
The spotahome RedisFailover operator ships no backup CRD, so add a
dedicated strategy.backups.cozystack.io/Redis driver that runs a one-shot
Job: it dumps an RDB snapshot from the current master (discovered via the
operator's sentinel Service) to the system bucket over S3, and on restore
replays it (RDB -> RESP via redis-cli --pipe) into the target's master.

Structurally a specialisation of the Job/Altinity script-Job pattern with
a Redis applicationRef gate and an app-readiness precondition. Backup
parameters round-trip through the Backup's DriverMetadata (no
underlyingResources snapshot); restore drives from .Backup.ApplicationRef
so a to-copy restore reads the source's object-storage prefix while
writing into a differently-named target. Both in-place and to-copy are
supported; PITR is not (RDB is a discrete snapshot).

- api: Redis/RedisList strategy type + generated CRD and deepcopy
- driver: reconcileRedis / reconcileRedisRestore + pure helpers, wired
  into both dispatch switches, supportedBackupStrategyKinds and the no-op
  restore cleanup case
- default: gated cozy-default-redis strategy (system-bucket coords via
  cozy-backups-creds env secretKeyRef, https:// scheme + optional CA) and
  the Redis route in the cozy-default BackupClass
- image: images/redis-backup bundling redis-cli + curl + an RDB->RESP
  converter, wired via backupStrategyController.redisBackupClientImage
- tests: driver unit tests (gate, precondition, render, param round-trip,
  both reconcile paths) and the helm rendering-gate coverage
- example: examples/backups/redis honest marker round-trip for both
  restore flows

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-08 17:56:20 +04:00
Andrey Kolkov e94fcf8219 feat(backups): RabbitMQ definitions backup via a dedicated strategy driver (#3990)
## What this PR does

Adds backup/restore support for the managed `RabbitMQ` application.
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so this backs up broker
**definitions** (vhosts, users, permissions, queues, exchanges,
bindings, policies, parameters) over the management HTTP API
(`GET`/`POST /api/definitions`) and stores them in the system bucket.
Message payloads and point-in-time recovery are intentionally out of
scope; a full-data backup would be a Velero volume snapshot, which is a
separate strategy.

It is delivered as a **first-class
`strategy.backups.cozystack.io/Rabbitmq` driver**, following the
Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

- `reconcileRabbitmq` / `reconcileRabbitmqRestore` render the strategy's
`PodTemplateSpec` — `curl` `GET`/`POST /api/definitions`,
pushing/pulling the JSON to S3 with `curl --aws-sigv4` — as a
`batch/v1.Job`, covering both restore flows: in-place (into the source
broker) and to-copy (into a separate, differently-named RabbitMQ in the
same namespace, authenticating with the target's own default-user
credentials and reading the source's object key).
- A **Go-read precondition** gates each run on the `RabbitmqCluster`
reporting `AllReplicasReady` (read through a small `rabbitmqtypes`
partial type), so a not-yet-ready broker surfaces a precise
`Ready=False` wait on the BackupJob/RestoreJob rather than a Job that
starts and fails against an unreachable management API.
- The produced `Backup` records the stored object's location on
`status.artifact.uri`, rendered from the strategy's
`spec.artifactURITemplate`. The `Backup` CRD has no status subresource,
so this persists on Create, mirroring the Etcd driver.

Dispatch is wired in both controllers (plus
`supportedBackupStrategyKinds`), `rabbitmqtypes` is registered in the
scheme, and RBAC grants read on `rabbitmqclusters.rabbitmq.com` (the Job
itself is already covered by the existing `batch/jobs` grant). It ships
`cozy-default-rabbitmq` (a `$bucketName`-gated `Rabbitmq` strategy) and
routes `apps.cozystack.io/RabbitMQ` to it in the `cozy-default`
BackupClass, which also enrols it in the `DefaultObjectsGate`. The
RabbitMQ application chart is left untouched — it had no prior backup
wiring, and the strategy carries the S3 coordinates itself.

Unit tests cover the driver's pure helpers (applicationRef validation,
cluster-name derivation, the readiness-message builder, artifact-URI
rendering, parameter round-trip) and the reconcile precondition gate
(defers with `Ready=False` and no Job until the cluster is ready, then
proceeds). A runnable example under `examples/backups/rabbitmq/` and an
honest-roundtrip Chainsaw e2e under `hack/e2e-chainsaw/rabbitmq/` seed a
sentinel vhost, back it up, drop it, and assert it round-trips through
object storage on both restore flows.

The Job pod reuses the existing `platform-migrations` image (curl +
ca-certificates) under a new
`backupStrategyController.rabbitmqBackupClientImage` key, stamped in
lockstep with `chBackupClientImage` in `packages/core/platform/Makefile`
so the repository stays pinned at a single digest
(`hack/image-pin-consistency.bats`). A dedicated slim image built by
this package is a follow-up.

**Validation:** `go build`/`go vet`/`gofmt`/`go test
./internal/backupcontroller/... ./internal/template/...`, deepcopy +
Rabbitmq CRD generation, `helm template` proving the CRD-ordering gate
both on and off, `helm lint`, and `helm unittest` (18/18). The backup +
in-place + to-copy roundtrip was verified end-to-end on a dev cluster (a
real management-API export and a real S3 `--aws-sigv4` upload/download
against SeaweedFS), so the driver's core contract is exercised on a real
cluster; the in-CI Chainsaw run is the remaining automated evidence. The
object-cleanup, terminating-namespace teardown and readiness-deadline
paths added during review are unit-tested (including `Reconcile`-level
finalizer retention and the terminating-namespace release) but not yet
cluster-exercised. (Note found while testing: RabbitMQ 4.2 is OOMKilled
on the `nano` preset, so the example/e2e use `c1.small`.)

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map against the diff. This is an additive backup
strategy plus a new strategy CRD; no downstream repository is
mechanically forced by the change. A documentation page for RabbitMQ
backup on [cozystack/website](https://github.com/cozystack/website)
would be a reasonable follow-up, but it is not required by this diff —
leaving the boxes unticked for a human to decide rather than ticking "no
downstream" to clear the checklist.

### Release note

```release-note
feat(backups): add RabbitMQ definitions backup/restore via a dedicated strategy.backups.cozystack.io/Rabbitmq driver; it gates each run on the RabbitmqCluster being ready and records the stored object on Backup.status.artifact.uri.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added RabbitMQ backup and restore support for definitions, including
vhosts, users, queues, exchanges, bindings, policies, and related
settings.
  * Added in-place and restore-to-copy workflows with readiness checks.
  * Added scheduled and on-demand backup examples.
  * Integrated RabbitMQ backups with the default backup configuration.
* Backups store definitions only; message payloads and point-in-time
recovery are not included.
* **Documentation**
  * Added a complete RabbitMQ backup and restore demonstration guide.
* **Tests**
* Added end-to-end coverage for readiness and backup/restore round
trips.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-03 16:03:46 +04:00
Andrey KolkovandClaude Opus 4.8 ba32df69c0 feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.

Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

  - reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
    PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
    --aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
  - A Go-read precondition gates each run on the RabbitmqCluster reporting
    AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
    precise Ready=False wait instead of a Job that starts and fails.
  - The produced Backup records the object location on status.artifact.uri
    (rendered from spec.artifactURITemplate; the Backup CRD has no status
    subresource, so it persists on Create, as in the Etcd driver).

Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.

Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.

The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-01 18:21:10 +04:00
Andrey KolkovandClaude Opus 4.8 5bcdb8cfb8 fix(postgres): let in-place restore re-bootstrap instead of wedging
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.

1. Archive-vs-recovery serverName collision. The driver set the restored
   cluster's WAL-archive serverName equal to the recovery source
   (bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
   chart pinned the archiver serverName to the release name. In place those
   coincide, so the cluster archived onto the source's own WAL prefix and
   barman-cloud-check-wal-archive refused to start ("Expected empty
   archive"). Fixed by giving the restored cluster a fresh
   bootstrap.newServerName, keyed by the RestoreJob UID (unique per
   restore, stable across reconciles); the chart archives there while
   externalClusters still recovers from bootstrap.serverName. This is what
   the manual CNPG strategies hit (serverName defaults to the cluster
   name); the cozy-default flow dodged it (serverName is <ns>-<app>).

2. useSystemBucket guard. The restore populates explicit S3 coordinates
   (destinationPath / s3CredentialsSecret / endpointCA) but left
   backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
   incompatible with useSystemBucket" guard failed the HelmRelease upgrade
   and the cluster never re-rendered. This is what the cozy-default
   (system-bucket) flow hits on an in-place restore. Fixed by clearing
   useSystemBucket in the restore patch, since the app is now on the
   explicit-creds path. The driver's app view (postgresapp.Backup) gains
   the field, without omitempty so a false survives the merge patch.

Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-25 17:31:53 +04:00
Andrey Kolkov b30c7f4272 fix(backups): requeue MongoDB restore on a transient source-CR read error
resolveMongoDBBackupSource folded two failure classes into one error return:
a transient API read error on the live PerconaServerMongoDBBackup CR (cache
not yet synced, apiserver hiccup) and the terminal no-restorable-destination
case. The RestoreJob caller turned every error into markRestoreJobFailed, so a
retryable read error failed a restore that would have succeeded on the next
reconcile. The backup path already treats the symmetric live-CR read error as
transient (return ctrl.Result{}, err).

Wrap the transient read error in a new errTransientRestoreSource sentinel and
requeue on it at the caller; only a genuinely unresolvable source stays
terminal. Adds a non-vacuous interceptor-driven test.

Addresses the MINOR robustness note from PR review.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Andrey Kolkov b97c443a16 fix(backups): harden MongoDB restore auth + PITR option handling from review
Address the two blocking findings in PR #3562:

- Restore into a differently-named target now authenticates with the
  TARGET cluster's own S3 credentials instead of the source's. The
  backupSource carried the source release's -s3-creds Secret, which is
  deleted together with the source app — the disaster-recovery case
  backups exist for — leaving a to-copy restore unable to reach S3. The
  restore replays into the target (gated on backup.enabled just above),
  whose own -s3-creds Secret is present and grants access to the same
  bucket; only the credential reference is swapped, bucket/prefix/endpoint
  still come from the source archive. Falls back to the source reference
  when the target declares no discoverable s3 credentials (in-place
  restore resolves to the same Secret, so it is a no-op there).

- A non-string recoveryTime (spec.options is free-form runtime.RawExtension,
  so {"recoveryTime": 20260805} passes admission) no longer silently
  degrades a PITR request to a plain snapshot restore reported Succeeded.
  When the typed decode fails and the blob carries a recoveryTime key, the
  RestoreJob now fails terminally — the same guarantee pitrSpec already
  makes for a malformed RFC3339 string.

Minor findings also addressed:

- BackupJob tolerates a not-yet-present MongoDB app CR with the same
  bounded grace the psmdb-CR existence gate uses, instead of failing
  terminally on a GitOps single-commit ordering race.
- examples: the to-copy target writes into its OWN S3 prefix
  (mongodb-target/) rather than the source's, so the two clusters' oplog
  and backup metadata no longer interleave under shared keys (both run
  replset rs0).
- examples: wait_psmdb_ready no longer treats a transient state=error as
  terminal — the cluster state is recomputed each reconcile and can blip
  through error during replset init before converging to ready.

Unit tests cover the target-credential selector (preferred/default/sole
storage) and the load-bearing recoveryTime-key detection.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Andrey KolkovandClaude Opus 4.8 0d4a3e3134 refactor(backups): make MongoDB PITR option uniform with CNPG + warn on unknown restore options
PITR: replace the bespoke spec.options.pitr{type,date} blob with a single
spec.options.recoveryTime RFC3339 string, matching the CNPG strategy's option
exactly (CNPGRestoreOptions.RecoveryTime). The driver converts recoveryTime to
the psmdb Restore's pitr{type:date,date} in the operator's "YYYY-MM-DD HH:MM:SS"
UTC format. An empty recoveryTime restores the snapshot as-taken (documented
difference from CNPG, whose empty value means latest WAL — a psmdb logical
backup is already consistent). A non-RFC3339 recoveryTime fails terminally.

Unknown options: parseMongoDBRestoreOptions now also returns the set of
spec.options keys the driver does not recognise; reconcileMongoDBRestore emits a
UnknownRestoreOption Warning event per unknown key (and logs it) instead of
silently ignoring it — so a "recoverytime" typo no longer silently degrades a
PITR restore to a snapshot restore. (BackupJob carries no spec.options blob, so
there is no backup-path equivalent; the only backup-time input is the free-form
BackupClass parameters used as template vars.)

Docs + examples updated to recoveryTime; tests cover the RFC3339→UTC mapping,
empty/invalid recoveryTime, and unknown-key reporting.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Andrey KolkovandClaude Opus 4.8 540bb94f58 test(backups): cover createMongoDBBackupArtifact producer seam
Address branch-review blocker: add TestCreateMongoDBBackupArtifact_ArtifactShape,
mirroring the MariaDB sibling. createMongoDBBackupArtifact writes the
driver-metadata keys (backup-name/namespace/destination), the Status.Artifact
URI, and the persisted snapshot that the restore path reads back — the
producer half of a seam previously exercised only end-to-end. A key-name drift
between this producer and resolveMongoDBBackupSource would otherwise pass every
unit test and surface only in e2e. The test asserts the metadata keys, phase,
artifact URI, and that the snapshot decodes to a usable by-reference
backupSource; a second case pins the empty-destination path (nil Artifact, no
destination key).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Andrey KolkovandClaude Opus 4.8 bc5e47c64f fix(backups): add explicit MongoDB cleanup-on-delete branch + its guard test
Address branch-review recommendation: give the MongoDB strategy an explicit
case in BackupReconciler.cleanupOnDelete, matching CNPG / Altinity / MariaDB /
FoundationDB. A deleted MongoDB Backup artifact previously fell through to the
Velero default (a safe no-op — MongoDB driverMetadata never carries
velero.io/backup-name), but the explicit branch documents the "we do not own
the psmdb S3 archive" contract and guards the seam against a future shared
helper incidentally stamping velero.io/backup-name onto MongoDB metadata.

Ships with TestBackupCleanup_MongoDB_DoesNotFallThroughToVelero, mirroring the
one-test-per-branch convention its three siblings follow: it seeds the velero
metadata keys + a matching velero.io/Backup and asserts no DeleteBackupRequest
is synthesised and the Velero Backup survives — so deleting the branch makes
the test fail, which is what makes the guard self-verifying.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Andrey KolkovandClaude Opus 4.8 dff56aa998 feat(backups): add a native MongoDB (psmdb) backup-strategy driver
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so
`apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and
ClickHouse native drivers — logical backups, restore-to-differently-named
instances, and point-in-time recovery through the unified
BackupJob/RestoreJob/BackupClass/Plan interface (closes #3545).

The driver orchestrates the operator's PerconaServerMongoDBBackup /
PerconaServerMongoDBRestore CRDs:

- BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls
  status.state, and persists the S3 destination + storage descriptor
  (by-reference, no credentials) onto the Cozystack Backup artifact.
- RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back
  from the source backup status, snapshot fallback) so the same artifact
  restores in-place or into a differently-named instance; PITR via
  spec.options.pitr (date/latest).

Because psmdb resolves the backup storage by name against the cluster's
spec.backup.storages, the driver names the chart's storage (s3-storage) via the
cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear
Ready=False precondition when a target has backups disabled.

Wiring:
- api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy.
- internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp
  partial-schema mirrors; dispatch, supported-kinds and scheme registration.
- backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb
  Strategy CR, and psmdb.percona.com RBAC.
- apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no
  CA-bundle field) so backups can target Cozystack's self-signed seaweedfs.

Tests + e2e:
- unit tests for the driver (validation, ensure-idempotency, precondition,
  backupSource resolution, snapshot round-trip, PITR option parsing).
- examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that
  drives run-all.sh end-to-end (sentinel document round-trips through S3 into a
  restored copy while the source stays untouched), per docs/agents/e2e-testing.md.
- docs/operations/backup-classes.md updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Mattia Eleuteri f859b710a4 fix(backupstrategy-controller): repair the bucket credentials Secret from the gate, not a chart fail
Addresses the four findings on #3524.

**The `fail` in the bucket chart (MAJOR).** The <bucket>-<user>-credentials
Secret was made to `fail` the render while its COSI lookup was empty, so the
release would retry and converge instead of skipping the Secret permanently.
Helm cannot render a partial set, so that fail aborts the WHOLE
<bucket>-system release: the other users' Secrets, and the bucket UI
Deployment/Service/Ingress/HTTPRoute with them. One declared user whose
BucketAccess never provisions takes down every other user of that bucket,
and on an already-installed release parks it in Failed, blocking every later
upgrade.

**No per-bucket escape (MAJOR).** Rather than add one, remove the need for
it. The gate already forces the release that renders the lookup-gated
Strategy CRs; the credentials Secret is the same trap one release earlier,
and the gate reads that Secret to resolve the bucket name in the first
place. So while it is absent, or carries no bucket name, force the bucket's
<bucket>-system release instead of doing nothing. That cannot deadlock its
own precondition: the BucketClaim and the BucketAccess whose COSI Secret the
lookup reads are rendered unconditionally by the parent release, not by the
one being forced.

The two releases are throttled independently — they are forced in sequence
on a bootstrap, so a shared timestamp would delay the second by a full
MinForceInterval for no reason. Coordinates come from the chart under
provisionBucket, and are omitted on external S3 where the Secret is
admin-managed and no release renders it. No RBAC delta: patch on
helmreleases is already cluster-scoped.

The chart goes back to a partial render, with the comment now explaining why
the skip is not self-healing and why failing is the wrong lever, and a test
that renders the UI with every user unresolvable so a reintroduced `fail`
fails the suite. Tenant buckets keep their pre-existing behaviour;
generalising the repair to every bucket needs a controller that owns them
and is tracked separately.

**Stale gauge (MINOR).** cozystack_backup_default_objects_missing was only
written on the happy path, so the state the gate exists to catch reported 0
and the documented alert never fired. It is now written on the unresolved
path too — the credentials Secret counts as one of the objects the default
backups depend on, which is what it always was. On an API error the gauge is
deliberately left alone rather than flapping, so a new
cozystack_backup_default_objects_check_errors_total marks it stale, and the
doc pairs the two instead of overstating the gauge.

**Suspended releases (MINOR).** forceHelmRelease now does a point Get and
skips a release with spec.suspend: true. helm-controller ignores forceAt and
requestedAt while suspended (`cozyhr suspend` sets exactly that), so the gate
was re-stamping every MinForceInterval for the whole suspension and climbing
the force counter — which the runbook attributes to a render that is not
producing the objects, the wrong diagnosis. The skip is logged and not
counted, so a climbing counter keeps its documented meaning.

Reported-by: IvanHunters <xorokhotnikov@gmail.com>
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-By: Claude <noreply@anthropic.com>
2026-08-07 11:10:41 +02:00
IvanHunters 8b006391c2 test(backupstrategy-controller): mirror all six cozy-default routes in the gate fixture
The gate reads the BackupClass as the manifest of what must exist, but the Go
fixture listed only CNPG, Etcd and the two Velero routes while the shipped
backupclass-default.yaml also routes MariaDB and Altinity. Add both to the
fixture, and to the RESTMapper and list-kinds derived from it, so the
expected-object set the tests exercise equals the real route set.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 10:09:04 +03:00