Commit Graph
186 Commits
Author SHA1 Message Date
Alexey Artamonov 0195a66fae fix(backups): release a MariaDB Backup when the operator CRDs are gone
Listing the operator Restores of a Backup before reading its operator
Backup made the cleanup fail on a cluster without the mariadb-operator
CRDs, retried for good, so the Backup and its namespace never went
away. The operator Backup read that follows already releases the
Backup in that case; with no Restore kind no restore can run, so the
listing now lets the cleanup reach it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:35:33 +03:00
Alexey Artamonov 8246808289 fix(backups): hold a MariaDB dump while an operator Restore reads it
The cleanup waited for a restore by the phase of its RestoreJob, but the
phase is not the restore: a RestoreJob fails on its deadline while the
operator Restore it created keeps running, and that Restore still reads
the dump. Deleting the dump then made mariadb-operator restore the
nearest other one.

The cleanup now also holds the Backup while an operator Restore of its
operator Backup has not been marked Complete, which mariadb-operator
does once the Restore's Job has completed or failed, so what keeps the
dump is the restore that reads it rather than the status of the
request.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:30:41 +03:00
Alexey Artamonov ee6404a0c8 fix(backups): retry a failed lookup of a MariaDB operator Restore
The lookup now goes to the API server, which can time out or throttle,
and a failed lookup failed the RestoreJob. It says nothing about
whether the operator Restore exists, so the RestoreJob could be failed
while its Restore was running, and the Backup's cleanup would then stop
waiting for it and delete the dump the Restore reads.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:28:45 +03:00
Alexey Artamonov 14a842ad16 fix(backups): find a MariaDB operator Restore the cache has not seen
The MariaDB restore creates its operator Restore once and then follows
it, finding it by label in the informer cache. Nothing watches operator
Restores, so a reconcile that ran before the new Restore reached the
cache saw none: it created a second one, or, now that a restore refuses
a Backup being deleted, failed the RestoreJob while its Restore was
still running, after which the cleanup deleted the dump that Restore
was reading.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:15:18 +03:00
Alexey Artamonov d57ebbe808 docs(backups): state what the MariaDB dump delete does not cover
The comment on who deletes a MariaDB Backup listed namespace teardown
without saying that teardown can remove the operator Backup first, in
which case the dump is left behind with a Warning, so it promised a
delete the code does not make. The remover contract claimed a missing
key is never an error, which holds only for a backend that answers the
delete with 204, the one status minio-go accepts.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:09:15 +03:00
Alexey Artamonov 3334152da7 fix(backups): report a recorded MariaDB dump that was not found
A recorded dump missing from the listing was logged as deleted, with no
event. That is right when an earlier attempt or maxRetention removed it,
but the listing goes through the platform's current endpoint, and after
an endpoint move the dump can still sit on the old storage: the cleanup
then deleted the operator Backup that pointed at it and said it had
deleted the dump.

The cleanup now records a Normal ArtifactAlreadyAbsent event naming the
key and the endpoint it listed, so whoever later looks for that dump
can tell where it was not found.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:08:43 +03:00
Alexey Artamonov bbeca34535 fix(backups): retry a failed read of the platform backup credentials
The MariaDB cleanup reaches the platform bucket through the platform's
current endpoint so that a Backup taken before the endpoint moved is
still deleted. A failed read of the platform credentials made it fall
back to the endpoint recorded on the operator Backup, which may be the
old address: there the dump is not found, a missing dump counts as
already deleted, and the operator Backup that pointed at the real dump
was deleted with a success log.

A read that fails with an API error now holds the Backup and retries.
A missing or malformed platform Secret still falls back, since it is a
standing state that a retry would not change.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:07:03 +03:00
Alexey Artamonov 37a74e5939 test(backups): reach the S3 calls the MariaDB cleanup timeout bounds
Contesting a recorded dump lists the bucket before the delete, and the
timeout test supplied no lister, so a real client failed on DNS before
the blocking delete was ever called and the suite stayed green with the
timeout removed. The test now lists the dump through the fake bucket,
blocks in the listing and in the delete in turn, and requires that the
blocking call was entered.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:05:51 +03:00
Alexey Artamonov 654c543264 fix(backups): see a MariaDB restore the cache has not caught up with
A MariaDB restore now checks on the API server that its Backup is not
being deleted before it starts, so a restore that started first exists
on the API server before the delete. The cleanup read RestoreJobs from
the informer cache, which can lag behind, and could delete the dump
under a restore it had not seen yet. Reading them uncached closes that
window from both sides.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:04:53 +03:00
Alexey Artamonov 320d0299ad fix(backups): refuse a MariaDB restore from a Backup being deleted
Deleting a MariaDB Backup now deletes its dump, and mariadb-operator
restores the dump nearest the requested time when the exact one is
gone. A restore that starts while the Backup's cleanup runs can lose
its dump between the check and the operator's read, and then completes
on another run's data without any error.

The restore now reads the Backup from the API server before creating
the operator Restore and fails when it is being deleted or is gone. The
copy the reconcile was handed comes from the cache and can predate the
delete; a transient read error keeps the RestoreJob waiting like any
other failed dump check.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 15:03:59 +03:00
Alexey Artamonov 74fbb34aca docs(backups): say who deletes a MariaDB Backup
The comment justified deleting a dump in a tenant's own bucket by the
tenant asking for it, but tenant roles grant no delete on Backups: only
an administrator or the namespace teardown deletes one, and the reader
of this branch should know whose request the delete answers.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 18:47:08 +03:00
Alexey Artamonov 0972690f0d fix(backups): contest a recorded MariaDB dump before deleting it
The size probe records the dump it attributes to a Backup, and it lets
only runs that had already completed contest the attribution. A run can
upload its dump and prune an older run's through maxRetention before its
own operator Backup completes, so the probe of the pruned run can record
the newer run's dump, and deleting that Backup then deleted a dump that
another Backup still restores from.

Deletion now contests the recorded dump against every run on the same
bucket and prefix as they stand at deletion time, the unfinished ones
included, and leaves it behind with a Warning when any could have
written it, as the unrecorded path already does. A run that never
completes therefore keeps every later dump its lifetime covers, which
leaks storage rather than losing a backup. The probe is unchanged here.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 14:03:21 +03:00
Alexey Artamonov e66b764734 fix(backups): let an unfinished MariaDB run block a legacy dump delete
A Backup taken before dumps were recorded finds its dump by window, and
only finished neighbour runs were allowed to contest a candidate. A run
uploads its dump and prunes older ones through maxRetention before its
operator Backup turns Complete, so in that gap the pruned run's cleanup
saw the newer run's dump as the only candidate and deleted it, leaving
that Backup Ready with nothing to restore.

The size probe and the restore keep ignoring unfinished runs, where a
wrong guess costs a size or a refused restore. Deletion now treats a run
that has not completed as one whose window is still open, so the dump is
left behind with a Warning, as for any other unproven attribution.
Holding the Backup would not help: once that run completes its window
still covers the candidate, and the attribution stays ambiguous.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:21:22 +03:00
Alexey Artamonov f147b22d86 fix(backups): delete a MariaDB dump when its Backup is deleted
Deleting a MariaDB Backup left its dump in the bucket for good:
mariadb-operator's maxRetention prunes only from inside the next run of
the same prefix, and a Plan never deletes a Backup, so on the platform
bucket nothing else ever removes it.

Every run of an application writes under one prefix, so the key deleted
has to be proven to be this Backup's. It is the dump the Backup records,
or, for a Backup taken before dumps were recorded, the one the existing
ownership match attributes to its operator Backup. That match is
recorded on the Backup before the delete, so a retry deletes the same
key by name instead of matching again against a bucket that no longer
holds it. Without proof the Backup is released with a Warning and
nothing is deleted, the operator Backup included, since any later
attribution starts from it. The operator Backup is deleted only after
its dump, and the delete waits while a RestoreJob still restores from
the Backup.

The delete runs in the controller through the same minio client the
size probe uses, so it verifies TLS the way mariadb-operator did when
it wrote the dump. A dump on the platform bucket, which the operator
Backup names by its bucket and by the projected credentials Secret, is
reached through the platform's current endpoint and keys, still
trusting a CA the strategy pinned, so a Backup taken before the
endpoint moved is still deleted.

Platform credentials that cannot be read fall back to the ones the
operator Backup names. A failed delete holds the Backup and retries
with a Warning naming the skip-artifact-cleanup annotation, except in
a terminating namespace, where the Backup is released with a Warning
so it does not hold the namespace.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:24:29 +03:00
Myasnikov Daniil 195de033e8 fix(mariadb): pin legacy Backups to a proven dump, refuse failed runs
A Backup taken before Backups recorded their dump gives a restore
nothing to pin. Refusing such Backups would leave every existing one
unrestorable, so a restore of one now makes the same ownership-proven
match the backup side makes, against the operator Backup it names, and
restores that dump by its timestamp. When no dump can be proven to be
the run's own, or the storage cannot be read, mariadb-operator restores
the newest dump under the prefix as it always did, and a BestEffortDump
Warning on the RestoreJob says so and why.

An operator Backup whose Job failed is Complete=True with reason
JobFailed, and a BackupJob that took Complete=True for success made a
Backup of it. That run left no dump of its own, and restoring it would
quietly bring back another run's dump, so the RestoreJob now fails
instead. The condition is read by its reason, because its status does
not tell a failure from a success.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 15:57:20 +00:00
Myasnikov Daniil 21d1e9f02b fix(mariadb): restore the dump a Backup records
A RestoreJob created a mariadb-operator Restore that named only the
operator Backup. The operator then lists the prefix that Backup wrote
to and restores the dump nearest to the moment the restore started:
the application's newest dump, not the one the Backup was taken as.

When the Backup records its dump, the Restore now carries the
timestamp in that dump's name as targetRecoveryTime. mariadb-operator
returns the dump whose timestamp equals the target before it looks for
the nearest one, and the nearest one is exactly what it would restore
if the recorded dump were gone. So the controller first checks, with
its own bounded call, that the dump is still in the operator Backup's
storage. A dump that is gone fails the RestoreJob, naming the dump,
instead of restoring a neighbour in its place. A check that cannot be
made keeps the RestoreJob waiting with Ready=False until the restore
deadline, then fails it with the cause: the Backup could record its
dump only because the controller reached that storage when it was
taken, so an unreachable storage is not a reason to guess.

The dump is chosen once, as the operator Restore is created; a Restore
that already exists is followed as before.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 15:53:38 +00:00
Myasnikov Daniil a042f33590 fix(mariadb): record the dump a Backup was taken as
Every run of a MariaDB application writes its dump under the same
storage prefix, and the Backup did not say which of those dumps is its
own. A restore had nothing to give mariadb-operator but the operator
Backup, which resolves to the newest dump under the prefix.

The Backup now records the dump's file name and the timestamp in that
name in driverMetadata, taken from the match the size probe already
makes. Nothing is recorded when that match cannot prove the dump
belongs to the operator Backup, so a Backup never names another run's
dump.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 15:49:35 +00:00
Myasnikov Daniil 59f0013b1f fix(backups): attribute a MariaDB dump only when no other run can own it
Every backup of a MariaDB application writes its dump under one shared
prefix, so the probe took the dump whose file-name timestamp (the
backup pod's clock) and LastModified (the object store's clock) both
fall in the operator Backup's lifetime, with a minute of slack for
skew. That keeps a neighbour's dump out only while this run's own dump
is in the window. With the pod clock more than the slack behind, the
own dump is named before the window opens, and a previous run that
finished within the minute before this one was created becomes the
single match: a 123-byte dump was reported as 666 bytes. A pod clock
ahead or an object store clock behind does the same with the next run,
and an own dump that is gone by the time it is looked up, as it can be
for a restore long after the backup, leaves a neighbour as the single
match with no skew at all.

A dump is now attributed only when ownership is proven:

- it is the one dump that either clock places in the window. A dump
  that one clock places there and the other does not may be this run's
  own with a skewed clock, so it leaves any other match unproven, the
  way a second dump in the window already did, and is not taken itself;
- both of its clocks place it in the window;
- neither clock places it in the window of another operator Backup
  that writes under the same bucket and prefix and has completed. This
  rejects a neighbour's dump when this run's own is missing, which the
  clocks alone cannot tell.

Operator Backups still running are not counted as neighbours: their
window has no end, and one stuck short of Complete (a Job whose pod
never starts stays JobRunning for good) would otherwise keep every
later run of the application from being sized. A dump such a run
writes inside this run's window still competes with this run's own.

Narrower slack does not prove ownership: a neighbour that finished
within the narrower slack still matches, and a smaller skew pushes the
own dump out. The Job's start and completion times are stamped by the
control plane, like the operator Backup's own, so they narrow the
window by seconds without removing the dependence on the two clocks a
dump carries. What is left: a neighbour's dump is taken only when this
run's own is missing from the listing or misplaced on both clocks and
the neighbour's operator Backup no longer exists or has not completed.
Every other unproven case leaves the size unset, as two runs that
overlap or follow each other within the slack already did.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:24:09 +00:00
Myasnikov Daniil 560c44f71e fix(backups): read a MariaDB dump from where its operator Backup wrote it
The size probe took the bucket, endpoint, prefix, credentials and TLS
settings from the strategy as it renders on the reconcile that creates
the Backup, which is not necessarily what it rendered when the operator
Backup was created: an existing operator Backup is reused untouched,
and a backup can run for up to the driver's 30-minute deadline. An
admin who edits the strategy in between sends the probe to the new
location, where it measures the wrong prefix or nothing at all.

The operator Backup's own spec.storage.s3 is where the dump went, so
the probe reads that. It is also the only source a restore can use,
since it targets that operator Backup long after the strategy may have
changed. An operator Backup that does not store to S3 is refused before
anything is listed.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:18:39 +00:00
Myasnikov Daniil 21f51a2e74 refactor(backups): return the matched MariaDB dump, not only its size
A MariaDB restore has to name the dump it restores from: without a
target time the operator restores the newest dump under the prefix,
which is not necessarily the one of the Backup being restored. The size
probe already finds that dump, but it returns only its size, folds a
missing, an ambiguous and an unreadable listing into the same 0, and
lives on BackupJobReconciler, which the restore reconciler is not.

Split the probe in two: findMariaDBDump reads the bucket and returns the
matched object (key, file name, the timestamp in its name, size and
LastModified) or an error that tells "not found" from "ambiguous"; the
size path keeps its timeout and reads only the size. The Secret lookup
and the S3 client move onto an objectStore value that any reconciler
can build from its own client, and BackupJobReconciler keeps one-line
wrappers so the CNPG probe is unchanged. sizeBytes is reported exactly
as before.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:16:15 +00:00
Myasnikov Daniil c140354977 test(backups): build operator Backups with the operator's success shape
mariadb-operator marks a Backup Complete=True for a failed Job as well
as a successful one; only the reason tells them apart (JobComplete with
message Success, or JobFailed with message Failed). The fixtures that
stand for a finished backup set Complete=True and nothing else, so they
describe a state the operator never produces and would read as a
failure to a driver that checks the reason. Give them the success shape
the operator writes.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:13:54 +00:00
Andrey Kolkov 92fa979690 fix(backups): trust only the endpoint CA where the backup writer does
With an endpoint CA the probe trusted the system roots plus that CA.
barman-cloud gets endpointCA as AWS_CA_BUNDLE, which replaces the
default trust store, so a CNPG probe could accept an endpoint the backup
itself would have refused. Let the connection say whether the CA adds to
the system roots or replaces them, and replace them for CNPG.
mariadb-operator appends its CA to the system pool, so MariaDB keeps
that.

Pin in reconcile tests that a Create the apiserver rejects still fails
the BackupJob, so only the read-back of an existing Backup is retried.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:23 +00:00
Andrey Kolkov 080df1b2f2 fix(backups): surface a Backup that cannot be read back and skip its probe
A Backup that exists but cannot be read back is retried through the
controller's backoff, which left the BackupJob Running with nothing on
it saying why. Set Ready=False with reason BackupNotReadBack before
handing the error back.

When the check for an existing Backup itself fails, the Backup may well
exist, so a size probe would only be discarded, and a read that keeps
failing reran it, up to the probe timeout on the single BackupJob
worker, on every retry. Skip the probe in that case and create the
Backup without a size.

Drive both behaviours through the MariaDB and CNPG reconciles in tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 523d74a503 fix(backups): retry instead of failing when an existing Backup cannot be read
When Create collides with a Backup an earlier reconcile already made and
the read that follows fails, the drivers failed the BackupJob, although
the backup and its Backup both exist. Return a distinct error instead,
which the drivers hand back to the controller so the reconcile is
retried with backoff and then finds the Backup.

minio.New already refuses an endpoint with a path prefix and accepts a
trailing slash, so drop the duplicate check in the shared client; the
CNPG probe keeps its own check because it parses the URL itself and
would otherwise drop the path before minio sees it.

Say in the sizeBytes API doc what the value covers, such as the base
backup without the shared WAL archive for CNPG, and that strategies
which cannot attribute stored bytes to one backup leave it unset.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 2b21763577 fix(backups): do not fail a backup on a transient retry-check read
The check for a Backup an earlier reconcile already created returned
its read error, and the drivers turn any error from that path into a
terminally Failed BackupJob. An apiserver timeout right after the
operator backup completed therefore lost a backup whose data was safely
in S3. The check only saves a probe, and the Create that follows already
handles an existing Backup, so log the error and go on to Create.

A trailing slash on the S3 endpoint is not a path prefix; the operators
accept it, so do not refuse it.

Pin the uncached AlreadyExists refetch, the upper slack of the MariaDB
window and the probe timeout with tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov a3d854739d fix(backups): attribute MariaDB dumps on two clocks and read retries uncached
The MariaDB probe identified this run's dump by the timestamp in its
file name, which comes from the backup pod's clock. With that clock
skewed past the slack, this run's dump falls outside the window while a
neighbouring run's dump can be the single match left, and its size is
reported as this backup's. A dump now has to fall inside the window by
both its name and its LastModified, which the object store stamps, so a
single skewed clock leaves the size unset instead.

The name parser also accepts the layout later mariadb-operator releases
write for compressed dumps, backup.<ts>.sql.<ext>, and checks the
compression token in either layout, so an operator upgrade does not
silently stop the measurement.

The retry check and the AlreadyExists refetch read the Backup through
the cache. A retry usually follows a Create made moments earlier; a
cache that does not show it yet turned the refetch into NotFound and
failed a backup that had succeeded. Read it from the API server instead.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 3b86f57803 fix(backups): refuse to probe an S3 endpoint behind a path prefix
An endpoint such as https://s3.example.org/prefix cannot be addressed
by minio-go, which has no path-prefix option, and the prefix cannot be
added below the client either, because the SigV4 signature covers the
request path. The CNPG probe was worse than failing: it kept only the
host of endpointURL and listed the bucket at a location barman never
writes to.

Reject such endpoints with an explicit error, in the CNPG probe and in
the shared client, so the size is left unset instead of measured
somewhere else.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 038f35376a fix(backups): do not report a partial listing or re-probe on retry
minio-go ends a listing whose context is cancelled between pages by
closing the channel without an error item. When the probe timeout
expired between pages of a large CNPG base backup, the pages already
read were summed and reported as the measured size. Check the context
once the listing channel closes and leave the size unset instead.

The probe ran before the Backup was created, so a reconcile that retried
after the Backup already existed probed the bucket again, held the
single BackupJob worker for up to the timeout, and then discarded the
result in favour of the existing object. Return an existing Backup
before probing.

Log the ambiguous MariaDB case (several dumps in the window) at info like
the other reasons the size stays unset, and cover the session-token
credential path with a test.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov f222e52863 fix(backups): harden the artifact size probe
The probe runs inside the reconcile that creates the Backup, and the
BackupJob controller has a single worker, so its timeout delays every
other BackupJob while an endpoint hangs. Lower it from 30s to 10s.

List the MariaDB prefix non-recursively, as mariadb-operator does. The
default MariaDB prefix and the default CNPG destinationPath are the same
<namespace>/<name>/ path, so a recursive listing of a MariaDB backed up
next to a same-named Postgres walked the whole WAL archive and ran out of
time. The CNPG base-backup prefix is still listed recursively.

Build the custom-CA transport from minio.DefaultTransport so it keeps the
proxy-from-environment and dial/TLS timeout settings a bare
http.Transport drops, and cover it with a test against a TLS endpoint
with a self-signed certificate.

Log the reasons the size could not be measured (credentials, client,
listing, empty CNPG prefix) at info. The chart runs the controller at
info, which hid them before. The expected ambiguity of several MariaDB
dumps in the window stays at debug.

Skip an endpointCA without a Secret name, as buildBarmanObjectStore does.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov de60488a2c fix(backups): report sizeBytes for CNPG backup artifacts
CNPG backups arrived without Backup.status.artifact.sizeBytes:
cnpg.io/Backup status carries no size, and the data lives in barman's
object-store layout rather than in a single object.

barman-cloud uploads every base backup under
<destinationPath>/<serverName>/base/<backupID>/ (backup.info plus the
data and tablespace tarballs, possibly split into parts). On completion
the driver now lists that prefix with the credentials and endpoint CA of
the rendered barmanObjectStore and records the total. The WAL archive is
not counted: it is one stream shared by every base backup of the server.
The field stays unset when the backup ID is unknown, the bucket cannot be
read, or nothing is stored under the prefix.

The S3 client and Secret lookup are now shared with the MariaDB driver.

Refs #4316

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov d4ad5612d3 fix(backups): report sizeBytes for MariaDB backup artifacts
Backup.status.artifact.sizeBytes was only ever set by the etcd strategy,
so MariaDB backups always arrived without a size.

mariadb-operator writes each logical dump as
<prefix>/backup.<RFC3339>[.<compression>].sql under a prefix shared by
every backup of the application, and the URI the driver records is
synthetic, so there is no single object to HEAD. On completion the driver
now lists the prefix with the tenant credentials the operator Backup was
rendered with and takes the size of the one dump whose timestamp falls
inside the operator Backup's lifetime. When the bucket cannot be read or
the window does not match exactly one dump, the field stays unset.

The API doc now states that an absent sizeBytes means "unknown", and the
Velero strategy documents why it leaves the field unset: kopia stores
data content-addressed across backups, so there is no per-backup stored
size.

Refs #4316

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034)
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
2026-11-03 23:09:46 +05:00
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
Mattia Eleuteri b5d7e43c49 docs(backups): say CDI adopts a restored PVC through the DataVolume
The restore controller and the user-restore example described the
allowClaimAdoption annotation its resource modifiers put on restored
PVCs as what lets CDI adopt them. CDI v1.66 reads that annotation only
on the DataVolume; on a PVC it reads nothing but createdForDataVolume.
Adoption rests on the vm-disk chart annotating the DataVolume Flux
recreates, and a reader trusting the PVC rule could rely on it for a
restore without such a DataVolume, or drop the DataVolume annotation as
redundant.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-21 12:36:22 +02:00
Andrey Kolkov 9af1c0f902 fix(backups): prepare standalone VMDisk restore targets before the data mover runs
collectUnderlyingResources returned nothing for a non-VMInstance kind, so
prepareForRestore did nothing for a standalone VMDisk: no HelmRelease suspend and
no DataVolume delete. An in-place VMDisk restore then raced Flux/CDI — the
restored HelmRelease reinstalled and CDI recreated the DataVolume over the same
PVC name the data mover was repopulating, wedging the RestoreJob.

Report a standalone VMDisk's own disk as its underlying resource, so the existing
(VMInstance-proven) restore preparation suspends the vm-disk-<name> HelmRelease,
deletes its DataVolume and — with keepOriginalPVC — renames the live PVC before
the data mover recreates and repopulates it. No VM, so no IP/MAC is collected.

A VMInstance is halted before its disks are touched; a VMDisk target is not,
and every VMInstance disk is a VMDisk. Renaming the PVC of a disk a running VM
mounts leaves the original Terminating behind pvc-protection, Velero skips a
PVC that still exists, and the RestoreJob would report Succeeded having moved
nothing while the VM loses its disk on the next restart. An in-place VMDisk
restore now fails before changing anything when a Pod that has not terminated
mounts the disk's PVC, naming the Pod and telling the user to stop the VM or
restore the VMInstance instead.

A VMDisk Backup taken before the disk was recorded as its underlying
resource has none, so its disk is now derived from the application name
and such a restore is prepared and checked for a mounting Pod as well.

The per-disk orLabelSelectors stay a VMInstance concern. A VMDisk
strategy already selects its disk, and Velero rejects a Backup or
Restore that carries both labelSelector and orLabelSelectors, so
appending them for a VMDisk would break a strategy written with
labelSelector.

When an in-place restore fails after preparation, nothing is rolled
back: resuming the disk's HelmRelease while its PVC is renamed and its
DataVolume deleted would have CDI import a fresh disk under the original
name, which looks healthy and hides that the data sits in the -orig-
PVC. The failure message and a Warning event now name the suspended
HelmReleases and the -orig- PVCs so the user can recover them, read back
from the cluster, so a failure before a step ran does not name what it
would have left.

Preparation takes the target apart before Velero validates the Restore,
so it is refused up front when the Velero Backup is gone or not
Completed. That check now runs on every pass behind the same failure
path as the rest of preparation, and a FailedValidation Restore fails
the RestoreJob like a Failed one instead of leaving it Running. A
HelmRelease suspend or a VM halt that did not happen now stops
preparation instead of renaming and deleting under a running release or
VM, and a VMInstance's disks are renamed only once its launcher Pod has
let go of them. A VMDisk held only by a Pod already being deleted is
waited for rather than refused, for
up to 30 minutes after the RestoreJob started.

Between renaming a kept disk's PVC and deleting its DataVolume, CDI can
recreate an empty PVC under the original name. Velero then skips the
PVC as already present, the RestoreJob reports Succeeded and the VM
boots a blank disk. Once the DataVolume is gone, preparation now
deletes a PVC found under the original name, read uncached, before the
Velero Restore is created.

Preparation spans several reconciles, so the RestoreJob records a
PreparationStarted condition before it first changes the target, and
retries without changing anything when recording it fails. A
refusal on a later pass (the Velero Backup expired, a VM started on the
disk) then names what preparation left behind instead of claiming that
nothing changed, and a preparation still waiting 30 minutes after it
started fails the same way instead of polling forever.

Now that a VMDisk target is prepared, keepOriginalPVC=false reaches it
too: its DataVolume is deleted with orphan propagation and the bound PVC
is deleted before the Restore, as for a VMInstance disk. Left in place,
Velero would skip it and CDI would refuse the DataVolume the resumed
HelmRelease creates.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-13 18:05:45 +04:00
Andrey Kolkov 0c2f131192 feat(backups): back up Valkey with the Redis strategy
Valkey runs on the same spotahome RedisFailover operator as Redis and
speaks the same replication protocol and RDB format, so it needs no
strategy kind of its own. The only difference the backup Job sees is the
chart's resource-name prefix: valkey-<app> instead of redis-<app> for the
sentinel Service and the auth Secret.

The Redis driver now accepts applicationRef kind Valkey as well as Redis
and hands the template .Release.Prefix, derived from the app's kind, which
cozy-default-redis uses for every name it builds. A kind with no known
prefix fails the render instead of producing an empty one. cozy-default
routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must
target an app of the same kind as the backup's source: replaying a Redis
dump into Valkey, or the reverse, is not something the strategy can
promise across engine releases.

The shipped template reads the prefix as `.Release.Prefix | default
"redis-"`: during an upgrade the previous controller, which has no Prefix
in its render context and serves only Redis, renders the new CR, and a
bare reference would print "<no value>" into the names.

The redis:8 image already loads RDB v11, which both shipped Valkey
versions (7.2, 8.1) write.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-10-13 12:34:24 +04:00
Andrey Kolkov 6a75883013 fix(backups): keep the CNPG purge for unmarked Clusters this restore did not render
A recovery Cluster without the restored-server-name annotation fell back
to the creation-time comparison. On the current chart the annotation is
absent when bootstrap.newServerName is empty, i.e. a manually configured
recovery: re-rendered during the WAL-archive wait it is created after
StartedAt, the purge was skipped and the restore could report success on
the manual recovery source. Use the creation-time fallback only when the
application's bootstrap.newServerName already carries this restore's
server name, which keeps it for Clusters rendered by a chart that
predates the annotation.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-10 23:49:09 +03:00
Andrey Kolkov 35ca217839 fix(backups): name the restore that goes first in ConcurrentRestore
restoreAhead returned the alphabetically smallest of the jobs ahead, so
with more than one ahead the ConcurrentRestore message could name a job
that is not the one holding the application. Return the first job in
restoreGoesFirst order, and pin the creation-time tie-break with cases
where creation order and name order disagree.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-10 23:49:09 +03:00
Andrey Kolkov e4e991e837 fix(backups): skip the CNPG restore purge only for this restore's Cluster
The in-place CNPG restore skips deleting the live Cluster when it
carries bootstrap.recovery and was created after the RestoreJob started,
taking that as "this restore's own purge and re-render produced it". The
timestamp does not say which restore rendered the Cluster. A Cluster
deleted while the restore waits for the WAL archive is re-rendered by
the still-active HelmRelease from the previous restore's values and is
created after StartedAt too; the restore then skipped its purge and
reported Succeeded on the previous restore's data.

The restore patch already sets bootstrap.newServerName to
restoredServerName(cluster, RestoreJob UID). The chart now annotates the
Cluster with it as postgres.cozystack.io/restored-server-name, whether
or not it renders the barman plugin that also carries the value, and the
purge is skipped only when the annotation names this RestoreJob. The
mark lives on the rendered Cluster rather than in RestoreJob status, so
it still covers the failed status write the skip exists for. A Cluster
rendered by a chart that predates the annotation keeps the creation-time
check.

Fixes #3322

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-10 23:49:08 +03:00
Andrey Kolkov 3daf942dce fix(backups): refuse a restore into an application another restore holds
Nothing stopped two RestoreJobs from rewriting one application at once.
Each restores the target from its own Backup, so the application ends up
on whichever finishes last, and the CNPG driver's purge decision cannot
tell which of the two produced the live Cluster: the second job could
skip its purge and report Succeeded on the first job's data.

A RestoreJob that has not started now fails with reason
ConcurrentRestore when another unfinished RestoreJob in the namespace
writes into the same application and goes first. A started restore goes
before one that has not; between two that have not, the older one goes
first, by creation time and then name, so a running restore is never
failed by a newer one. The refused job fails instead of waiting, so the
application is not restored a second time later, after the first
restore, by a job its author may no longer expect to run.

The gate lists RestoreJobs through the manager's API reader rather than
the informer cache. Creation times have one-second precision, so two
jobs created in the same second are ordered by name, and a job reconciled
before the cache held its peer's StartedAt would see that peer as not
started and proceed alongside it. The controller runs one worker, so a
peer's StartedAt written by an earlier reconcile is on the API server by
the time the next job reads it; SetupWithManager now states that.

The target is resolved as the BackupJob hold resolves it, so a Velero
restore into another namespace does not collide with one in place.

Refs #3322

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-10 23:49:08 +03:00
Aleksei Sviridkin f4bde9e50d fix(backups): delete the original PVC of an in-place VM restore without keepOriginalPVC (#4672)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

An in-place VMInstance restore with `keepOriginalPVC: false` reports
`Succeeded` without restoring the disks, and leaves every disk it
touched unable to come back. `prepareForRestore` deletes each DataVolume
with Orphan propagation and leaves the PVC in place, on the assumption,
stated in the `deleteDataVolume` comment, that Velero overwrites it.
Velero does not overwrite a PVC. Its CSI restore action returns early
for a claim that already exists, so no DataDownload is ever created, and
with `existingResourcePolicy: update` the merge patch it attempts
instead is rejected because a bound claim's spec is immutable. All that
reaches the claim is the labels-only fallback patch, which also drops
the `cdi.kubevirt.io/allowClaimAdoption` annotation the resource
modifier added.

The claim left behind then blocks the disk for good. Orphan propagation
has removed its owner reference to the DataVolume, and it lacks the
`cdi.kubevirt.io/storage.populatedFor` annotation that
kubevirt-velero-plugin writes on the backed-up copy, so when the
HelmRelease resumes, the CDI webhook refuses the DataVolume it creates
with `Destination PVC <ns>/<name> already exists`, on every reconcile.
Deleting the disk through the API leaves the PVC behind, and a new disk
of the same name fails the same way.

Observed on a v1.6.3 cluster, for a VMInstance restore with
`keepOriginalPVC: false` whose disk was attached to the VM. The Velero
log for the restore, trimmed:

```text
level=info msg="Retrieved per-restore resource modifiers from configmap cozy-velero/restore-modifiers-..."
level=warning msg="PVC already exists and ExistingVolumeDataPolicy is not in-place restore. Skip restore this PVC." Action=PVCCSIRestoreItemAction PVC=tenant-datalab/vm-disk-e2edisk
level=info msg="Applying resource modifier patch on tenant-datalab/vm-disk-e2edisk"
level=warning msg="patch attempt failed for /v1, Kind=PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk: PersistentVolumeClaim \"vm-disk-e2edisk\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims"
level=info msg="backup/restore labels successfully updated for PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk"
```

`velero restore describe --details` lists the PVC as `(failed)` and the
Restore is `Completed` with two warnings. Afterwards the claim had its
original creation timestamp and UID, no owner reference, no
`allowClaimAdoption`, no `populatedFor`, and the disk's HelmRelease was
`Ready=False` with the webhook error above. That cluster runs a Velero
build from `main`, whose wording is quoted above; the released v1.17.1
and v1.18.3 return early at the same point with `PVC already exists.
Skip restore this PVC.` (`pkg/restore/actions/csi/pvc_action.go`).

The fix deletes the PVC when `keepOriginalPVC` is false, only once its
DataVolume is confirmed gone so that CDI cannot recreate it, and holds
the Velero Restore until the claim has actually disappeared. A claim
still mounted stays `Terminating` behind `kubernetes.io/pvc-protection`,
and creating the Restore then would hit the same existing-claim path.
Velero then creates the PVC from the backup, the data mover fills it,
and the recreated DataVolume adopts it through `populatedFor`. A failed
deletion now fails the RestoreJob instead of letting it report
`Succeeded` over the old data. Before 2234824fe the DataVolume was
deleted with background propagation, so garbage collection removed the
claim, but nothing waited for it.

Deleting the claim is only safe when Velero can put the disk back, so
before any preparation the reconciler now gets the Velero Backup and
fails the RestoreJob unless it exists and is `Completed`. The Backup
artifact outlives the Velero Backup, which Velero garbage-collects at
its TTL, and without the check a restore of an expired backup would halt
the VM and delete its disks before failing. The check runs for every
Velero restore, whatever `keepOriginalPVC` says. A DataVolume that
cannot be deleted now fails the RestoreJob too, and the preparation
waits for every DataVolume to be gone before it goes further: a
surviving DataVolume makes CDI recreate the claim, which Velero then
skips. Reading the DataVolume back needs `get` on `datavolumes`, which
the backupstrategy-controller ClusterRole now grants next to `delete`.

`TestPrepareForRestore_KeepOriginalPVCFalse_SkipsRename` asserted the
old behaviour, that the original PVC survives, and is replaced by
`TestPrepareForRestore_KeepOriginalPVCFalse_DeletesPVC`.
`TestPrepareForRestore_KeepOriginalPVCFalse_WaitsForTerminatingPVC`
covers the wait. Both fail on `upstream/main` without the controller
change. `TestReconcileVeleroRestore_UnusableVeleroBackupTouchesNothing`
covers an expired Velero Backup and one being deleted,
`TestPrepareForRestore_KeepOriginalPVCFalse_KeepsPVCWhileDataVolumeRemains`
a failed and a pending DataVolume deletion, and
`TestPrepareForRestore_KeepOriginalPVCTrue_FailedRenameKeepsPVC` a
failed rename under `keepOriginalPVC: true`, which must leave the claim
in place. A helm-unittest suite, `tests/restore_rbac_test.yaml`, checks
the new verb. Each test fails with its fix reverted, with one exception:
`TestPrepareForRestore_KeepOriginalPVCTrue_FailedRenameKeepsPVC` still
passes with the Step 5 condition cut to `vmRes != nil`, because a failed
rename holds the restore back before Step 5 is reached. That guard is
pinned by `TestPrepareForRestore_KeepOriginalPVC_CacheLagsBehindWrites`,
which fails with the same cut. `helm unittest
packages/system/backupstrategy-controller` passes. `go test
./internal/backupcontroller/...` passes.

Related: #4601 reports the same Velero mechanism for a standalone
VMDisk, where `prepareForRestore` does nothing at all because
`collectUnderlyingResources` records no disk. #3427 makes a VMDisk
report its own disk; with `keepOriginalPVC: false` that path then runs
into this defect too, and this change covers it once both are in. This
PR does not touch the VMDisk path itself.

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map against the diff, which is the restore
preparation in `internal/backupcontroller/velerostrategy_controller.go`
and its unit tests, plus one verb in the backupstrategy-controller
ClusterRole and its helm-unittest: no package added or renamed, no
`values.yaml` or `values.schema.json` change, no CRD,
ApplicationDefinition, label, annotation, namespace or metric change.
`keepOriginalPVC` lives in the opaque `RestoreJob.spec.options`, and a
search of the organisation finds it only in this repository, so neither
the website nor the Terraform provider documents or models it.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): an in-place VMInstance RestoreJob with keepOriginalPVC=false now deletes each original disk PVC, and waits for it to be gone, before Velero restores it. Previously the PVC was left in place, Velero skipped restoring its data, the RestoreJob still reported Succeeded, and the disk's HelmRelease could never recreate its DataVolume ("Destination PVC already exists"). A Velero RestoreJob now also fails before touching the application when its Velero Backup is gone or not Completed, for example after the Backup expired at its TTL.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* When a restore is configured not to keep the original disk claim, the
claim is deleted before Velero restores it, allowing Velero to recreate
it.
* Restores wait while the original disk claim or its DataVolume is being
deleted. Deletion or status-check errors stop restore preparation.
* Restores configured to keep the original disk claim continue to
preserve it.
* Restores stop before changing resources if the referenced Velero
backup is missing or has not completed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 22:26:10 +03:00
Aleksei Sviridkin b1192bfcbc fix(backups): create no driver object for a job that already finished (#4673)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The BackupJob and RestoreJob reconcilers decide a job is still active
from the informer cache, and the cache can trail their own writes. A
driver's first pass stamps `StartedAt`, which requeues the job, and may
then fail the job in the same pass on a transient error, for instance a
503 from the aggregated `apps.cozystack.io` API while
`createVeleroBackup` reads the application. The requeued pass can start
before the informer has seen `Failed`: it retries the step that failed
and, once the error has cleared, creates the driver object anyway. Its
own status write then conflicts on the stale resourceVersion, the next
pass sees `Failed` and skips the job, and a Velero Backup is left
running that no `Backup` artifact will reference and nothing tracks.

Reproduced against a real apiserver (envtest 1.34) with the real manager
cache and the reconciler wired through `SetupWithManager`, nothing else
writing the BackupJobs. The only fault injected is one 503 on the first
GET of each application through `apps.cozystack.io`. Out of 40
BackupJobs, every one ends `Failed` with `failed to create Velero
Backup: the server is currently unable to handle the request`, and on
`main` (6addef4d3) 10, 8 and 5 of them still got a Velero Backup over
three runs. With this change: 0, 0 and 0. The harness is in the
collapsed block below; it needs `KUBEBUILDER_ASSETS` and `REPO_ROOT`,
and is not part of the PR.

A restore pass that starts on such a stale copy does worse, since the
step it retries can be destructive: suspending the VM's HelmRelease and
halting the VM before creating a Velero Restore, or patching the
Postgres application and purging its cluster. The reproduction above
covers the backup side only; on the restore side the unit test below
shows what a pass on a stale copy does.

Both reconcilers now re-read the job through the manager's uncached
reader when the cached copy says it is still active, and skip it when
the apiserver holds it as finished. The check sits ahead of the
dispatch, so it covers all eleven drivers at once. The cost is one GET
per pass of an active job; a job the cache already reports as finished
costs nothing extra.

`TestReconcile_DoesNotRetryAJobItAlreadyFailed` and
`TestReconcileRestore_DoesNotRetryAJobItAlreadyFailed` run the full
`Reconcile` on a cache holding the job active while the apiserver holds
it `Failed`. Each first checks that the same fixture does create the
Velero Backup, or suspend the HelmRelease and create the Velero Restore,
when the apiserver agrees with the cache. With the re-read reverted to
the cached client, both fail: a Velero Backup is created and the phase
is written back to `Running`, the HelmRelease is suspended and a Velero
Restore is created. `go test ./internal/backupcontroller/...` passes.

<details>
<summary>envtest reproduction harness</summary>

```go
//go:build repro

// Reproduction, not for commit: does the real BackupJob controller, on a real
// apiserver and the manager's real cache, create a Velero Backup for a
// BackupJob it has itself marked Failed?
//
// The only fault injected is one 503 on the first GET of each application
// through apps.cozystack.io, the aggregated cozystack-api, standing for a
// blip of that API. Nothing else writes the BackupJob.
//
// Run: KUBEBUILDER_ASSETS=<envtest bin dir> go test -tags repro -run TestReproFinishedJobRace -v ./internal/backupcontroller/
package backupcontroller

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"os"
	"strings"
	"sync"
	"testing"
	"time"

	corev1 "k8s.io/api/core/v1"
	apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
	"k8s.io/client-go/kubernetes/scheme"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/envtest"
	"sigs.k8s.io/controller-runtime/pkg/log/zap"
	metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"

	strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1"
	backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
	velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
)

type blipOnce struct {
	next http.RoundTripper
	mu   sync.Mutex
	seen map[string]bool
}

func (b *blipOnce) RoundTrip(req *http.Request) (*http.Response, error) {
	p := req.URL.Path
	if req.Method == http.MethodGet && strings.Contains(p, "/apis/apps.cozystack.io/v1alpha1/namespaces/") {
		b.mu.Lock()
		first := !b.seen[p]
		b.seen[p] = true
		b.mu.Unlock()
		if first {
			body := `{"kind":"Status","apiVersion":"v1","status":"Failure","message":"the server is currently unable to handle the request","reason":"ServiceUnavailable","code":503}`
			return &http.Response{
				StatusCode: http.StatusServiceUnavailable,
				Header:     http.Header{"Content-Type": []string{"application/json"}},
				Body:       io.NopCloser(strings.NewReader(body)),
				Request:    req,
			}, nil
		}
	}
	return b.next.RoundTrip(req)
}

func TestReproFinishedJobRace(t *testing.T) {
	ctrl.SetLogger(zap.New(zap.WriteTo(io.Discard)))
	root := os.Getenv("REPO_ROOT")
	dummyCRD := &apiextensionsv1.CustomResourceDefinition{
		ObjectMeta: metav1.ObjectMeta{Name: "dummies.apps.cozystack.io"},
		Spec: apiextensionsv1.CustomResourceDefinitionSpec{
			Group: "apps.cozystack.io",
			Names: apiextensionsv1.CustomResourceDefinitionNames{Kind: "Dummy", ListKind: "DummyList", Plural: "dummies", Singular: "dummy"},
			Scope: apiextensionsv1.NamespaceScoped,
			Versions: []apiextensionsv1.CustomResourceDefinitionVersion{{
				Name: "v1alpha1", Served: true, Storage: true,
				Schema: &apiextensionsv1.CustomResourceValidation{OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{
					Type: "object", XPreserveUnknownFields: new(true),
				}},
			}},
		},
	}
	env := &envtest.Environment{
		CRDInstallOptions: envtest.CRDInstallOptions{
			Paths: []string{
				root + "/packages/system/backup-controller/definitions",
				root + "/packages/system/backupstrategy-controller/definitions",
				root + "/packages/system/velero/charts/velero/crds/backups.yaml",
			},
			CRDs:               []*apiextensionsv1.CustomResourceDefinition{dummyCRD},
			ErrorIfPathMissing: true,
		},
	}
	cfg, err := env.Start()
	if err != nil {
		t.Fatalf("envtest: %v", err)
	}
	defer env.Stop()

	s := runtime.NewScheme()
	_ = scheme.AddToScheme(s)
	_ = backupsv1alpha1.AddToScheme(s)
	_ = strategyv1alpha1.AddToScheme(s)
	_ = velerov1.AddToScheme(s)

	direct, err := client.New(cfg, client.Options{Scheme: s})
	if err != nil {
		t.Fatal(err)
	}
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	mgrCfg := *cfg
	mgrCfg.WrapTransport = func(rt http.RoundTripper) http.RoundTripper {
		return &blipOnce{next: rt, seen: map[string]bool{}}
	}
	mgr, err := ctrl.NewManager(&mgrCfg, ctrl.Options{Scheme: s, Metrics: metricsserver.Options{BindAddress: "0"}})
	if err != nil {
		t.Fatal(err)
	}
	r := &BackupJobReconciler{Client: mgr.GetClient(), Scheme: s, Recorder: mgr.GetEventRecorderFor("repro")}
	if err := r.SetupWithManager(mgr); err != nil {
		t.Fatal(err)
	}
	go func() { _ = mgr.Start(ctx) }()

	must := func(o client.Object) {
		if err := direct.Create(ctx, o); err != nil {
			t.Fatalf("create %T %s: %v", o, o.GetName(), err)
		}
	}
	must(&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: "tenant"}})
	must(&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: veleroNamespace}})
	must(&strategyv1alpha1.Velero{ObjectMeta: metav1.ObjectMeta{Name: "velero"}})
	strategyGroup := strategyv1alpha1.GroupVersion.Group
	appGroup := "apps.cozystack.io"
	must(&backupsv1alpha1.BackupClass{
		ObjectMeta: metav1.ObjectMeta{Name: "bc"},
		Spec: backupsv1alpha1.BackupClassSpec{Strategies: []backupsv1alpha1.BackupClassStrategy{{
			StrategyRef: corev1.TypedLocalObjectReference{APIGroup: &strategyGroup, Kind: strategyv1alpha1.VeleroStrategyKind, Name: "velero"},
			Application: backupsv1alpha1.ApplicationSelector{APIGroup: &appGroup, Kind: "Dummy"},
		}}},
	})
	time.Sleep(2 * time.Second) // let the cache list the fixtures

	n := 40
	if v := os.Getenv("REPRO_N"); v != "" {
		fmt.Sscan(v, &n)
	}
	for i := range n {
		app := &unstructured.Unstructured{}
		app.SetGroupVersionKind(schema.GroupVersionKind{Group: appGroup, Version: "v1alpha1", Kind: "Dummy"})
		app.SetNamespace("tenant")
		app.SetName(fmt.Sprintf("app-%d", i))
		must(app)
		must(&backupsv1alpha1.BackupJob{
			ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: fmt.Sprintf("bj-%d", i)},
			Spec: backupsv1alpha1.BackupJobSpec{
				BackupClassName: "bc",
				ApplicationRef:  corev1.TypedLocalObjectReference{APIGroup: &appGroup, Kind: "Dummy", Name: app.GetName()},
			},
		})
		time.Sleep(200 * time.Millisecond)
	}
	time.Sleep(5 * time.Second)

	phases := map[backupsv1alpha1.BackupJobPhase]int{}
	orphans := 0
	for i := range n {
		j := &backupsv1alpha1.BackupJob{}
		if err := direct.Get(ctx, client.ObjectKey{Namespace: "tenant", Name: fmt.Sprintf("bj-%d", i)}, j); err != nil {
			t.Fatal(err)
		}
		phases[j.Status.Phase]++
		vbs := &velerov1.BackupList{}
		if err := direct.List(ctx, vbs, client.InNamespace(veleroNamespace), client.MatchingLabels{
			backupsv1alpha1.OwningJobNamespaceLabel: "tenant", backupsv1alpha1.OwningJobNameLabel: j.Name,
		}); err != nil {
			t.Fatal(err)
		}
		if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed && len(vbs.Items) > 0 {
			orphans++
			if orphans <= 3 {
				t.Logf("%s: phase=%s message=%q, yet Velero Backup %s/%s exists (created %s, job completedAt %s)",
					j.Name, j.Status.Phase, j.Status.Message, veleroNamespace, vbs.Items[0].Name,
					vbs.Items[0].CreationTimestamp.Format(time.RFC3339Nano), j.Status.CompletedAt.Format(time.RFC3339Nano))
			}
		}
	}
	t.Logf("jobs=%d phases=%v failed-with-velero-backup=%d", n, phases, orphans)
}
```

</details>
### Screenshots

N/A — no UI changes.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked the trigger map against the diff, which is the two job
reconcilers in `internal/backupcontroller/` and a test file: no package,
`values.yaml`, `values.schema.json`, CRD, ApplicationDefinition, label,
annotation, namespace or `hack/` change.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): a BackupJob or RestoreJob that the controller has just marked Failed no longer gets a driver object created for it by the next reconcile, which could start on a cache that had not yet seen the Failed status and retry the step that failed — leaving, for instance, an untracked Velero Backup behind a Failed BackupJob, or a restore that halts the VM after its RestoreJob failed.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Backup and restore jobs that have already succeeded or failed are no
longer retried based on outdated status, preventing duplicate operations
and unnecessary changes to related resources.
* Reconciliation stops when a job has been removed and reports errors if
its current status cannot be checked.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 22:15:54 +03:00
Mattia Eleuteri 06e72b26dd test(backups): give the restore stale-cache test a Completed Velero Backup
A Velero RestoreJob is about to require its Velero Backup to exist and
be Completed before it halts the application. Without one in the
fixture, the first pass of this test fails the RestoreJob before it
suspends the HelmRelease, so the test would stop exercising the stale
cache it is about.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-08 14:09:08 +02:00
Aleksei Sviridkin f203a47afe fix(backups): hold a Postgres backup until the plugin rollout is over (#4674)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The first BackupJob on a Postgres app whose `cnpg.io/Cluster` has never
carried the barman-cloud plugin — a fresh platform-flow app, or one an
in-place restore re-rendered — attaches the plugin and creates the
`cnpg.io/Backup` in the same reconcile. Attaching the plugin changes the
instance pod spec, so CNPG rolls every instance (`Pod rollout required:
volumes: element plugins has been added`). The Backup then lands on an
instance without the plugin sidecar (`requested plugin is not available:
barman-cloud.cloudnative-pg.io`) or on one CNPG restarts under it
(`instance manager was restarted during backup`). CNPG does not retry a
failed Backup, so the BackupJob stays Running in
`CNPGBackupTransientFailure` until `cnpgDefaultBackupDeadline` and
fails. The first backup of each such Postgres is always lost; this was
observed on a v1.6.3 cluster and reproduced against `main`.

The driver now holds the Backup back until that rollout is over. Before
attaching the plugin to a Cluster that has none, it records
`PluginRolledOut=False` on the BackupJob, with `Ready=False` reason
`WaitingForPluginRollout` for the user, and requeues; the next pass
attaches the plugin. The Backup is created once CNPG reports the Cluster
in its healthy phase and every instance pod (`cnpg.io/cluster`,
`cnpg.io/podRole=instance`) was created after the condition and is
Ready; the condition then turns True and `Ready` is cleared, as before.
Recording the condition first means a lost status write leaves the
plugin unattached for the next pass instead of a Cluster that looks
ready while it rolls. A Cluster that already runs the plugin is not
rolled by the driver and its Backup is created on the first pass, as
today. The wait is bounded by the existing `cnpgDefaultBackupDeadline`,
after which the BackupJob fails with the phase or pod CNPG is stuck on.

Why this signal, checked against the CNPG 1.29 source and the 1.30
Cluster CRD vendored here: the Cluster status has no
`observedGeneration` and its `Ready` condition is derived from
`status.phase`, so right after the attach the phase still reads healthy
from before until CNPG reconciles the new spec. `status.pluginStatus`
does not help either, since CNPG also lists plugins referenced only from
`externalClusters` there, which is exactly the state of a Cluster a
restore re-rendered. A pod spec change is never applied in place (only a
pending Postgres restart is), so the instances are recreated and pods
newer than the attach are the ones running the sidecar. This reads pod
metadata and CNPG's documented labels only, not the plugin's container
or volume names, and needs no RBAC change: the controller already lists
and watches pods.

Tests: `TestReconcileCNPG_WaitsForPluginRolloutBeforeBackup` drives a
Cluster without the plugin through the attach, a stale healthy phase
over the old pod, a rolling phase over the recreated pod, and the
healthy phase, and asserts no Backup exists until the last step.
`TestReconcileCNPG_AttachedPluginDoesNotDelayBackup` covers a Cluster
that already runs the plugin, and
`TestReconcileCNPG_PluginRolloutWaitIsBounded` the deadline. With the
gate disabled, the first and third fail; with only the phase checked and
not the pods, the first fails at the attach pass.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not applicable: no UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

The diff touches only `internal/backupcontroller` Go code: no package,
values, chart, CRD, `hack/` tooling, telemetry metric or contributor
convention. Walked against the trigger map, it reaches none of the
repositories below, and the website documents no workaround for the lost
first backup that this would make obsolete.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): the CNPG backup driver no longer loses the first backup of a Postgres app whose Cluster had no barman-cloud plugin yet (a new platform-flow app, or one restored in place). It now waits for CNPG to finish rolling the instances onto the plugin before creating the cnpg.io Backup, and reports the wait on the BackupJob as Ready=False with reason WaitingForPluginRollout.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Reliability**
* Backups wait for the backup plugin rollout to complete and cluster
instances to become healthy before starting.
* If rollout does not finish before the backup deadline, the backup job
fails instead of waiting indefinitely.
* Backups can start without waiting for a new rollout when the plugin is
already attached.
* **Backup Configuration**
* When no server name is configured, backups use the cluster name and
application UID, giving recreated applications distinct archive
prefixes.
* Existing server names are preserved when the plugin is already
attached.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-08 13:11:46 +03:00
Mattia Eleuteri eafa441b9c fix(backups): create no driver object for a job that already finished
The BackupJob and RestoreJob reconcilers decide a job is still active
from the informer cache, and the cache can trail their own writes. A
driver's first pass stamps StartedAt, which requeues the job, and may
then fail the job in the same pass on a transient error, such as a 503
from the aggregated apps.cozystack.io API while it reads the
application. The requeued pass can start before the informer has seen
Failed: it retries the step that failed and, once the error has
cleared, creates the driver object anyway. Its own status write then
conflicts, the pass after it sees Failed and skips the job, and a
Velero Backup is left running that no Backup artifact will reference
and nothing tracks.

Against a real apiserver and the manager's cache, with one 503 on the
first application read of each job, 5 to 10 BackupJobs out of 40 ended
Failed with a Velero Backup.

A restore pass that starts on such a stale copy does worse, since the
step it retries can be destructive: suspending the VM's HelmRelease and
halting the VM before creating a Velero Restore, or patching the
Postgres application and purging its cluster.

Both reconcilers now re-read the job through the manager's uncached
reader when the cached copy says it is still active, and skip it when
the apiserver holds it as finished. The check sits ahead of the
dispatch, so it covers every driver, and a job the cache already
reports as finished costs no extra request.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-07 11:32:46 +02:00
Mattia Eleuteri 6d064a8602 fix(backups): delete a restored disk's PVC only once the restore can run
Deleting the original PVC of a keepOriginalPVC=false restore is only
safe when Velero can put the disk back. The Backup artifact outlives
the Velero Backup it points to, which Velero garbage-collects at TTL,
and nothing checked it before prepareForRestore halted the VM and
deleted the disks: a restore of an expired backup lost the volume where
it used to fail with the disk intact. The Velero Backup must now exist
and be Completed before any preparation starts.

A DataVolume that survives its deletion makes CDI recreate the claim
under the original name, which Velero then skips as an existing PVC,
so the restore would end Succeeded on a freshly imported disk. A failed
DataVolume deletion now fails the restore, and the PVCs are deleted
only once every DataVolume is gone. Reading the DataVolume back needs
get on datavolumes, which the ClusterRole did not grant.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-07 11:23:37 +02:00
Mattia Eleuteri 6ea713d09b fix(backups): delete the original PVC of an in-place VM restore without keepOriginalPVC
With keepOriginalPVC=false the restore deleted each DataVolume with
Orphan propagation and left its PVC in place for Velero to overwrite.
Velero never overwrites a PVC: its CSI restore action skips the data
mover for a claim that already exists, and existingResourcePolicy=update
fails to patch the immutable spec of a bound claim, so it only stamps
its labels on it. The RestoreJob then reports Succeeded with the disk
still holding its pre-restore data.

The claim left behind also breaks the disk for good. It has lost its
owner reference to the orphaned DataVolume, and it lacks the
cdi.kubevirt.io/storage.populatedFor annotation that the backed-up copy
carries, so the CDI webhook refuses the DataVolume the resumed
HelmRelease creates ("Destination PVC already exists"), on every
reconcile.

Delete the PVC once its DataVolume is gone, and hold the Velero Restore
until the claim has actually disappeared: one still mounted stays
Terminating behind pvc-protection, and Velero would take it for an
existing claim again.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-07 11:23:17 +02:00
Mattia Eleuteri 54ab863c0e fix(backups): hold a Postgres backup until the plugin rollout is over
A Cluster that has never carried the barman-cloud plugin gets it from
the first BackupJob: a fresh platform-flow app, or one an in-place
restore re-rendered. Attaching it changes the instance pod spec, so
CNPG rolls every instance. The cnpg.io/Backup the same reconcile
created then runs on an instance without the plugin sidecar ("requested
plugin is not available") or on one CNPG restarts under it ("instance
manager was restarted during backup"). CNPG never retries it, so the
BackupJob sat in CNPGBackupTransientFailure until the backup deadline
and failed: the first backup of every such Postgres was lost.

The driver now records a PluginRolledOut=False condition before it
attaches the plugin, and creates no Backup until CNPG reports the
Cluster healthy and every instance pod was created after that
condition and is Ready. Recording it first means a lost status write
leaves the plugin unattached for the next pass, rather than a Cluster
that looks ready while it rolls. A Cluster that already runs the plugin
is not rolled by the driver and is not held back.

The phase alone cannot tell the rollout is over. Cluster status has no
observedGeneration and its Ready condition is derived from the phase,
so until CNPG reconciles the new spec the phase still reads healthy
from before the attach. status.pluginStatus is no better: CNPG also
lists plugins referenced only from externalClusters there, which is
exactly the state of a Cluster a restore re-rendered. A pod spec change
is never applied in place, so the instances are recreated, and pods
newer than the attach are the ones running the sidecar. That reads only
pod metadata and CNPG's documented labels, not the plugin's container
names. The wait stays bounded by the existing backup deadline.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-10-07 11:15:43 +02:00
Aleksei Sviridkin 520af57905 chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out
internal mirror block, an internal product name, and a reviewer's
name in a comment do not belong in a public repository; drop them
and replace what needs a stand-in with neutral placeholders.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 04:38:06 +03:00
Aleksei Sviridkin d2e3eb4f82 fix(kubernetes, e2e): unblock autoscaling below min size, size the computeplane pool, fix the backupcontroller build (#4768)
## What this PR does

Makes the backupcontroller test package compile again on main. #4587
added a flavor argument to `createCNPGBackupArtifact` and
`marshalCNPGBackupSnapshot`, and #4669, whose CI ran before that, added
two test calls with the old argument lists. Both PRs were green on their
own, and together they break `go test ./internal/backupcontroller/`. The
two calls now pass the same default flavor as the other tests in the
file.

The second commit fixes an e2e suite that fails the required E2E check
here. It is the `computeplane` suite from #4713, and its worker pool was
one node with a single vCPU. After the kubelet reservations that node
has 900m of CPU to allocate. The tenant addons (cilium, coredns, csi,
ingress-nginx, metrics-server, cert-manager) request exactly 900m
between them. If cert-manager's startupapicheck hook pod is created
last, it can't be scheduled and the install times out after 10m. The
reinstall after remediation runs into the same full node. Whether that
pod comes last depends on install order, so the suite passed on #4713
and failed here.

The pool now has two workers with one vCPU each, the same shape the
kubernetes suites use. One vCPU per worker stays on purpose, because of
#3513.

Two workers exposed a deadlock in the tenant cluster-autoscaler. The
suite's pool is created at zero replicas and then gets a min size of 2,
which CAPI does not apply to the live replica count on upgrade (#4786).
The autoscaler's scale-up for a pending pod asks for one node, and the
clusterapi provider refuses it because one is under the min size,
although it is an increase. That refusal backs the group off, and the
min-size enforcement skips groups in backoff, so the pool never leaves
zero. Upstream had the same report in kubernetes/autoscaler#4942, closed
as stale, and it is now tracked in kubernetes/autoscaler#10400. Our
autoscaler image already patches the mirror case for max size, and the
patch now refuses only a decrease below the min size.

`go vet` and `go test ./internal/backupcontroller/...` pass, and so do
the bats pins for the computeplane suite. The autoscaler patch adds a
test for each direction, and the clusterapi package tests pass with it
applied to 1.32.0.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved cluster autoscaling for worker groups outside configured size
limits: groups below their minimum can scale up toward it, and nodes can
be removed when a group exceeds its maximum. Scale-down requests below
the minimum remain restricted.
* **Tests**
* Updated backup and restore test calls with PostgreSQL and empty flavor
values.
* Updated ComputePlane checks for a two-worker pool and clarified
cert-manager installation timing expectations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 04:36:10 +03:00