Commit Graph
58 Commits
Author SHA1 Message Date
Aleksei Sviridkin 7c46e1e384 fix(backups): give the barman-cloud sidecar an explicit CPU limit (#5280)
## What this PR does

The barman-cloud sidecar of Postgres backups is deliberately left
without a CPU limit, so that a throttled sidecar slows WAL archiving
down instead of failing it. In a tenant with `resourceQuotas` set that
intent does not hold: `packages/apps/tenant/templates/quota.yaml`
creates the `tenant-range-limits` LimitRange, whose default CPU limit is
`250m`, and the sidecar receives it. Full backups and WAL segments are
then gzip-compressed on a quarter of a core. On a production tenant the
first full backup of a 1.6 GB database took 30 minutes, uploading at
about 0.5 MB/s with the sidecar pinned at its 250m limit, while a small
database on the same destination finished in 35 seconds.

This sets an explicit `limits.cpu: "1"` on the sidecar, in the two
places that build its configuration: `barmanSidecarConfiguration` in the
backup driver (platform-managed ObjectStores) and
`cozy-lib.barman.sidecarConfiguration` (ObjectStores rendered by the
postgres and keycloak charts). This mirrors how the memory request and
limit there already override the LimitRange's 128Mi default. One core
removes the 250m ceiling without letting the sidecar starve the
instance. Inside a tenant it counts against the `limits.cpu` quota, one
core per instance pod; the comments now say so.

The sidecar is injected when the instance pod is created, so existing
instances pick the new limit up at their next restart.

Tests: the Go test that asserted the absence of a CPU limit now asserts
one core, and so does
`packages/apps/postgres/tests/barman_resources_test.yaml`. Both fail
with the implementation reverted and pass with it. `go test
./internal/backupcontroller/`, `helm unittest packages/apps/postgres`
(220) and `helm unittest packages/system/keycloak` (121) pass.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff (the backup driver, `cozy-lib`'s
barman template, two tests): no app, value, schema, default, CRD,
annotation or tooling change, so none of the listed repositories
restates anything touched here.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the barman-cloud sidecar of Postgres backups now has an explicit one-core CPU limit instead of inheriting the tenant LimitRange's 250m default, which made full backups and WAL archiving several times slower. Each instance pod of a tenant with a CPU quota now counts 750m more against `limits.cpu`; check the headroom before instance pods are recreated. Existing ObjectStores pick the limit up at their next backup (platform-managed) or HelmRelease upgrade (chart-rendered), and running pods at their next restart.
```
2026-11-13 14:54:51 +03:00
Alexey Artamonov 1d22013646 fix(postgres): hold a backup until the plugin sees its new destination
The plugin-barman-cloud sidecar reads the ObjectStore through its own
cache, kept for ten seconds (DefaultTTLSeconds in plugin-barman-cloud
v0.13.0) and invalidated only by the sidecar's own writes, and WAL
archiving keeps it warm. The pass that changed the destination created
the cnpg.io/Backup a moment later, so the base backup could be taken
against the previous destination while the artifact names the new one,
and a restore from it finds nothing. On a live cluster with the shipped
plugin, a Backup created two to four seconds after switching the
destination landed in the old bucket in two of three tries; with a
fifteen second wait it landed in the new one every time.

The apply that changes the ObjectStore configuration now stamps the
time on the ObjectStore itself, so the record cannot be lost apart from
the change, and the job creates its Backup only once that change is
older than the cache can be. A change is judged on the configuration a
server-side dry run of the same apply returns, so values the apiserver
defaults or another field manager owns do not read as a change on every
pass.

That wait leaves a job with the destination written and no Backup yet,
which the queue, looking only for Backups in flight, did not see: a
second job with another destination wrote its own, the first took that
as a change and stamped again, and the two restarted each other's wait
forever. Every apply of the ObjectStore therefore also names the job
that made it, which only a job without a Backup of its own and let
through by the queue does, and the queue holds every other job while
that one runs and has no Backup yet. Once it has one, that Backup holds
the destination for as long as CloudNativePG runs it, as before, so a
job stuck after its Backup finished does not hold every later backup of
the application. This also covers a job waiting for the
rollout of the plugin it attached, during which a second job used to
switch the destination as well, including when its own write left the
configuration as a restore had re-rendered it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 15:13:58 +03:00
Mattia Eleuteri f1dd2d17fc fix(backups): give the barman-cloud sidecar an explicit CPU limit
The sidecar was left without a CPU limit so that a throttled sidecar
would slow WAL archiving down rather than fail it. In a tenant with
resourceQuotas that does not hold: the tenant LimitRange gives every
container a 250m CPU limit by default, so the sidecar gets it and
compresses full backups and WAL segments on a quarter of a core. A
1.6 GB database took 30 minutes to back up.

Set an explicit one-core limit in both places that build the sidecar
configuration, the backup driver and cozy-lib, the same way the memory
request and limit already override the LimitRange defaults.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-11 09:20:19 +01:00
Alexey Artamonov 59a11c1788 refactor(postgres): judge an attach record's clock drift in one place
The job that attached the plugin and a job waiting on another job's
attach each parsed the Cluster's attach record and applied the
clock-drift margin on their own, so the two could drift apart on the
next change. Both now take the record from one parser that also
refuses one too far ahead, and differ only in what they do with a
record they cannot use. The margin is now pinned for the waiting job
from both sides: a record a minute ahead still holds it for the
rollout.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 16:00:09 +03:00
Alexey Artamonov a27de054ca fix(postgres): wait for the rollout another backup job's attach started
Only the job that attaches the barman-cloud plugin waited for the
instance rollout it causes. A second job of the same application that
started meanwhile found the plugin already there, had no rollout of its
own to wait for and created its cnpg.io/Backup at once, against
instances CNPG was still recreating, and such a Backup fails for good.
The queue does not hold it either, since the attaching job has no
Backup yet.

The attach record on the Cluster says when the plugin was attached, so
a job without a rollout of its own now waits for instances newer than
it while the attach is recent enough for that rollout to be running.
An older attach has had its rollout judged by the job that made it.
The record belongs to that job, so one this job cannot read, or one
further ahead than controller clocks drift, is skipped rather than
failing it; the latter would otherwise hold every later job of the
application for as long as it lies ahead.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 01:27:21 +03:00
Alexey Artamonov d37701a327 fix(postgres): tolerate an attach record slightly ahead of the clock
An attach record in the future failed the backup job at once, because
it would hold every instance as older than the plugin. A record a few
minutes ahead is also what a leadership change between controllers
whose clocks differ leaves behind after an attach that worked, and
failing then throws away a backup that would have run. Within that
margin the job now keeps waiting: the instances predate the record, so
the rollout check holds the backup, and the deadline counted from the
record is not spent by the wait. Only a record further ahead is taken
as corrupt and fails the job.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov c69787ff49 perf(postgres): list only the Cluster's own backups for the queue
Every pass of a backup job waiting for another one reads the
cnpg.io/Backups from the apiserver, and the read covered every Backup in
the namespace, so a namespace with a long backup history paid for it on
each poll of each waiting job. The queue only looks at Backups this
controller created for the Cluster, which have carried the cluster and
owning-job labels since the driver was introduced, so it now asks the
apiserver for those alone.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov 9a8025ca14 fix(postgres): record the serverName a backup was archived under
The artifact took its destination from the record written when the
cnpg.io/Backup was created, but its serverName, the archive prefix
under that destination, from the live Cluster when the backup
completed. A restore that re-renders the Cluster in between gives it
another serverName, and the artifact then pairs the original
destination with a prefix the backup was never written to, so a later
restore searches the wrong archive. The record now carries the
serverName too; a record written before it did still falls back to the
live Cluster.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov b4551eb379 fix(postgres): read restores from the apiserver before writing the destination
A backup job that is about to write the WAL archive destination first
checks for a restore of its application, because a write landing during
the restore puts the restored Cluster back on the source's WAL prefix
and the restore fails with an expected-empty-archive error after it has
purged the target. That check read RestoreJobs from the informer cache,
which can still miss one created a moment ago.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov 8a631311a7 fix(postgres): fail a backup job whose plugin attach record is in the future
The rollout check takes the later of the latch and the Cluster's attach
record, and the record is never rewritten while the plugin stays
attached. A record that parses but lies in the future made every
instance look older than the plugin and counted the rollout deadline
from that time, so the job waited past the deadline by as far as the
record lay ahead, without end for one far enough out. Such a record
says no more about when the plugin was attached than one that does not
parse, and fails the job the same way.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov 8d92aee796 test(postgres): pin that an older attach record keeps the latch
The rollout check takes the later of the latch and the Cluster's attach
record. A record left by an earlier attach of a plugin since removed is
older than the latch when someone who keeps no record attaches the
plugin again, and taking it would accept an instance recreated before
the latch as one that rolled with the plugin.

The record is absent not only when no backup job attached the plugin
but also when the chart did, or a controller that kept no record, so
the comment on the field says that.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov 325d2b815c fix(postgres): keep the plugin attach time when its status write is lost
The rollout check took the attach time from the PluginRolledOut
condition on the BackupJob, which the pass that attaches the plugin
writes only after the Cluster apply went through. When that status
write is lost to a conflict or a controller restart, the next pass
finds the plugin attached and falls back to the latch written before
the attach. An instance recreated between the latch and the attach then
counts as one that rolled with the plugin, and the cnpg.io/Backup is
created against an instance that has no plugin sidecar; a latch older
than the deadline also fails a rollout that has only just begun.

The apply that attaches the plugin now records the time on the Cluster
itself, so the record cannot be lost apart from the attach. The rollout
check and its deadline use the later of the two times. A record that
does not parse fails the job rather than guess which instances rolled.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov e5ccde82e2 fix(postgres): read cnpg Backups from the apiserver before writing the destination
The queue that keeps two BackupJobs of one application from writing the
WAL archive destination at once listed cnpg.io/Backups from the
informer cache. The controller runs one reconcile at a time under
leader election, so a Backup created by one job's pass already exists
when the next pass starts, but the cache can still miss it. The next
job then took the queue as empty, rewrote the destination under the
running backup and left that backup's recorded destination naming a
place its WAL no longer went. A job could likewise miss its own fresh
Backup, write the destination again and create a second Backup.

The queue now lists Backups from the apiserver, and a job that finds
no Backup of its own in the cache checks the apiserver before it
writes or creates one. Only a leader handover, when two processes
reconcile for a moment, can still let two jobs through.

The pass that creates the Backup no longer looks it up a second time
right before the create. Nothing can create it in between, and once
the lookup can reach the apiserver, a failed second lookup would come
after the destination write and fail the job instead of retrying.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov a6af14ce74 fix(postgres): build a backup's artifact from the destination it was written to
The WAL archive destination is written by the pass that creates the
cnpg.io/Backup and not again while it runs, yet the artifact took the
destination, the endpoint and the credentials Secret from the strategy
rendered when the Backup completed. A strategy or BackupClass edited
while the backup ran gave an artifact that pointed somewhere nothing
had been written, and a restore from it found no data.

The pass that writes the destination now records it on the
cnpg.io/Backup it creates, and the artifact is built from that record.
The record lives on the Backup rather than on the BackupJob because a
tenant can edit its BackupJobs but has no access to postgresql.cnpg.io
objects: a record a tenant could rewrite would point the artifact, and
a restore from it, which takes its S3 keys from the strategy, at an
endpoint of the tenant's choosing.

The artifact size probe measures the same place: the recorded
location and credentials Secret, with the Secret keys and the endpoint
CA, which the record does not carry, from the strategy. Should those no
longer fit after an edit, the probe reports no size instead of
measuring somewhere else.

A record that cannot be read, or that names no destination path,
creates no artifact and fails the job, since the strategy may no longer
match the write. A Backup without a record was created by a controller
that kept none, and its artifact is built from the strategy as before.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 00:08:41 +03:00
Alexey Artamonov 258c5359b6 fix(postgres): restart the plugin rollout whenever a backup job attaches it
Only the first attach after the latch restarted the rollout. In the
platform flow the chart renders no spec.plugins, so an in-place restore
recreates the Cluster without the plugin, and a backup job held through
that restore attaches it again once released. The rollout condition
then still carried its earlier start, the restored instances were newer
than it and ready, and the Backup was created in the same pass, before
CloudNativePG had rolled anything; such a Backup fails with "requested
plugin is not available" or "instance manager was restarted".

Every write of the job that finds the plugin missing now starts the
rollout over. A plugin attached by another writer is still timed from
the latch.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 23:52:25 +03:00
Alexey Artamonov 32ca678077 fix(postgres): give a held backup job its plugin rollout deadline back
A job that latched the barman-cloud plugin rollout and then waited for
another backup of the application, or for a restore of it, had that
wait charged to the rollout deadline. Released after more than the
deadline, it failed on the first instance that was not ready yet with
"rollout did not finish", although no rollout of its own had been
running while it waited.

The hold is now recorded in a DestinationWriteHeld condition, and its
release time restarts the rollout deadline when it is later than the
rollout's start. The instance pods are still measured against the
rollout's start: one that rolled while the job was held carries the
plugin and counts.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 23:52:25 +03:00
Alexey Artamonov a02bff7fad docs(postgres): say what the CNPG backup deadlines are counted from
The rollout deadline starts at the latch when another writer attached
the plugin and at the write otherwise, and a wait for a restore resets
the destination retry budget just as a wait for another backup does.
The comments on the deadline and on the two conditions still described
only the attach and the queue.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 23:52:25 +03:00
Alexey Artamonov 66268d8508 refactor(postgres): drop a Ready reset every later path overwrites
After a successful destination write each path of the CNPG driver sets
or removes Ready later in the same pass, and writes the status after
it, so clearing a stale WaitingForOtherBackup or
ClusterPluginApplyRetrying reason there could never outlive the pass,
and no test could pin it.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-11-09 23:52:25 +03:00
Alexey Artamonov 567f07d708 fix(postgres): time the plugin rollout from the write that attached it
The PluginRolledOut latch is recorded before the Cluster write that
attaches the barman-cloud plugin, and its time was used both as the
start of the rollout deadline and as the age every instance pod must
exceed to count as rolled. Now that a failing write is retried for up
to the backup deadline, the attach can come long after the latch: the
deadline was then spent before the rollout began, and a pod recreated
between the latch and the attach, without the plugin sidecar, passed
as rolled, so a Backup was created that CloudNativePG fails for good.

The latch now carries its own reason, and the write that finds the
plugin missing and attaches it re-stamps the condition, once, with the
time it was attempted. A plugin someone else attached after the latch,
another backup job or a restore, keeps the latch time: the rollout it
started can be over by the time this job writes, and its instances
would otherwise never count as rolled.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-11-09 23:52:25 +03:00
Alexey Artamonov 570fcec725 fix(postgres): hold a queued backup job while its application is restored
The restore gate in Reconcile looks only at a BackupJob that has not
started, and the CNPG driver stamps StartedAt on its first pass. A job
that then waits behind another backup of the application, for as long
as that backup runs, writes the WAL archive destination and the
Cluster plugin once released without looking for a restore that began
in the meantime. A write landing while the restore re-renders the
Cluster can put the restored Cluster back on the source's WAL prefix,
where recovery fails with "Expected empty archive".

Every pass that is about to write the destination now looks for a
restore of the application and holds the job the way the gate does.
A job whose Backup already runs writes nothing and is not held. The
hold resets the retry budget of the destination write, as the wait
for another backup does, so time spent behind a restore does not fail
the job on the first error after it.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-11-09 23:52:25 +03:00
Alexey Artamonov 9cc9a3b48d fix(postgres): stop backup jobs from fighting over the WAL archive destination
The ObjectStore a Postgres Cluster archives its WAL to is shared by
every backup of the app, yet each running BackupJob server-side applied
its own strategy's destination onto it every five seconds with forced
ownership. Two BackupJobs of one app with different BackupClasses
flipped the destination back and forth until the healthy backup failed
too, and a webhook or apiserver blip during that apply failed the
BackupJob at once.

A BackupJob now writes the destination only before its cnpg.io/Backup
exists, and waits, with a WaitingForOtherBackup condition, while
another running BackupJob of the app has a Backup in flight. The check
reads the informer cache, so jobs released together, or one created
while another's Backup is not yet cached, can still both proceed; an
atomic per-application claim is a separate change.

Write errors that cannot succeed on retry (invalid, bad request,
forbidden) still fail the job at once. Any other is retried for the
backup deadline counted from its first failure since the job last left
the queue, recorded in a DestinationApplied condition. Once the Backup
exists the Cluster is only read, and a failed read is returned for a
controller retry instead of failing the running backup. The plugin
rollout deadline counts from the attach rather than from the job's
start, so queueing before the attach no longer shortens it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 23:52:25 +03:00
scooby87 10e0ef9ece fix(postgres): fail a backup job as soon as its cnpg Backup fails (#5211)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Part of #5201. The remaining part, a bound on a Backup that stays
`pending`, is in #5209.

A Postgres BackupJob whose `cnpg.io/Backup` had failed stayed `Running`
for 30 minutes with reason `CNPGBackupTransientFailure`, on the premise
that CloudNativePG may move a Backup out of the failed phase. It does
not: in CloudNativePG 1.30.0 the backup controller returns early on any
Backup already `failed` or `completed`, so the wait only delayed the
verdict. The BackupJob now fails as soon as the Backup is `failed`, with
the error CloudNativePG recorded. When that error is only the plugin's
exit status, the message points to the `plugin-barman-cloud` sidecar
log, where the storage error (`AccessDenied`, a connect timeout) is.

A Backup in any other phase keeps the BackupJob waiting as before, but
the BackupJob now carries a `Ready=False` / `WaitingForCNPGBackup`
condition naming the Backup and its phase instead of no condition at
all. These phases stay uncapped on purpose: CloudNativePG keeps a
healthy Backup `pending` while another backup of the same Cluster runs
and while it restarts one on another instance, so a fixed deadline here
would fail healthy backups, in particular scheduled ones that queue
behind a long previous run. #5209 tracks bounding a Backup that is stuck
rather than queued, together with the Plan concurrency that makes
queueing common.

A table-driven test drives the reconciler through each Backup phase and
asserts the job phase, the message, the persisted condition, that an
unchanged phase does not rewrite the BackupJob status, and that only a
completed Backup produces a Cozystack `Backup`.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): a Postgres BackupJob now fails as soon as its CloudNativePG Backup fails, with the recorded error, instead of after 30 minutes, and a BackupJob waiting on a CloudNativePG Backup reports a WaitingForCNPGBackup condition with the Backup's phase.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* CNPG backups that enter a failed state now fail their BackupJob
immediately, with the recorded error shown when available.
* BackupJobs waiting for a CNPG backup to finish remain in a waiting
state, even after the backup deadline; the deadline applies while
waiting for the backup to be created.
* Waiting status now includes a clear reason, and failure messages
provide a sidecar-log hint when relevant.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 18:17:03 +03:00
Andrey Kolkov 080df1b2f2 fix(backups): surface a Backup that cannot be read back and skip its probe
A Backup that exists but cannot be read back is retried through the
controller's backoff, which left the BackupJob Running with nothing on
it saying why. Set Ready=False with reason BackupNotReadBack before
handing the error back.

When the check for an existing Backup itself fails, the Backup may well
exist, so a size probe would only be discarded, and a read that keeps
failing reran it, up to the probe timeout on the single BackupJob
worker, on every retry. Skip the probe in that case and create the
Backup without a size.

Drive both behaviours through the MariaDB and CNPG reconciles in tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 523d74a503 fix(backups): retry instead of failing when an existing Backup cannot be read
When Create collides with a Backup an earlier reconcile already made and
the read that follows fails, the drivers failed the BackupJob, although
the backup and its Backup both exist. Return a distinct error instead,
which the drivers hand back to the controller so the reconcile is
retried with backoff and then finds the Backup.

minio.New already refuses an endpoint with a path prefix and accepts a
trailing slash, so drop the duplicate check in the shared client; the
CNPG probe keeps its own check because it parses the URL itself and
would otherwise drop the path before minio sees it.

Say in the sizeBytes API doc what the value covers, such as the base
backup without the shared WAL archive for CNPG, and that strategies
which cannot attribute stored bytes to one backup leave it unset.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 2b21763577 fix(backups): do not fail a backup on a transient retry-check read
The check for a Backup an earlier reconcile already created returned
its read error, and the drivers turn any error from that path into a
terminally Failed BackupJob. An apiserver timeout right after the
operator backup completed therefore lost a backup whose data was safely
in S3. The check only saves a probe, and the Create that follows already
handles an existing Backup, so log the error and go on to Create.

A trailing slash on the S3 endpoint is not a path prefix; the operators
accept it, so do not refuse it.

Pin the uncached AlreadyExists refetch, the upper slack of the MariaDB
window and the probe timeout with tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov a3d854739d fix(backups): attribute MariaDB dumps on two clocks and read retries uncached
The MariaDB probe identified this run's dump by the timestamp in its
file name, which comes from the backup pod's clock. With that clock
skewed past the slack, this run's dump falls outside the window while a
neighbouring run's dump can be the single match left, and its size is
reported as this backup's. A dump now has to fall inside the window by
both its name and its LastModified, which the object store stamps, so a
single skewed clock leaves the size unset instead.

The name parser also accepts the layout later mariadb-operator releases
write for compressed dumps, backup.<ts>.sql.<ext>, and checks the
compression token in either layout, so an operator upgrade does not
silently stop the measurement.

The retry check and the AlreadyExists refetch read the Backup through
the cache. A retry usually follows a Create made moments earlier; a
cache that does not show it yet turned the refetch into NotFound and
failed a backup that had succeeded. Read it from the API server instead.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 038f35376a fix(backups): do not report a partial listing or re-probe on retry
minio-go ends a listing whose context is cancelled between pages by
closing the channel without an error item. When the probe timeout
expired between pages of a large CNPG base backup, the pages already
read were summed and reported as the measured size. Check the context
once the listing channel closes and leave the size unset instead.

The probe ran before the Backup was created, so a reconcile that retried
after the Backup already existed probed the bucket again, held the
single BackupJob worker for up to the timeout, and then discarded the
result in favour of the existing object. Return an existing Backup
before probing.

Log the ambiguous MariaDB case (several dumps in the window) at info like
the other reasons the size stays unset, and cover the session-token
credential path with a test.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov de60488a2c fix(backups): report sizeBytes for CNPG backup artifacts
CNPG backups arrived without Backup.status.artifact.sizeBytes:
cnpg.io/Backup status carries no size, and the data lives in barman's
object-store layout rather than in a single object.

barman-cloud uploads every base backup under
<destinationPath>/<serverName>/base/<backupID>/ (backup.info plus the
data and tablespace tarballs, possibly split into parts). On completion
the driver now lists that prefix with the credentials and endpoint CA of
the rendered barmanObjectStore and records the total. The WAL archive is
not counted: it is one stream shared by every base backup of the server.
The field stays unset when the backup ID is unknown, the bucket cannot be
read, or nothing is stored under the prefix.

The S3 client and Secret lookup are now shared with the MariaDB driver.

Refs #4316

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Alexey Artamonov fe6968be7f fix(postgres): fail a backup job as soon as its cnpg Backup fails
A failed cnpg.io/Backup kept its BackupJob Running for 30 minutes as a
"transient" failure, on the premise that CloudNativePG may move a
Backup out of the failed phase. It does not: its backup controller
returns early on any Backup already failed or completed, so the wait
only delayed the verdict and hid it behind a misleading reason.

A failed Backup now fails the BackupJob at once with the error CNPG
recorded, pointing to the plugin-barman-cloud sidecar log when that
error is only the plugin's exit status. A Backup in any other phase
keeps the BackupJob waiting as before, now with a WaitingForCNPGBackup
condition naming the Backup and its phase instead of no condition at
all. It is still not capped: CloudNativePG keeps a healthy Backup
pending while another backup of the same Cluster runs or while it
restarts one, and a fixed deadline would fail those.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 15:38:20 +03:00
Andrey Kolkov 6a75883013 fix(backups): keep the CNPG purge for unmarked Clusters this restore did not render
A recovery Cluster without the restored-server-name annotation fell back
to the creation-time comparison. On the current chart the annotation is
absent when bootstrap.newServerName is empty, i.e. a manually configured
recovery: re-rendered during the WAL-archive wait it is created after
StartedAt, the purge was skipped and the restore could report success on
the manual recovery source. Use the creation-time fallback only when the
application's bootstrap.newServerName already carries this restore's
server name, which keeps it for Clusters rendered by a chart that
predates the annotation.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-10 23:49:09 +03:00
Andrey Kolkov e4e991e837 fix(backups): skip the CNPG restore purge only for this restore's Cluster
The in-place CNPG restore skips deleting the live Cluster when it
carries bootstrap.recovery and was created after the RestoreJob started,
taking that as "this restore's own purge and re-render produced it". The
timestamp does not say which restore rendered the Cluster. A Cluster
deleted while the restore waits for the WAL archive is re-rendered by
the still-active HelmRelease from the previous restore's values and is
created after StartedAt too; the restore then skipped its purge and
reported Succeeded on the previous restore's data.

The restore patch already sets bootstrap.newServerName to
restoredServerName(cluster, RestoreJob UID). The chart now annotates the
Cluster with it as postgres.cozystack.io/restored-server-name, whether
or not it renders the barman plugin that also carries the value, and the
purge is skipped only when the annotation names this RestoreJob. The
mark lives on the rendered Cluster rather than in RestoreJob status, so
it still covers the failed status write the skip exists for. A Cluster
rendered by a chart that predates the annotation keeps the creation-time
check.

Fixes #3322

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-10 23:49:08 +03:00
Aleksei Sviridkin f203a47afe fix(backups): hold a Postgres backup until the plugin rollout is over (#4674)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The first BackupJob on a Postgres app whose `cnpg.io/Cluster` has never
carried the barman-cloud plugin — a fresh platform-flow app, or one an
in-place restore re-rendered — attaches the plugin and creates the
`cnpg.io/Backup` in the same reconcile. Attaching the plugin changes the
instance pod spec, so CNPG rolls every instance (`Pod rollout required:
volumes: element plugins has been added`). The Backup then lands on an
instance without the plugin sidecar (`requested plugin is not available:
barman-cloud.cloudnative-pg.io`) or on one CNPG restarts under it
(`instance manager was restarted during backup`). CNPG does not retry a
failed Backup, so the BackupJob stays Running in
`CNPGBackupTransientFailure` until `cnpgDefaultBackupDeadline` and
fails. The first backup of each such Postgres is always lost; this was
observed on a v1.6.3 cluster and reproduced against `main`.

The driver now holds the Backup back until that rollout is over. Before
attaching the plugin to a Cluster that has none, it records
`PluginRolledOut=False` on the BackupJob, with `Ready=False` reason
`WaitingForPluginRollout` for the user, and requeues; the next pass
attaches the plugin. The Backup is created once CNPG reports the Cluster
in its healthy phase and every instance pod (`cnpg.io/cluster`,
`cnpg.io/podRole=instance`) was created after the condition and is
Ready; the condition then turns True and `Ready` is cleared, as before.
Recording the condition first means a lost status write leaves the
plugin unattached for the next pass instead of a Cluster that looks
ready while it rolls. A Cluster that already runs the plugin is not
rolled by the driver and its Backup is created on the first pass, as
today. The wait is bounded by the existing `cnpgDefaultBackupDeadline`,
after which the BackupJob fails with the phase or pod CNPG is stuck on.

Why this signal, checked against the CNPG 1.29 source and the 1.30
Cluster CRD vendored here: the Cluster status has no
`observedGeneration` and its `Ready` condition is derived from
`status.phase`, so right after the attach the phase still reads healthy
from before until CNPG reconciles the new spec. `status.pluginStatus`
does not help either, since CNPG also lists plugins referenced only from
`externalClusters` there, which is exactly the state of a Cluster a
restore re-rendered. A pod spec change is never applied in place (only a
pending Postgres restart is), so the instances are recreated and pods
newer than the attach are the ones running the sidecar. This reads pod
metadata and CNPG's documented labels only, not the plugin's container
or volume names, and needs no RBAC change: the controller already lists
and watches pods.

Tests: `TestReconcileCNPG_WaitsForPluginRolloutBeforeBackup` drives a
Cluster without the plugin through the attach, a stale healthy phase
over the old pod, a rolling phase over the recreated pod, and the
healthy phase, and asserts no Backup exists until the last step.
`TestReconcileCNPG_AttachedPluginDoesNotDelayBackup` covers a Cluster
that already runs the plugin, and
`TestReconcileCNPG_PluginRolloutWaitIsBounded` the deadline. With the
gate disabled, the first and third fail; with only the phase checked and
not the pods, the first fails at the attach pass.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not applicable: no UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

The diff touches only `internal/backupcontroller` Go code: no package,
values, chart, CRD, `hack/` tooling, telemetry metric or contributor
convention. Walked against the trigger map, it reaches none of the
repositories below, and the website documents no workaround for the lost
first backup that this would make obsolete.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): the CNPG backup driver no longer loses the first backup of a Postgres app whose Cluster had no barman-cloud plugin yet (a new platform-flow app, or one restored in place). It now waits for CNPG to finish rolling the instances onto the plugin before creating the cnpg.io Backup, and reports the wait on the BackupJob as Ready=False with reason WaitingForPluginRollout.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Reliability**
* Backups wait for the backup plugin rollout to complete and cluster
instances to become healthy before starting.
* If rollout does not finish before the backup deadline, the backup job
fails instead of waiting indefinitely.
* Backups can start without waiting for a new rollout when the plugin is
already attached.
* **Backup Configuration**
* When no server name is configured, backups use the cluster name and
application UID, giving recreated applications distinct archive
prefixes.
* Existing server names are preserved when the plugin is already
attached.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-08 13:11:46 +03:00
Mattia Eleuteri 54ab863c0e fix(backups): hold a Postgres backup until the plugin rollout is over
A Cluster that has never carried the barman-cloud plugin gets it from
the first BackupJob: a fresh platform-flow app, or one an in-place
restore re-rendered. Attaching it changes the instance pod spec, so
CNPG rolls every instance. The cnpg.io/Backup the same reconcile
created then runs on an instance without the plugin sidecar ("requested
plugin is not available") or on one CNPG restarts under it ("instance
manager was restarted during backup"). CNPG never retries it, so the
BackupJob sat in CNPGBackupTransientFailure until the backup deadline
and failed: the first backup of every such Postgres was lost.

The driver now records a PluginRolledOut=False condition before it
attaches the plugin, and creates no Backup until CNPG reports the
Cluster healthy and every instance pod was created after that
condition and is Ready. Recording it first means a lost status write
leaves the plugin unattached for the next pass, rather than a Cluster
that looks ready while it rolls. A Cluster that already runs the plugin
is not rolled by the driver and is not held back.

The phase alone cannot tell the rollout is over. Cluster status has no
observedGeneration and its Ready condition is derived from the phase,
so until CNPG reconciles the new spec the phase still reads healthy
from before the attach. status.pluginStatus is no better: CNPG also
lists plugins referenced only from externalClusters there, which is
exactly the state of a Cluster a restore re-rendered. A pod spec change
is never applied in place, so the instances are recreated, and pods
newer than the attach are the ones running the sidecar. That reads only
pod metadata and CNPG's documented labels, not the plugin's container
names. The wait stays bounded by the existing backup deadline.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-10-07 11:15:43 +02:00
Aleksei Sviridkin 520af57905 chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out
internal mirror block, an internal product name, and a reviewer's
name in a comment do not belong in a public repository; drop them
and replace what needs a stand-in with neutral placeholders.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 04:38:06 +03:00
Mattia Eleuteri 5956b459fb fix(backups): give each Postgres instance its own default WAL prefix
The default CNPG strategy archived a Postgres app under
<namespace>-<name>, inside a destinationPath that is also keyed by
namespace and name, and the driver fell back to the Cluster name when a
strategy set no serverName. Deleting the app does not empty the bucket,
so an app recreated under the same name archived its WALs and base
backups into its predecessor's stream: both start on timeline 1, and a
later restore of the old archive could pick the new cluster's base
backup or WAL segments.

barman-cloud refuses a non-empty archive only while
.check-empty-wal-archive exists in PGDATA. CNPG removes that file as
soon as ContinuousArchiving turns True, and with no plugin attached
archive_command is a successful no-op, so the file is gone minutes
after initdb, before the first BackupJob attaches the plugin.

Suffix the default serverName with the application UID, in the
strategy template and in the driver's fallback. The template guards
the UID with `with`, so an object without one renders the previous name
rather than "<no value>". A Cluster that already has the plugin keeps
its serverName, so only clusters never backed up take the new prefix,
and restores keep reading the source prefix from driverMetadata.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-10-06 16:35:42 +03:00
Aleksei Sviridkin 3fc45ed183 fix(backups, postgres): refuse a restore into another PostgreSQL major (#4669)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A CNPG restore is physical, so the recovered data files only open on the
PostgreSQL major version that wrote them. `reconcileCNPGRestore` purged
the target Cluster and its PVCs without comparing the two. Observed on
v1.6.3: a PostgreSQL 18 backup restored into a v16 application emptied
the target, the `full-recovery` pods then looped on `FATAL: database
files are incompatible with server`, and the RestoreJob failed at the
30-minute deadline with the target left empty. An in-place restore into
an application whose `spec.version` was raised since the backup ends the
same way.

The restore now compares the two majors before the purge, ahead of the
WAL-archive gate, and when they differ marks the RestoreJob `Failed`
with reason `MajorVersionMismatch` and a message naming both versions.
Nothing is suspended, patched or deleted on that path.

- Target major: the Postgres application's `spec.version` (`v13` to
`v18`). That is what the chart renders the recovered Cluster from once
the live one is gone, so the live Cluster's image would be the wrong
thing to compare.
- Backup major: the `cnpg.io/Backup`'s `status.majorVersion`, now copied
into the artifact's `driverMetadata` as `cnpg.io/major-version` at
backup time, because barman retention deletes the `cnpg.io/Backup` while
the artifact stays restorable. An artifact taken before this change
falls back to the `cnpg.io/Backup` while it still exists.
- When either side is unknown (no record and the `cnpg.io/Backup` gone,
a CNPG that predates `majorVersion`, an empty `spec.version`), the
restore goes ahead as before and the controller logs it, so existing
backups keep restoring.

`TestReconcileCNPGRestore_RefusesCrossMajorRestore` drives the real
reconcile against a fake client with a live Cluster and PVC on the
target and counts every `Delete`. With the check removed, the two
refusal cases (18 into a v16 copy, 16 in place into an app moved to v18)
and the fallback case fail: the Cluster and the PVC are deleted and the
app is patched for recovery. With the check in place they end `Failed`
with zero deletes, while the same-major and unknown-version cases still
reach the purge. `TestCreateCNPGBackupArtifact_RecordsMajorVersion`
covers the `driverMetadata` record.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked against the diff: it touches only `internal/backupcontroller`,
with no CRD, no app `values.yaml` or schema and no ApplicationDefinition
change. The new `cnpg.io/major-version` entry lives in the free-form
`Backup.spec.driverMetadata` map, which no downstream repository reads
by key.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups, postgres): a Postgres restore whose backup was taken on another PostgreSQL major version than the target's spec.version now fails before touching the target, instead of purging it and then failing to recover.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Cross-major PostgreSQL restores are now refused before the target is
changed. The error identifies the backup and target versions.
* Restores proceed when either version is unavailable or cannot be
parsed.
* **New Features**
* Backup metadata now records the PostgreSQL major version when it is
available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 16:35:27 +03:00
Mattia Eleuteri cfa31d3a17 fix(postgres): record the live cluster's flavor on a backup
The chart refuses a flavor change on an existing Cluster, but the
Postgres app keeps the refused value. A BackupJob taken after such a
change recorded the app's flavor, not the one the data directory was
written by, so a RestoreJob would accept a target of the wrong image
family and refuse the right one.

The driver already reads the live Cluster to keep its barman
serverName; it now takes the flavor from that Cluster's imageName the
same way the chart's flavor guard compares images.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-05 08:52:32 +02:00
Mattia Eleuteri f514f8722b fix(backups, postgres): refuse a restore into another PostgreSQL major
A CNPG restore is physical: the recovered data files only open on the
major version that wrote them. The driver purged the target Cluster and
its PVCs without comparing the two, so a PostgreSQL 18 backup restored
into a v16 application, or in place into an application upgraded since
the backup, emptied the target. The recovery pods then looped on
"database files are incompatible with server" until the restore
deadline, and the RestoreJob failed with nothing left to fall back on.

The restore now fails before the purge, naming both versions, when they
differ. The target side is spec.version, since that is what the chart
renders the recovered Cluster from once the live one is gone. The
backup side is CNPG's status.majorVersion, copied into the artifact's
driverMetadata at backup time because retention deletes the
cnpg.io/Backup while the artifact stays restorable; artifacts taken
before that fall back to the cnpg.io/Backup. When either side is
unknown the restore goes ahead as before, so older backups keep
restoring.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-01 21:58:09 +02:00
Mattia Eleuteri f714ada567 fix(postgres): refuse a physical restore across image flavors
A recovery replays the source's data directory, so it carries the same
risk as switching the image of a live cluster: the postgresql and
postgis images are built on different Debian releases, and the glibc
change invalidates collatable indexes, while a postgis-to-postgresql
move also loses the libraries behind every PostGIS object. The chart
only guarded the live cluster, so a fresh release bootstrapped from a
backup of the other flavor went through, and the guard's own advice was
to restore a backup, which is that very move.

The backup-controller now records the source flavor in the snapshot it
stores on each Backup and fails a RestoreJob whose target runs another
flavor, before the purge deletes the target's Cluster and PVCs. A
snapshot without a flavor predates flavors and so comes from a
postgresql cluster. For the chart-managed bootstrap, the chart compares
against the recovery source's Cluster while it still exists. Both paths,
and the live-cluster guard, point a flavor change to a logical dump.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-01 18:31:18 +02:00
Mattia Eleuteri 67bf9ca317 fix(postgres): read the live Cluster when deciding the WAL-archive prefix
applyClusterPluginBackup preserves the serverName already on the Cluster, so
that a backup does not flip the archive prefix mid-stream. It read that value
through the manager's cached client, and this controller registers
For(&BackupJob{}) and nothing else, so a Cluster only reaches the cache
through a lazily started informer with no resync tied to its own changes.

An in-place RestoreJob deletes the Cluster and the chart re-renders it with
bootstrap.newServerName seconds later. A read served during that window
describes the object that was purged, so the source's serverName is written
back over the freshly rendered one and the restored cluster archives onto the
source's WAL prefix. barman-cloud-check-wal-archive then refuses to start with
"Expected empty archive" and the RestoreJob ends on its deadline -- the
failure newServerName was added to prevent.

The stale read carries a stale UID too, and that UID owns the ObjectStore this
function applies, so the store is garbage-collected as soon as it is written.

Reading through the dynamic client keeps the preservation rule and makes the
value it rests on truthful.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:06:12 +02:00
Mattia Eleuteri 497dd6fc4f fix(backups): start a Postgres restore from the backup it names
Without recoveryTarget.backupID the barman-cloud plugin chooses the
base backup itself: the newest one in the catalog when no recoveryTime
is set, and the newest one ending by recoveryTime otherwise, on any
timeline since no targetTLI is passed. A RestoreJob naming an older
Backup therefore started from another one, and on an archive holding
sibling timelines - left by in-place restores that archived back under
the source serverName - from another branch, bringing back that
branch's data.

The restore reads barman's backupId and the backup's stoppedAt from
the live cnpg.io/Backup behind the Backup, and passes the id through
the chart as bootstrap.recovery.recoveryTarget.backupID. The plugin
gives backupID precedence over time-based selection, and CNPG does not
count it among the mutually exclusive targets, so it combines with
targetTime.

Two cases keep the plugin's choice. A recoveryTime earlier than one
second past stoppedAt: the named backup cannot serve an instant before
it is consistent, and stoppedAt is truncated to the second while
barman compares its end time to the microsecond. And a cnpg.io/Backup
that is gone: plugin-barman-cloud's retention deletes it together with
the base backup it describes, and the driver deletes one only with its
artifact, so pinning its id would purge the target and then fail the
recovery on every retry. That restore records a BaseBackupGone warning
event instead.

A re-restore into the same target clears a backupID left by a previous
one.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 09:38:11 +02:00
Aleksei Sviridkin 2c2c0301f7 chore(deps): roll up pending Renovate updates and move to Go 1.27.1 (#4447)
## What this PR does

This replaces the open Renovate PRs with one rollup. Go modules move to
their newest releases that keep the Kubernetes libraries and
controller-runtime where they are, the toolchain and builder images move
to Go 1.27.1, and the first-party code gets the Go 1.27 modernizers.
Tests for the code the bumps touch come first, in their own commits, and
pass on both the old and the new dependencies.

Before the bumps:

- The e2e sandbox image builds again: dl.min.io answers 410 for mc since
MinIO archived it, so mc now comes from its last GitHub release with a
pinned checksum.
- Renovate stops offering tags of the apimachinery fork. They mirror
upstream, so #2999 is a downgrade that drops our patch.
- `make unit-tests` now runs the tests of the nested Go modules
(kubevirt-csi-driver, token-proxy, kubeovn-webhook, api/apps/v1alpha1).
They never ran in CI before.
- New tests: JWT verification in the dashboard token proxy, the kube-ovn
plunger metrics, cert-manager object idempotency in the tenant gateway,
values-file layering in the operator, and the kubevirt CSI driver
wrapper.
- One real fix. The lineage webhook decoded admitted objects with
encoding/json, so an int64 above 2^53 came back rounded. jsonpatch v2.5
compares numbers exactly and would emit a replace that rewrites such a
field in the pod. The webhook now decodes with apimachinery's json.

Held back on purpose, with the reasons in the commit messages:

- Kubernetes libraries stay on v0.35. #3001 needs the apimachinery fork
rebased onto release-1.37 first.
- cel-go v0.29.2 and etcd v3.6.15, the lines k8s.io/apiserver v0.35 is
built and tested with.
- grpc v1.83.2. v1.84.0 is affected by GO-2026-6443 and no v1.84 patch
release carries the fix yet.
- kubevirt.io/api and the CDI API stay on what upstream csi-driver
builds against. From CDI v1.60 it no longer compiles.
- api/apps/v1alpha1 keeps go 1.26.4, so consumers of the published types
are not forced onto Go 1.27.
- go fix runs without embedlit (Go 1.27-only syntax, breaks backports to
release branches on Go 1.26) and omitzero (would make metadata, spec and
status required in the published OpenAPI schemas), and one errorsastype
rewrite that did not compile is left out.

govulncheck reports nothing reachable in any of the five modules,
standard library included.

Supersedes #4311, #4310, #4309, #3452, #3993, #3927, #3902, #3805,
#3451, #3415, #3414, #3413, #3302, #4050 and #3363. #3001 stays open.

Not checked locally: the images that build upstream projects (kube-ovn,
kamaji, cilium, linstor-csi and others) with the Go 1.27 builder. The
image build jobs of this PR cover them.

Closes #4487.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): move the toolchain and builder images to Go 1.27.1 and raise Go dependencies to their latest compatible releases, closing the published advisories; the lineage webhook no longer rewrites int64 fields above 2^53 in admitted objects.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added a restore resource policy option for Velero strategies.
  * Added a test target to run Go tests across tracked modules.
* **Bug Fixes**
* Updated the end-to-end test environment to download and verify the
MinIO client from its current release location.
* Added coverage for token validation, webhook patching, controller
behavior, and reconciliation.
* **Maintenance**
* Updated the Go toolchain, dependencies, and container image versions.
  * Refreshed pinned workflow actions and base image digests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-26 10:17:29 +04:00
Alexey Artamonov 70dcd9daed fix(backups): let a restored Postgres app converge its credentials
A restore sets bootstrap.enabled on the target app, which skips the
init-job, so the recovered roles keep the source's password hashes while
the target's credentials Secret holds generated ones and every
application login fails. Once recovery converges, the driver records
RecoveryConverged and then clears bootstrap.enabled, so the next
HelmRelease reconcile runs the init-job and applies the passwords. Ready
carries RestoreCompletedCredentialsPending and an Event says the
convergence is still pending. A clear that keeps failing is retried
under BootstrapDisablePending within a fixed grace window after
convergence and then fails the restore. bootstrap.enabled is serialized
without omitempty so the patch sends false instead of dropping the key.

Running the init-job on a restored cluster exposes its delete pass to a
catalog the spec does not describe: the driver sets spec.databases to
the Backup's snapshot, but recovery replays WAL past it, so a database
created after the Backup comes back still marked as managed and would
be dropped with its data. The first run after a restore, told apart by
the per-RestoreJob bootstrap.newServerName that the cluster has not yet
recorded, releases such a database and its roles from management
instead, and the run records the value once every pass has succeeded.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:11:07 +03:00
Aleksei Sviridkin 7a0aab2bdc refactor: apply the Go 1.27 modernizers
Run go fix over every first-party module: interface{} becomes any,
hand-written loops become slices, maps, min/max and range-over-int,
errors.As becomes errors.AsType, pointer helpers become new(expr) and
are inlined at their call sites, and so on. Every rewrite keeps
behaviour.

Three rewrites stay out. embedlit flattens embedded-field literals
such as ObjectMeta, syntax only Go 1.27 accepts, which would break
backports to release branches still on Go 1.26. omitzero drops
omitempty from struct fields, and openapi-gen reads omitempty as
optional, so metadata, spec and status would turn required in the
published schemas. And errorsastype rewrote an errors.As whose
target, apierrors.APIStatus, does not implement error, which
errors.AsType rejects at compile time.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 17:23:18 +03:00
Yan Bondarenko 4dc6065ec9 fix(backups): give the sidecar sizing its reason per caller
The Keycloak comment and the helper doc claimed the tenant LimitRange as the
reason for the resources, but cozy-keycloak has no LimitRange and neither
does a tenant that leaves resourceQuotas empty: there the sidecar had no
requests and no limit at all, and the change gives it a 1Gi ceiling where
there was none. The comments and both doc blocks now say what applies to
each caller. No rendered object changes.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-18 17:30:20 +04:00
Yan Bondarenko 9214430bd8 fix(backups): give the barman-cloud sidecar its own resources
The barman-cloud plugin injects its sidecar without resource requests or
limits. Tenant namespaces ship a LimitRange that defaults containers to
128Mi, so the sidecar inherits that default and is OOMKilled mid-backup:
the ObjectStore stays healthy, the Cluster stays Ready, and only the
backup fails. A cgroup high-water mark of 254 MiB on a 38 MB database
shows the default cannot hold it.

The shared cozy-lib helper and the Go path that builds the platform's own
ObjectStore now request 100m/256Mi and cap memory at 1Gi. No CPU limit is
set, matching the entityOperator precedent in the kafka chart: a
throttled sidecar stalls WAL archiving instead of failing it.

The helper is renamed from checksumSidecarConfiguration to
sidecarConfiguration, since it no longer carries only the checksum pin.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-18 17:30:20 +04:00
Andrey KolkovandClaude Opus 4.8 5bcdb8cfb8 fix(postgres): let in-place restore re-bootstrap instead of wedging
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.

1. Archive-vs-recovery serverName collision. The driver set the restored
   cluster's WAL-archive serverName equal to the recovery source
   (bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
   chart pinned the archiver serverName to the release name. In place those
   coincide, so the cluster archived onto the source's own WAL prefix and
   barman-cloud-check-wal-archive refused to start ("Expected empty
   archive"). Fixed by giving the restored cluster a fresh
   bootstrap.newServerName, keyed by the RestoreJob UID (unique per
   restore, stable across reconciles); the chart archives there while
   externalClusters still recovers from bootstrap.serverName. This is what
   the manual CNPG strategies hit (serverName defaults to the cluster
   name); the cozy-default flow dodged it (serverName is <ns>-<app>).

2. useSystemBucket guard. The restore populates explicit S3 coordinates
   (destinationPath / s3CredentialsSecret / endpointCA) but left
   backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
   incompatible with useSystemBucket" guard failed the HelmRelease upgrade
   and the cluster never re-rendered. This is what the cozy-default
   (system-bucket) flow hits on an in-place restore. Fixed by clearing
   useSystemBucket in the restore patch, since the app is now on the
   explicit-creds path. The driver's app view (postgresapp.Backup) gains
   the field, without omitempty so a false survives the merge patch.

Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-25 17:31:53 +04:00
Andrey Kolkov 8aeedde088 feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383)
## What this PR does

Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported,
tested capability for the new `backups.cozystack.io` API. Fixes #2774.

Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the
`recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime`
plumbing; #3313 has since merged and this branch is rebased on `main`.
This PR adds the reliability + verification layer #2774 asked for.

### Changes

- **Precise failure reason for an unreachable target.** The restore
deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the
sole authority for "this restore is stuck" — the client waits the full
window either way. What changes is the *diagnosis*: once the deadline
has elapsed, if `recoveryTime` was set and the recovery Cluster never
went healthy, the CNPG driver reads the `full-recovery` pod logs and —
when several distinct attempts exited with PostgreSQL's `FATAL: recovery
ended before configured recovery target was reached` — fails the
RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason
`RecoveryTargetUnreachable`) and an actionable message, instead of a
bare generic timeout. Timing is unchanged; only the reason/message
differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.)
Design points:
- **We deliberately do NOT trip early off the FATAL.** A valid near-now
target emits that same FATAL transiently — repeatedly, for many
attempts, on a slow archiver — before it converges, so any earlier trip
would wrongly reject a recoverable restore (observed twice in CI). The
FATAL is used only to *explain* a failure the deadline has already
declared.
- **Health wins unconditionally** — the deadline/classification check
precedes the healthy check, but a recovery that converges within the
window reaches success and is never classified as failed (a live dev7
restore did exactly this).
- **Counting distinct FATAL-ing attempts** (not pod failures) tolerates
a one-off archive-lag miss on a near-now target and ignores transient
recovery-pod crashes (a node blip / brief API-server unreachability
never emits that FATAL).
- Adds the `pods/log` RBAC grant the log read needs (distinct
subresource from `pods`); a `Forbidden` is logged loudly and surfaced in
the failure message rather than silently disabling the classification.
- **e2e** (`examples/backups/postgres/run-all.sh`, wired into the
Chainsaw `postgres-2-backup-roundtrip` suite):
- Step 45 — write `before`, capture server timestamp (µs precision),
write `after`, restore to that timestamp, assert `before` present and
`after` absent.
- Step 46 — a `recoveryTime` an hour ahead must fail with reason
`RecoveryTargetUnreachable` once the deadline elapses (asserts the
*reason*, so it proves the classification fired, not just that a timeout
occurred).
- Chainsaw budget raised 25m → 40m to cover three sequential restores +
the post-deadline classification.
- **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a
`docs/operations/backup-classes.md` PITR section (recoverable window,
discovering earliest/latest restorable time from the backup catalog
since `firstRecoverabilityPoint` is unreliable under the plugin,
archive-lag tolerance, idempotency under GitOps).

### Verification

- `go build` / `go vet` clean; unit tests pass
(`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`,
`recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level
success-path test asserting `Succeeded` + `RecoveryConverged=True`, and
— added per review —
`TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`,
a reconcile-level test driving the deadline branch through all three
sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL
absent → generic timeout, `Forbidden` log read → generic timeout +
missing-`pods/log`-RBAC note. It fails if the call-site reason is
reverted to the generic one).
- Reviewed via adversarial branch-review; fixes include a critical
missing-RBAC dead-on-arrival, a classification false-positive on
transient recovery-pod crashes, and a sub-second e2e flake.
- **dev7**: backup + WAL archiving + recovery convergence verified live
on the #3313 build. The load-bearing assumption of the classification —
CNPG **accumulates** failed `full-recovery` pods rather than reusing one
— was confirmed (observed 7 simultaneous failed recovery pods). A
fully-green live step-46 run was blocked by intermittent COSI
bucket-provisioning flakiness on dev7; the negative path is covered by
the CI Chainsaw e2e.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The only user-facing surface is the new
`docs/operations/backup-classes.md` PITR section; the website keeps a
hand-mirrored copy per docs version.

- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: PITR docs mirror warranted (not yet opened; same pending
mirror as #3313's backup-classes changes)
- [x] No other downstream repository is affected by this change

```release-note
feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added PostgreSQL point-in-time recovery (PITR) support via RFC3339
`recoveryTime`.
* Introduced a PITR RestoreJob example, plus helper variables and an
unreachable-target scenario.
* **Bug Fixes**
* Improved restore deadline handling with deterministic, log-based
fail-fast classification for unreachable recovery targets, including
clearer conditions.
* **Documentation**
* Added detailed PITR operational documentation (recovery windows,
failure behavior, and GitOps/idempotent behavior).
* Expanded README and example comments for PITR verification and failure
expectations.
* **Tests**
* Extended unit tests and end-to-end chainsaw workflow to cover PITR
success and unreachable failures.
* **Chores**
* Updated RBAC to allow pod log reads; refined cleanup and run scripts
for additional RestoreJob artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-23 14:14:58 +04:00
Andrey KolkovandClaude Opus 4.8 b574e2176f fix(backups): cover Go-driven system-bucket ObjectStore + share the checksum helper
Address review on the S3 request-checksum fix:

- Extend the fix to the platform-managed useSystemBucket=true flow, whose
  ObjectStore is built in Go by applyClusterPluginBackup (SSA-applied), not
  chart-rendered. Add InstanceSidecarConfiguration to cnpgtypes.ObjectStoreSpec
  and set AWS_REQUEST_CHECKSUM_CALCULATION=when_required on it via
  barmanSidecarConfiguration(). The platform's own default system bucket is
  SeaweedFS, a non-AWS S3 gateway of the same class as Ceph RGW. Unit test in
  cnpgstrategy_controller_test.go asserts the applied ObjectStore carries it.

- Factor the duplicated instanceSidecarConfiguration block (3x across two
  charts) into a shared cozy-lib helper (cozy-lib.barman.checksumSidecarConfiguration).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
2026-07-22 15:35:32 +04:00
Andrey Kolkov ad3f8b2487 fix(backups): check cluster health before the restore deadline (PITR)
Addresses review CHANGES_REQUESTED on #3383.

CRITICAL: reconcileCNPGRestore checked the restore deadline BEFORE the
cluster-health check, so a reconcile that first observes "deadline elapsed"
AFTER the recovery Cluster already reached a healthy state (a controller
restart or stalled workqueue spanning restoreTimeoutSeconds) would mark an
already-converged restore Failed / RecoveryTargetUnreachable. A false Failed
can then escalate to data loss: a resubmitted RestoreJob's purge guard treats
the recovered Cluster as a stale leftover and deletes it + its PVCs.

Fix: check hasRecovery + cnpgClusterHealthy first and return Succeeded
unconditionally when healthy, regardless of how late the reconcile runs; only
a not-yet-healthy restore reaches the deadline/classification branch. Adds
TestReconcileCNPGRestore_HealthyPastDeadlineSucceeds (StartedAt 72h past the
deadline + healthy Cluster + recoveryTime set; the injected readPodLog fails
the test if the classification path is entered, proving health short-circuits).

MINOR: handle the deferred Close error on the pod-log reader (errcheck).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-22 15:31:31 +04:00