Every pass of a backup job waiting for another one reads the
cnpg.io/Backups from the apiserver, and the read covered every Backup in
the namespace, so a namespace with a long backup history paid for it on
each poll of each waiting job. The queue only looks at Backups this
controller created for the Cluster, which have carried the cluster and
owning-job labels since the driver was introduced, so it now asks the
apiserver for those alone.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The artifact took its destination from the record written when the
cnpg.io/Backup was created, but its serverName, the archive prefix
under that destination, from the live Cluster when the backup
completed. A restore that re-renders the Cluster in between gives it
another serverName, and the artifact then pairs the original
destination with a prefix the backup was never written to, so a later
restore searches the wrong archive. The record now carries the
serverName too; a record written before it did still falls back to the
live Cluster.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A backup job that is about to write the WAL archive destination first
checks for a restore of its application, because a write landing during
the restore puts the restored Cluster back on the source's WAL prefix
and the restore fails with an expected-empty-archive error after it has
purged the target. That check read RestoreJobs from the informer cache,
which can still miss one created a moment ago.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The rollout check takes the later of the latch and the Cluster's attach
record, and the record is never rewritten while the plugin stays
attached. A record that parses but lies in the future made every
instance look older than the plugin and counted the rollout deadline
from that time, so the job waited past the deadline by as far as the
record lay ahead, without end for one far enough out. Such a record
says no more about when the plugin was attached than one that does not
parse, and fails the job the same way.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A status write that fails with the apiserver unavailable takes the same
path as one lost to a conflict, but the attach it follows can itself be
the retry of an apply that failed first, which leaves its own condition
on the job. The test now runs that sequence as well, so the rollout check
is held to it whichever way the record was lost.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The rollout check takes the later of the latch and the Cluster's attach
record. A record left by an earlier attach of a plugin since removed is
older than the latch when someone who keeps no record attaches the
plugin again, and taking it would accept an instance recreated before
the latch as one that rolled with the plugin.
The record is absent not only when no backup job attached the plugin
but also when the chart did, or a controller that kept no record, so
the comment on the field says that.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The rollout check took the attach time from the PluginRolledOut
condition on the BackupJob, which the pass that attaches the plugin
writes only after the Cluster apply went through. When that status
write is lost to a conflict or a controller restart, the next pass
finds the plugin attached and falls back to the latch written before
the attach. An instance recreated between the latch and the attach then
counts as one that rolled with the plugin, and the cnpg.io/Backup is
created against an instance that has no plugin sidecar; a latch older
than the deadline also fails a rollout that has only just begun.
The apply that attaches the plugin now records the time on the Cluster
itself, so the record cannot be lost apart from the attach. The rollout
check and its deadline use the later of the two times. A record that
does not parse fails the job rather than guess which instances rolled.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The queue that keeps two BackupJobs of one application from writing the
WAL archive destination at once listed cnpg.io/Backups from the
informer cache. The controller runs one reconcile at a time under
leader election, so a Backup created by one job's pass already exists
when the next pass starts, but the cache can still miss it. The next
job then took the queue as empty, rewrote the destination under the
running backup and left that backup's recorded destination naming a
place its WAL no longer went. A job could likewise miss its own fresh
Backup, write the destination again and create a second Backup.
The queue now lists Backups from the apiserver, and a job that finds
no Backup of its own in the cache checks the apiserver before it
writes or creates one. Only a leader handover, when two processes
reconcile for a moment, can still let two jobs through.
The pass that creates the Backup no longer looks it up a second time
right before the create. Nothing can create it in between, and once
the lookup can reach the apiserver, a failed second lookup would come
after the destination write and fail the job instead of retrying.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The WAL archive destination is written by the pass that creates the
cnpg.io/Backup and not again while it runs, yet the artifact took the
destination, the endpoint and the credentials Secret from the strategy
rendered when the Backup completed. A strategy or BackupClass edited
while the backup ran gave an artifact that pointed somewhere nothing
had been written, and a restore from it found no data.
The pass that writes the destination now records it on the
cnpg.io/Backup it creates, and the artifact is built from that record.
The record lives on the Backup rather than on the BackupJob because a
tenant can edit its BackupJobs but has no access to postgresql.cnpg.io
objects: a record a tenant could rewrite would point the artifact, and
a restore from it, which takes its S3 keys from the strategy, at an
endpoint of the tenant's choosing.
The artifact size probe measures the same place: the recorded
location and credentials Secret, with the Secret keys and the endpoint
CA, which the record does not carry, from the strategy. Should those no
longer fit after an edit, the probe reports no size instead of
measuring somewhere else.
A record that cannot be read, or that names no destination path,
creates no artifact and fails the job, since the strategy may no longer
match the write. A Backup without a record was created by a controller
that kept none, and its artifact is built from the strategy as before.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Only the first attach after the latch restarted the rollout. In the
platform flow the chart renders no spec.plugins, so an in-place restore
recreates the Cluster without the plugin, and a backup job held through
that restore attaches it again once released. The rollout condition
then still carried its earlier start, the restored instances were newer
than it and ready, and the Backup was created in the same pass, before
CloudNativePG had rolled anything; such a Backup fails with "requested
plugin is not available" or "instance manager was restarted".
Every write of the job that finds the plugin missing now starts the
rollout over. A plugin attached by another writer is still timed from
the latch.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A job that latched the barman-cloud plugin rollout and then waited for
another backup of the application, or for a restore of it, had that
wait charged to the rollout deadline. Released after more than the
deadline, it failed on the first instance that was not ready yet with
"rollout did not finish", although no rollout of its own had been
running while it waited.
The hold is now recorded in a DestinationWriteHeld condition, and its
release time restarts the rollout deadline when it is later than the
rollout's start. The instance pods are still measured against the
rollout's start: one that rolled while the job was held carries the
plugin and counts.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The deadline no longer applies to a failed cnpg.io/Backup, which now
fails its BackupJob at once; it only bounds the wait for the
barman-cloud plugin rollout, and the comment still claimed the former.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The phase test read the BackupJob it passed to the reconciler, which
holds the phase in memory whether or not the status write happened. A
completed Backup could then leave the stored BackupJob Running, so the
Plan would never see the run finish, and no test in the package noticed.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
When two jobs of the application hold the queue, the job waiting
behind them names one of them in its Ready message, and the sort is
what keeps that name stable however the cache orders their Backups.
Without it the message can change from one poll to the next, and each
change writes the BackupJob status again; nothing failed when the sort
was removed.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Three guards of the CNPG driver could be reverted with the package
tests still green. The controller watches only BackupJobs, so a job
waiting in the queue or retrying the destination write is woken by
nothing but its own requeue, and the tests dropped the result. A job
whose Backup runs must not consult the queue, or two jobs that passed
it together wait on each other forever, and must patch neither the
ObjectStore nor the Cluster plugin, yet only the ObjectStore content
was checked.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The PluginRolledOut latch is recorded before the Cluster write that
attaches the barman-cloud plugin, and its time was used both as the
start of the rollout deadline and as the age every instance pod must
exceed to count as rolled. Now that a failing write is retried for up
to the backup deadline, the attach can come long after the latch: the
deadline was then spent before the rollout began, and a pod recreated
between the latch and the attach, without the plugin sidecar, passed
as rolled, so a Backup was created that CloudNativePG fails for good.
The latch now carries its own reason, and the write that finds the
plugin missing and attaches it re-stamps the condition, once, with the
time it was attempted. A plugin someone else attached after the latch,
another backup job or a restore, keeps the latch time: the rollout it
started can be over by the time this job writes, and its instances
would otherwise never count as rolled.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
The restore gate in Reconcile looks only at a BackupJob that has not
started, and the CNPG driver stamps StartedAt on its first pass. A job
that then waits behind another backup of the application, for as long
as that backup runs, writes the WAL archive destination and the
Cluster plugin once released without looking for a restore that began
in the meantime. A write landing while the restore re-renders the
Cluster can put the restored Cluster back on the source's WAL prefix,
where recovery fails with "Expected empty archive".
Every pass that is about to write the destination now looks for a
restore of the application and holds the job the way the gate does.
A job whose Backup already runs writes nothing and is not held. The
hold resets the retry budget of the destination write, as the wait
for another backup does, so time spent behind a restore does not fail
the job on the first error after it.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
The ObjectStore a Postgres Cluster archives its WAL to is shared by
every backup of the app, yet each running BackupJob server-side applied
its own strategy's destination onto it every five seconds with forced
ownership. Two BackupJobs of one app with different BackupClasses
flipped the destination back and forth until the healthy backup failed
too, and a webhook or apiserver blip during that apply failed the
BackupJob at once.
A BackupJob now writes the destination only before its cnpg.io/Backup
exists, and waits, with a WaitingForOtherBackup condition, while
another running BackupJob of the app has a Backup in flight. The check
reads the informer cache, so jobs released together, or one created
while another's Backup is not yet cached, can still both proceed; an
atomic per-application claim is a separate change.
Write errors that cannot succeed on retry (invalid, bad request,
forbidden) still fail the job at once. Any other is retried for the
backup deadline counted from its first failure since the job last left
the queue, recorded in a DestinationApplied condition. Once the Backup
exists the Cluster is only read, and a failed read is returned for a
controller retry instead of failing the running backup. The plugin
rollout deadline counts from the attach rather than from the job's
start, so queueing before the attach no longer shortens it.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A failed cnpg.io/Backup kept its BackupJob Running for 30 minutes as a
"transient" failure, on the premise that CloudNativePG may move a
Backup out of the failed phase. It does not: its backup controller
returns early on any Backup already failed or completed, so the wait
only delayed the verdict and hid it behind a misleading reason.
A failed Backup now fails the BackupJob at once with the error CNPG
recorded, pointing to the plugin-barman-cloud sidecar log when that
error is only the plugin's exit status. A Backup in any other phase
keeps the BackupJob waiting as before, now with a WaitingForCNPGBackup
condition naming the Backup and its phase instead of no condition at
all. It is still not capped: CloudNativePG keeps a healthy Backup
pending while another backup of the same Cluster runs or while it
restarts one, and a fixed deadline would fail those.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A recovery Cluster without the restored-server-name annotation fell back
to the creation-time comparison. On the current chart the annotation is
absent when bootstrap.newServerName is empty, i.e. a manually configured
recovery: re-rendered during the WAL-archive wait it is created after
StartedAt, the purge was skipped and the restore could report success on
the manual recovery source. Use the creation-time fallback only when the
application's bootstrap.newServerName already carries this restore's
server name, which keeps it for Clusters rendered by a chart that
predates the annotation.
Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The in-place CNPG restore skips deleting the live Cluster when it
carries bootstrap.recovery and was created after the RestoreJob started,
taking that as "this restore's own purge and re-render produced it". The
timestamp does not say which restore rendered the Cluster. A Cluster
deleted while the restore waits for the WAL archive is re-rendered by
the still-active HelmRelease from the previous restore's values and is
created after StartedAt too; the restore then skipped its purge and
reported Succeeded on the previous restore's data.
The restore patch already sets bootstrap.newServerName to
restoredServerName(cluster, RestoreJob UID). The chart now annotates the
Cluster with it as postgres.cozystack.io/restored-server-name, whether
or not it renders the barman plugin that also carries the value, and the
purge is skipped only when the annotation names this RestoreJob. The
mark lives on the rendered Cluster rather than in RestoreJob status, so
it still covers the failed status write the skip exists for. A Cluster
rendered by a chart that predates the annotation keeps the creation-time
check.
Fixes#3322
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
The first BackupJob on a Postgres app whose `cnpg.io/Cluster` has never
carried the barman-cloud plugin — a fresh platform-flow app, or one an
in-place restore re-rendered — attaches the plugin and creates the
`cnpg.io/Backup` in the same reconcile. Attaching the plugin changes the
instance pod spec, so CNPG rolls every instance (`Pod rollout required:
volumes: element plugins has been added`). The Backup then lands on an
instance without the plugin sidecar (`requested plugin is not available:
barman-cloud.cloudnative-pg.io`) or on one CNPG restarts under it
(`instance manager was restarted during backup`). CNPG does not retry a
failed Backup, so the BackupJob stays Running in
`CNPGBackupTransientFailure` until `cnpgDefaultBackupDeadline` and
fails. The first backup of each such Postgres is always lost; this was
observed on a v1.6.3 cluster and reproduced against `main`.
The driver now holds the Backup back until that rollout is over. Before
attaching the plugin to a Cluster that has none, it records
`PluginRolledOut=False` on the BackupJob, with `Ready=False` reason
`WaitingForPluginRollout` for the user, and requeues; the next pass
attaches the plugin. The Backup is created once CNPG reports the Cluster
in its healthy phase and every instance pod (`cnpg.io/cluster`,
`cnpg.io/podRole=instance`) was created after the condition and is
Ready; the condition then turns True and `Ready` is cleared, as before.
Recording the condition first means a lost status write leaves the
plugin unattached for the next pass instead of a Cluster that looks
ready while it rolls. A Cluster that already runs the plugin is not
rolled by the driver and its Backup is created on the first pass, as
today. The wait is bounded by the existing `cnpgDefaultBackupDeadline`,
after which the BackupJob fails with the phase or pod CNPG is stuck on.
Why this signal, checked against the CNPG 1.29 source and the 1.30
Cluster CRD vendored here: the Cluster status has no
`observedGeneration` and its `Ready` condition is derived from
`status.phase`, so right after the attach the phase still reads healthy
from before until CNPG reconciles the new spec. `status.pluginStatus`
does not help either, since CNPG also lists plugins referenced only from
`externalClusters` there, which is exactly the state of a Cluster a
restore re-rendered. A pod spec change is never applied in place (only a
pending Postgres restart is), so the instances are recreated and pods
newer than the attach are the ones running the sidecar. This reads pod
metadata and CNPG's documented labels only, not the plugin's container
or volume names, and needs no RBAC change: the controller already lists
and watches pods.
Tests: `TestReconcileCNPG_WaitsForPluginRolloutBeforeBackup` drives a
Cluster without the plugin through the attach, a stale healthy phase
over the old pod, a rolling phase over the recreated pod, and the
healthy phase, and asserts no Backup exists until the last step.
`TestReconcileCNPG_AttachedPluginDoesNotDelayBackup` covers a Cluster
that already runs the plugin, and
`TestReconcileCNPG_PluginRolloutWaitIsBounded` the deadline. With the
gate disabled, the first and third fail; with only the phase checked and
not the pods, the first fails at the attach pass.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
Not applicable: no UI change.
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
The diff touches only `internal/backupcontroller` Go code: no package,
values, chart, CRD, `hack/` tooling, telemetry metric or contributor
convention. Walked against the trigger map, it reaches none of the
repositories below, and the website documents no workaround for the lost
first backup that this would make obsolete.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
fix(backups): the CNPG backup driver no longer loses the first backup of a Postgres app whose Cluster had no barman-cloud plugin yet (a new platform-flow app, or one restored in place). It now waits for CNPG to finish rolling the instances onto the plugin before creating the cnpg.io Backup, and reports the wait on the BackupJob as Ready=False with reason WaitingForPluginRollout.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Reliability**
* Backups wait for the backup plugin rollout to complete and cluster
instances to become healthy before starting.
* If rollout does not finish before the backup deadline, the backup job
fails instead of waiting indefinitely.
* Backups can start without waiting for a new rollout when the plugin is
already attached.
* **Backup Configuration**
* When no server name is configured, backups use the cluster name and
application UID, giving recreated applications distinct archive
prefixes.
* Existing server names are preserved when the plugin is already
attached.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
A Cluster that has never carried the barman-cloud plugin gets it from
the first BackupJob: a fresh platform-flow app, or one an in-place
restore re-rendered. Attaching it changes the instance pod spec, so
CNPG rolls every instance. The cnpg.io/Backup the same reconcile
created then runs on an instance without the plugin sidecar ("requested
plugin is not available") or on one CNPG restarts under it ("instance
manager was restarted during backup"). CNPG never retries it, so the
BackupJob sat in CNPGBackupTransientFailure until the backup deadline
and failed: the first backup of every such Postgres was lost.
The driver now records a PluginRolledOut=False condition before it
attaches the plugin, and creates no Backup until CNPG reports the
Cluster healthy and every instance pod was created after that
condition and is Ready. Recording it first means a lost status write
leaves the plugin unattached for the next pass, rather than a Cluster
that looks ready while it rolls. A Cluster that already runs the plugin
is not rolled by the driver and is not held back.
The phase alone cannot tell the rollout is over. Cluster status has no
observedGeneration and its Ready condition is derived from the phase,
so until CNPG reconciles the new spec the phase still reads healthy
from before the attach. status.pluginStatus is no better: CNPG also
lists plugins referenced only from externalClusters there, which is
exactly the state of a Cluster a restore re-rendered. A pod spec change
is never applied in place, so the instances are recreated, and pods
newer than the attach are the ones running the sidecar. That reads only
pod metadata and CNPG's documented labels, not the plugin's container
names. The wait stays bounded by the existing backup deadline.
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
Two tests that came with the PostgreSQL major-version check call
createCNPGBackupArtifact and marshalCNPGBackupSnapshot with the
argument lists from before the flavor parameter was added, so the
backupcontroller test package does not compile on main. Both calls
take the default flavor the other tests in the file pass.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The default CNPG strategy archived a Postgres app under
<namespace>-<name>, inside a destinationPath that is also keyed by
namespace and name, and the driver fell back to the Cluster name when a
strategy set no serverName. Deleting the app does not empty the bucket,
so an app recreated under the same name archived its WALs and base
backups into its predecessor's stream: both start on timeline 1, and a
later restore of the old archive could pick the new cluster's base
backup or WAL segments.
barman-cloud refuses a non-empty archive only while
.check-empty-wal-archive exists in PGDATA. CNPG removes that file as
soon as ContinuousArchiving turns True, and with no plugin attached
archive_command is a successful no-op, so the file is gone minutes
after initdb, before the first BackupJob attaches the plugin.
Suffix the default serverName with the application UID, in the
strategy template and in the driver's fallback. The template guards
the UID with `with`, so an object without one renders the previous name
rather than "<no value>". A Cluster that already has the plugin keeps
its serverName, so only clusters never backed up take the new prefix,
and restores keep reading the source prefix from driverMetadata.
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
A CNPG restore is physical, so the recovered data files only open on the
PostgreSQL major version that wrote them. `reconcileCNPGRestore` purged
the target Cluster and its PVCs without comparing the two. Observed on
v1.6.3: a PostgreSQL 18 backup restored into a v16 application emptied
the target, the `full-recovery` pods then looped on `FATAL: database
files are incompatible with server`, and the RestoreJob failed at the
30-minute deadline with the target left empty. An in-place restore into
an application whose `spec.version` was raised since the backup ends the
same way.
The restore now compares the two majors before the purge, ahead of the
WAL-archive gate, and when they differ marks the RestoreJob `Failed`
with reason `MajorVersionMismatch` and a message naming both versions.
Nothing is suspended, patched or deleted on that path.
- Target major: the Postgres application's `spec.version` (`v13` to
`v18`). That is what the chart renders the recovered Cluster from once
the live one is gone, so the live Cluster's image would be the wrong
thing to compare.
- Backup major: the `cnpg.io/Backup`'s `status.majorVersion`, now copied
into the artifact's `driverMetadata` as `cnpg.io/major-version` at
backup time, because barman retention deletes the `cnpg.io/Backup` while
the artifact stays restorable. An artifact taken before this change
falls back to the `cnpg.io/Backup` while it still exists.
- When either side is unknown (no record and the `cnpg.io/Backup` gone,
a CNPG that predates `majorVersion`, an empty `spec.version`), the
restore goes ahead as before and the controller logs it, so existing
backups keep restoring.
`TestReconcileCNPGRestore_RefusesCrossMajorRestore` drives the real
reconcile against a fake client with a live Cluster and PVC on the
target and counts every `Delete`. With the check removed, the two
refusal cases (18 into a v16 copy, 16 in place into an app moved to v18)
and the fallback case fail: the Cluster and the PVC are deleted and the
app is patched for recovery. With the check in place they end `Failed`
with zero deletes, while the same-major and unknown-version cases still
reach the purge. `TestCreateCNPGBackupArtifact_RecordsMajorVersion`
covers the `driverMetadata` record.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Walked against the diff: it touches only `internal/backupcontroller`,
with no CRD, no app `values.yaml` or schema and no ApplicationDefinition
change. The new `cnpg.io/major-version` entry lives in the free-form
`Backup.spec.driverMetadata` map, which no downstream repository reads
by key.
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
fix(backups, postgres): a Postgres restore whose backup was taken on another PostgreSQL major version than the target's spec.version now fails before touching the target, instead of purging it and then failing to recover.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Cross-major PostgreSQL restores are now refused before the target is
changed. The error identifies the backup and target versions.
* Restores proceed when either version is unavailable or cannot be
parsed.
* **New Features**
* Backup metadata now records the PostgreSQL major version when it is
available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The chart refuses a flavor change on an existing Cluster, but the
Postgres app keeps the refused value. A BackupJob taken after such a
change recorded the app's flavor, not the one the data directory was
written by, so a RestoreJob would accept a target of the wrong image
family and refuse the right one.
The driver already reads the live Cluster to keep its barman
serverName; it now takes the flavor from that Cluster's imageName the
same way the chart's flavor guard compares images.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
A CNPG restore is physical: the recovered data files only open on the
major version that wrote them. The driver purged the target Cluster and
its PVCs without comparing the two, so a PostgreSQL 18 backup restored
into a v16 application, or in place into an application upgraded since
the backup, emptied the target. The recovery pods then looped on
"database files are incompatible with server" until the restore
deadline, and the RestoreJob failed with nothing left to fall back on.
The restore now fails before the purge, naming both versions, when they
differ. The target side is spec.version, since that is what the chart
renders the recovered Cluster from once the live one is gone. The
backup side is CNPG's status.majorVersion, copied into the artifact's
driverMetadata at backup time because retention deletes the
cnpg.io/Backup while the artifact stays restorable; artifacts taken
before that fall back to the cnpg.io/Backup. When either side is
unknown the restore goes ahead as before, so older backups keep
restoring.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
A recovery replays the source's data directory, so it carries the same
risk as switching the image of a live cluster: the postgresql and
postgis images are built on different Debian releases, and the glibc
change invalidates collatable indexes, while a postgis-to-postgresql
move also loses the libraries behind every PostGIS object. The chart
only guarded the live cluster, so a fresh release bootstrapped from a
backup of the other flavor went through, and the guard's own advice was
to restore a backup, which is that very move.
The backup-controller now records the source flavor in the snapshot it
stores on each Backup and fails a RestoreJob whose target runs another
flavor, before the purge deletes the target's Cluster and PVCs. A
snapshot without a flavor predates flavors and so comes from a
postgresql cluster. For the chart-managed bootstrap, the chart compares
against the recovery source's Cluster while it still exists. Both paths,
and the live-cluster guard, point a flavor change to a logical dump.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
applyClusterPluginBackup preserves the serverName already on the Cluster, so
that a backup does not flip the archive prefix mid-stream. It read that value
through the manager's cached client, and this controller registers
For(&BackupJob{}) and nothing else, so a Cluster only reaches the cache
through a lazily started informer with no resync tied to its own changes.
An in-place RestoreJob deletes the Cluster and the chart re-renders it with
bootstrap.newServerName seconds later. A read served during that window
describes the object that was purged, so the source's serverName is written
back over the freshly rendered one and the restored cluster archives onto the
source's WAL prefix. barman-cloud-check-wal-archive then refuses to start with
"Expected empty archive" and the RestoreJob ends on its deadline -- the
failure newServerName was added to prevent.
The stale read carries a stale UID too, and that UID owns the ObjectStore this
function applies, so the store is garbage-collected as soon as it is written.
Reading through the dynamic client keeps the preservation rule and makes the
value it rests on truthful.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Without recoveryTarget.backupID the barman-cloud plugin chooses the
base backup itself: the newest one in the catalog when no recoveryTime
is set, and the newest one ending by recoveryTime otherwise, on any
timeline since no targetTLI is passed. A RestoreJob naming an older
Backup therefore started from another one, and on an archive holding
sibling timelines - left by in-place restores that archived back under
the source serverName - from another branch, bringing back that
branch's data.
The restore reads barman's backupId and the backup's stoppedAt from
the live cnpg.io/Backup behind the Backup, and passes the id through
the chart as bootstrap.recovery.recoveryTarget.backupID. The plugin
gives backupID precedence over time-based selection, and CNPG does not
count it among the mutually exclusive targets, so it combines with
targetTime.
Two cases keep the plugin's choice. A recoveryTime earlier than one
second past stoppedAt: the named backup cannot serve an instant before
it is consistent, and stoppedAt is truncated to the second while
barman compares its end time to the microsecond. And a cnpg.io/Backup
that is gone: plugin-barman-cloud's retention deletes it together with
the base backup it describes, and the driver deletes one only with its
artifact, so pinning its id would purge the target and then fail the
recovery on every retry. That restore records a BaseBackupGone warning
event instead.
A re-restore into the same target clears a backupID left by a previous
one.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
## What this PR does
This replaces the open Renovate PRs with one rollup. Go modules move to
their newest releases that keep the Kubernetes libraries and
controller-runtime where they are, the toolchain and builder images move
to Go 1.27.1, and the first-party code gets the Go 1.27 modernizers.
Tests for the code the bumps touch come first, in their own commits, and
pass on both the old and the new dependencies.
Before the bumps:
- The e2e sandbox image builds again: dl.min.io answers 410 for mc since
MinIO archived it, so mc now comes from its last GitHub release with a
pinned checksum.
- Renovate stops offering tags of the apimachinery fork. They mirror
upstream, so #2999 is a downgrade that drops our patch.
- `make unit-tests` now runs the tests of the nested Go modules
(kubevirt-csi-driver, token-proxy, kubeovn-webhook, api/apps/v1alpha1).
They never ran in CI before.
- New tests: JWT verification in the dashboard token proxy, the kube-ovn
plunger metrics, cert-manager object idempotency in the tenant gateway,
values-file layering in the operator, and the kubevirt CSI driver
wrapper.
- One real fix. The lineage webhook decoded admitted objects with
encoding/json, so an int64 above 2^53 came back rounded. jsonpatch v2.5
compares numbers exactly and would emit a replace that rewrites such a
field in the pod. The webhook now decodes with apimachinery's json.
Held back on purpose, with the reasons in the commit messages:
- Kubernetes libraries stay on v0.35. #3001 needs the apimachinery fork
rebased onto release-1.37 first.
- cel-go v0.29.2 and etcd v3.6.15, the lines k8s.io/apiserver v0.35 is
built and tested with.
- grpc v1.83.2. v1.84.0 is affected by GO-2026-6443 and no v1.84 patch
release carries the fix yet.
- kubevirt.io/api and the CDI API stay on what upstream csi-driver
builds against. From CDI v1.60 it no longer compiles.
- api/apps/v1alpha1 keeps go 1.26.4, so consumers of the published types
are not forced onto Go 1.27.
- go fix runs without embedlit (Go 1.27-only syntax, breaks backports to
release branches on Go 1.26) and omitzero (would make metadata, spec and
status required in the published OpenAPI schemas), and one errorsastype
rewrite that did not compile is left out.
govulncheck reports nothing reachable in any of the five modules,
standard library included.
Supersedes #4311, #4310, #4309, #3452, #3993, #3927, #3902, #3805,
#3451, #3415, #3414, #3413, #3302, #4050 and #3363. #3001 stays open.
Not checked locally: the images that build upstream projects (kube-ovn,
kamaji, cilium, linstor-csi and others) with the Go 1.27 builder. The
image build jobs of this PR cover them.
Closes#4487.
### Screenshots
No UI changes.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
chore(deps): move the toolchain and builder images to Go 1.27.1 and raise Go dependencies to their latest compatible releases, closing the published advisories; the lineage webhook no longer rewrites int64 fields above 2^53 in admitted objects.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a restore resource policy option for Velero strategies.
* Added a test target to run Go tests across tracked modules.
* **Bug Fixes**
* Updated the end-to-end test environment to download and verify the
MinIO client from its current release location.
* Added coverage for token validation, webhook patching, controller
behavior, and reconciliation.
* **Maintenance**
* Updated the Go toolchain, dependencies, and container image versions.
* Refreshed pinned workflow actions and base image digests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
A restore sets bootstrap.enabled on the target app, which skips the
init-job, so the recovered roles keep the source's password hashes while
the target's credentials Secret holds generated ones and every
application login fails. Once recovery converges, the driver records
RecoveryConverged and then clears bootstrap.enabled, so the next
HelmRelease reconcile runs the init-job and applies the passwords. Ready
carries RestoreCompletedCredentialsPending and an Event says the
convergence is still pending. A clear that keeps failing is retried
under BootstrapDisablePending within a fixed grace window after
convergence and then fails the restore. bootstrap.enabled is serialized
without omitempty so the patch sends false instead of dropping the key.
Running the init-job on a restored cluster exposes its delete pass to a
catalog the spec does not describe: the driver sets spec.databases to
the Backup's snapshot, but recovery replays WAL past it, so a database
created after the Backup comes back still marked as managed and would
be dropped with its data. The first run after a restore, told apart by
the per-RestoreJob bootstrap.newServerName that the cluster has not yet
recorded, releases such a database and its roles from management
instead, and the run records the value once every pass has succeeded.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Run go fix over every first-party module: interface{} becomes any,
hand-written loops become slices, maps, min/max and range-over-int,
errors.As becomes errors.AsType, pointer helpers become new(expr) and
are inlined at their call sites, and so on. Every rewrite keeps
behaviour.
Three rewrites stay out. embedlit flattens embedded-field literals
such as ObjectMeta, syntax only Go 1.27 accepts, which would break
backports to release branches still on Go 1.26. omitzero drops
omitempty from struct fields, and openapi-gen reads omitempty as
optional, so metadata, spec and status would turn required in the
published schemas. And errorsastype rewrote an errors.As whose
target, apierrors.APIStatus, does not implement error, which
errors.AsType rejects at compile time.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The barman-cloud plugin injects its sidecar without resource requests or
limits. Tenant namespaces ship a LimitRange that defaults containers to
128Mi, so the sidecar inherits that default and is OOMKilled mid-backup:
the ObjectStore stays healthy, the Cluster stays Ready, and only the
backup fails. A cgroup high-water mark of 254 MiB on a 38 MB database
shows the default cannot hold it.
The shared cozy-lib helper and the Go path that builds the platform's own
ObjectStore now request 100m/256Mi and cap memory at 1Gi. No CPU limit is
set, matching the entityOperator precedent in the kafka chart: a
throttled sidecar stalls WAL archiving instead of failing it.
The helper is renamed from checksumSidecarConfiguration to
sidecarConfiguration, since it no longer carries only the checksum pin.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.
1. Archive-vs-recovery serverName collision. The driver set the restored
cluster's WAL-archive serverName equal to the recovery source
(bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
chart pinned the archiver serverName to the release name. In place those
coincide, so the cluster archived onto the source's own WAL prefix and
barman-cloud-check-wal-archive refused to start ("Expected empty
archive"). Fixed by giving the restored cluster a fresh
bootstrap.newServerName, keyed by the RestoreJob UID (unique per
restore, stable across reconciles); the chart archives there while
externalClusters still recovers from bootstrap.serverName. This is what
the manual CNPG strategies hit (serverName defaults to the cluster
name); the cozy-default flow dodged it (serverName is <ns>-<app>).
2. useSystemBucket guard. The restore populates explicit S3 coordinates
(destinationPath / s3CredentialsSecret / endpointCA) but left
backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
incompatible with useSystemBucket" guard failed the HelmRelease upgrade
and the cluster never re-rendered. This is what the cozy-default
(system-bucket) flow hits on an in-place restore. Fixed by clearing
useSystemBucket in the restore patch, since the app is now on the
explicit-creds path. The driver's app view (postgresapp.Backup) gains
the field, without omitempty so a false survives the merge patch.
Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What this PR does
Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported,
tested capability for the new `backups.cozystack.io` API. Fixes#2774.
Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the
`recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime`
plumbing; #3313 has since merged and this branch is rebased on `main`.
This PR adds the reliability + verification layer #2774 asked for.
### Changes
- **Precise failure reason for an unreachable target.** The restore
deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the
sole authority for "this restore is stuck" — the client waits the full
window either way. What changes is the *diagnosis*: once the deadline
has elapsed, if `recoveryTime` was set and the recovery Cluster never
went healthy, the CNPG driver reads the `full-recovery` pod logs and —
when several distinct attempts exited with PostgreSQL's `FATAL: recovery
ended before configured recovery target was reached` — fails the
RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason
`RecoveryTargetUnreachable`) and an actionable message, instead of a
bare generic timeout. Timing is unchanged; only the reason/message
differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.)
Design points:
- **We deliberately do NOT trip early off the FATAL.** A valid near-now
target emits that same FATAL transiently — repeatedly, for many
attempts, on a slow archiver — before it converges, so any earlier trip
would wrongly reject a recoverable restore (observed twice in CI). The
FATAL is used only to *explain* a failure the deadline has already
declared.
- **Health wins unconditionally** — the deadline/classification check
precedes the healthy check, but a recovery that converges within the
window reaches success and is never classified as failed (a live dev7
restore did exactly this).
- **Counting distinct FATAL-ing attempts** (not pod failures) tolerates
a one-off archive-lag miss on a near-now target and ignores transient
recovery-pod crashes (a node blip / brief API-server unreachability
never emits that FATAL).
- Adds the `pods/log` RBAC grant the log read needs (distinct
subresource from `pods`); a `Forbidden` is logged loudly and surfaced in
the failure message rather than silently disabling the classification.
- **e2e** (`examples/backups/postgres/run-all.sh`, wired into the
Chainsaw `postgres-2-backup-roundtrip` suite):
- Step 45 — write `before`, capture server timestamp (µs precision),
write `after`, restore to that timestamp, assert `before` present and
`after` absent.
- Step 46 — a `recoveryTime` an hour ahead must fail with reason
`RecoveryTargetUnreachable` once the deadline elapses (asserts the
*reason*, so it proves the classification fired, not just that a timeout
occurred).
- Chainsaw budget raised 25m → 40m to cover three sequential restores +
the post-deadline classification.
- **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a
`docs/operations/backup-classes.md` PITR section (recoverable window,
discovering earliest/latest restorable time from the backup catalog
since `firstRecoverabilityPoint` is unreliable under the plugin,
archive-lag tolerance, idempotency under GitOps).
### Verification
- `go build` / `go vet` clean; unit tests pass
(`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`,
`recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level
success-path test asserting `Succeeded` + `RecoveryConverged=True`, and
— added per review —
`TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`,
a reconcile-level test driving the deadline branch through all three
sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL
absent → generic timeout, `Forbidden` log read → generic timeout +
missing-`pods/log`-RBAC note. It fails if the call-site reason is
reverted to the generic one).
- Reviewed via adversarial branch-review; fixes include a critical
missing-RBAC dead-on-arrival, a classification false-positive on
transient recovery-pod crashes, and a sub-second e2e flake.
- **dev7**: backup + WAL archiving + recovery convergence verified live
on the #3313 build. The load-bearing assumption of the classification —
CNPG **accumulates** failed `full-recovery` pods rather than reusing one
— was confirmed (observed 7 simultaneous failed recovery pods). A
fully-green live step-46 run was blocked by intermittent COSI
bucket-provisioning flakiness on dev7; the negative path is covered by
the CI Chainsaw e2e.
### Downstream repositories
Walked the trigger map in `docs/agents/contributing.md` against the
diff. The only user-facing surface is the new
`docs/operations/backup-classes.md` PITR section; the website keeps a
hand-mirrored copy per docs version.
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: PITR docs mirror warranted (not yet opened; same pending
mirror as #3313's backup-classes changes)
- [x] No other downstream repository is affected by this change
```release-note
feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added PostgreSQL point-in-time recovery (PITR) support via RFC3339
`recoveryTime`.
* Introduced a PITR RestoreJob example, plus helper variables and an
unreachable-target scenario.
* **Bug Fixes**
* Improved restore deadline handling with deterministic, log-based
fail-fast classification for unreachable recovery targets, including
clearer conditions.
* **Documentation**
* Added detailed PITR operational documentation (recovery windows,
failure behavior, and GitOps/idempotent behavior).
* Expanded README and example comments for PITR verification and failure
expectations.
* **Tests**
* Extended unit tests and end-to-end chainsaw workflow to cover PITR
success and unreachable failures.
* **Chores**
* Updated RBAC to allow pod log reads; refined cleanup and run scripts
for additional RestoreJob artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review on the S3 request-checksum fix:
- Extend the fix to the platform-managed useSystemBucket=true flow, whose
ObjectStore is built in Go by applyClusterPluginBackup (SSA-applied), not
chart-rendered. Add InstanceSidecarConfiguration to cnpgtypes.ObjectStoreSpec
and set AWS_REQUEST_CHECKSUM_CALCULATION=when_required on it via
barmanSidecarConfiguration(). The platform's own default system bucket is
SeaweedFS, a non-AWS S3 gateway of the same class as Ceph RGW. Unit test in
cnpgstrategy_controller_test.go asserts the applied ObjectStore carries it.
- Factor the duplicated instanceSidecarConfiguration block (3x across two
charts) into a shared cozy-lib helper (cozy-lib.barman.checksumSidecarConfiguration).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Addresses review CHANGES_REQUESTED on #3383.
CRITICAL: reconcileCNPGRestore checked the restore deadline BEFORE the
cluster-health check, so a reconcile that first observes "deadline elapsed"
AFTER the recovery Cluster already reached a healthy state (a controller
restart or stalled workqueue spanning restoreTimeoutSeconds) would mark an
already-converged restore Failed / RecoveryTargetUnreachable. A false Failed
can then escalate to data loss: a resubmitted RestoreJob's purge guard treats
the recovered Cluster as a stale leftover and deletes it + its PVCs.
Fix: check hasRecovery + cnpgClusterHealthy first and return Succeeded
unconditionally when healthy, regardless of how late the reconcile runs; only
a not-yet-healthy restore reaches the deadline/classification branch. Adds
TestReconcileCNPGRestore_HealthyPastDeadlineSucceeds (StartedAt 72h past the
deadline + healthy Cluster + recoveryTime set; the injected readPodLog fails
the test if the classification path is entered, proving health short-circuits).
MINOR: handle the deferred Close error on the pod-log reader (errcheck).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review of #3383 found the RecoveryTargetUnreachable classification branch in
reconcileCNPGRestore was only covered through the pure helpers and
recoveryTargetUnreachable directly, never through reconcileCNPGRestore itself —
a mutation swapping the call-site reason back to the generic RestoreFailed kept
the suite green.
Add TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable: it
backdates StartedAt past the restore deadline with a recoveryTime set and drives
the real reconcile through the readPodLog seam, asserting all three deadline
sub-paths — FATAL present (reason RecoveryTargetUnreachable on both
RecoveryConverged and Ready), FATAL absent (generic timeout, no
RecoveryTargetUnreachable), and a Forbidden log read (generic timeout + the
missing-pods/log-RBAC note). Verified it fails when the call-site reason is
reverted to the generic RestoreFailed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested
capability for the new backups.cozystack.io API (issue #2774). The
recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing
already existed on the barman-cloud-plugin branch; this adds the reliability
and verification layer the issue asked for:
- Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set
and the recovery Cluster is not healthy, the driver reads the CNPG
full-recovery pod logs and, once several distinct attempts have exited with
PostgreSQL's "recovery ended before configured recovery target was reached"
FATAL, fails the RestoreJob with conditions RecoveryConverged=False /
Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the
restore deadline. Health is checked first, so a converged recovery always
wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a
transient archive-lag miss on a near-now target and ignores transient
recovery-pod crashes. Adds the pods/log RBAC grant the log read requires.
- e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw
postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures
the server timestamp, writes an after marker, restores to that timestamp and
asserts the before row survived and the after row did not; step 46 submits a
recoveryTime an hour ahead and asserts it fails fast with reason
RecoveryTargetUnreachable.
- Working, uncommented example (45-restorejob-pitr.yaml) and a docs section
(docs/operations/backup-classes.md) covering the recoverable window, how to
discover the earliest/latest restorable time from the backup catalog, the
archive-lag tolerance, and idempotency under GitOps.
Refs #2774
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Two defects of the same class — the WAL-archive S3 prefix (serverName)
changing or defaulting differently between components:
Recovery default regression (review blocker): native CNPG defaulted an
absent externalClusters serverName to the entry name (= bootstrap.oldName),
but the barman-cloud plugin defaults it to the NEW cluster's name, so a
legacy-flow restore with bootstrap.serverName unset would read the wrong
S3 prefix (hard failure, or silently restoring another cluster's data).
The chart now always emits serverName, defaulting to bootstrap.oldName.
Live-cluster serverName flip (found by the gated postgres backup e2e on a
real cluster): the chart renders spec.plugins without serverName
(effective prefix = cluster name) while a strategy template may name a
different one; the driver's ForceOwnership patch then flipped the prefix
mid-stream, splitting the WAL archive across two prefixes — the WAL
holding the backup's start was lost between them and the restore failed
with "WAL not found". The driver now preserves the live Cluster's
effective serverName (explicit parameter, or the cluster name when the
plugin is attached without one) and only applies the strategy's value to
a Cluster with no plugin attached; both charts pin serverName explicitly,
and the example strategy no longer sets one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-up (cozystack#3300): ensureCNPGBackup stamps spec.method=plugin +
spec.pluginConfiguration on the postgresql.cnpg.io/Backup it creates — the field
that routes a driver-initiated run through the barman-cloud plugin. If it
regressed to the default (barmanObjectStore) method, CNPG would run the legacy
method against a Cluster that no longer has one and every platform backup would
fail. Add TestEnsureCNPGBackup_UsesPluginMethod asserting method, plugin name,
and cluster ref. (The chart ScheduledBackup path was already covered; the driver
path was not.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-ups (cozystack#3300):
- Lifecycle/leak: applyClusterPluginBackup now sets an ownerReference from the
driver-created barmancloud.cnpg.io/ObjectStore to its cnpg.io/Cluster, so
Kubernetes garbage-collects it when the Cluster is deleted (tenant app
teardown or the RestoreJob purge). In the platform useSystemBucket flow the
chart does not render this ObjectStore, so previously it was orphaned in the
tenant namespace. Drop the now-accurate-only 'delete' verb on objectstores
from the driver RBAC (GC handles removal; the driver never deletes it) and
correct the RBAC comment that falsely claimed the purge deletes it.
- Diagnostics/comments: the RestoreJob WAL-archive-deadline failure message and
several doc comments still told operators to check spec.backup.barmanObjectStore
/ externalClusters[].barmanObjectStore, fields no longer emitted after the
migration. Point them at the barman-cloud plugin (spec.plugins + ObjectStore,
externalClusters[].plugin) instead so they don't mislead during an incident.
- Test: TestApplyClusterPluginBackup_PatchesExistingCluster now asserts the
ObjectStore carries an ownerReference to the Cluster (kind/name/uid).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Address review feedback on the repeat in-place restore fix:
- Make the freshness discriminator strict (creationTimestamp > StartedAt) so
an exact tie resolves to "not fresh" and the caller purges, matching the
function's stated conservative default; a fresh Cluster is always created
well after StartedAt so this never misclassifies the status-write-race path.
- Document why a wall-clock timestamp comparison was chosen over recording the
purged Cluster's UID on the TargetPurged condition (the UID approach leans on
the same status write the race path assumes can fail).
- Add a reconcile-level regression test that drives reconcileCNPGRestore end to
end with a fake client: a stale leftover recovery Cluster from a prior restore
is purged, and a Cluster this restore just re-created is not re-purged. This
covers the actual call site where #3311 lived, not just the pure helpers.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
A repeat in-place PostgreSQL RestoreJob into an already-restored target
silently no-opped: the destructive purge was skipped because the live
Cluster still carried spec.bootstrap.recovery left over from the previous
restore, and the re-purge guard mistook that for a freshly-recovered
Cluster produced by this restore's own purge. The job then reported
Succeeded while the PVC, disk, and data were never touched.
Tell the two apart by the Cluster's creationTimestamp relative to the
RestoreJob's StartedAt: only a recovery Cluster created at/after StartedAt
is treated as this restore's own and skipped; one that predates StartedAt
is leftover from an earlier completed restore and is purged so the restore
re-bootstraps from the backup.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>