fix(postgres): stop backup jobs from fighting over the WAL archive destination (#5221)

## What this PR does

Part of #5202. A second PR will restore the last good destination when a
job that changed it fails.

The ObjectStore a Postgres Cluster archives its WAL to is shared by
every backup of the application, yet every running BackupJob server-side
applied its own strategy's destination onto it every five seconds with
forced ownership. Two BackupJobs of one application with different
BackupClasses flipped the destination back and forth until the healthy
backup failed too, and a webhook or apiserver blip during that apply
failed the BackupJob at once.

- A BackupJob writes the destination and the Cluster plugin only before
its `cnpg.io/Backup` exists. Afterwards it only reads the live Cluster,
and a failed read is returned for a controller retry instead of failing
the running backup.
- Before writing, a BackupJob waits, with a `Ready=False` /
`WaitingForOtherBackup` condition naming the other job, while another
running BackupJob of the application has a `cnpg.io/Backup` in flight.
The wait is not capped, for the same reason as in #5211. The queue reads
the Cluster's own Backups from the apiserver; with leader election one
reconcile runs at a time, so only a leader handover can still let two
jobs through. An atomic per-application claim is #4559.
- Before writing, a BackupJob also checks the apiserver for a restore of
its application and holds while one runs, including a job released from
the queue: a write landing during the restore would put the restored
Cluster back on the source's WAL prefix.
- Write errors that cannot succeed on retry (invalid, bad request,
forbidden) still fail the job at once. Any other is retried with a
`ClusterPluginApplyRetrying` condition for the backup deadline, counted
from the first failure since the job last waited, and recorded in a new
`DestinationApplied` condition.
- The pass that creates the `cnpg.io/Backup` records on it the
destination, endpoint, credentials Secret and serverName it wrote
(`backups.cozystack.io/cnpg-written-destination`), and the artifact and
its size probe are built from that record, so a strategy edited, or a
Cluster re-rendered, while the backup runs does not redirect the
artifact. A Backup without the record falls back to the strategy as
before.
- The plugin rollout deadline and the instance check count from the
write that attached the plugin, not from the job's start. The apply that
attaches it also stamps the time on the Cluster
(`backups.cozystack.io/barman-plugin-attached-at`), so a lost BackupJob
status write after the attach does not fall back to an older time, and a
job that starts while another job's recent attach is still rolling waits
for that rollout too. A stamp a few minutes ahead of the controller's
clock is waited out; one further ahead is treated as corrupt.

Tests drive the reconciler through a running backup that must not
rewrite the destination, the queue and its release, a restore starting
while a job waits, each class of write error, a recovered write, a
failed Cluster read while the Backup runs, lost status writes around the
attach, another job's rollout, clock drift on the attach stamp, the
recorded destination and serverName, and the rollout deadline. Each
guard was checked by mutation.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): Postgres BackupJobs of the same application no longer overwrite each other's WAL archive destination: a job writes it only before its backup starts and waits while another backup or a restore of the application is in flight. A transient error while writing it is retried instead of failing the job at once, and the backup artifact names the destination and serverName the backup was actually written to.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Reliability**
* CNPG backups wait while another backup or restore is using the shared
destination, and retry temporary destination-write failures while
observing the backup deadline.
* Backup artifacts and size checks use the destination recorded when the
backup was created, even if destination settings later change.
* Backup processing handles plugin rollout timing and destination-hold
release more consistently.
* **Restore**
* Restore operations account for shared WAL archive destination
conflicts and guard against incompatible backup selections.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
scooby87 authored and GitHub committed 2026-11-13 08:58:55 +03:00
commit e978ed7f2f
5 files changed
+2949 -80

No files matched your search

@@ -315,14 +315,6 @@ func (r *BackupJobReconciler) handleProjectionError(ctx context.Context, j *back
}
// markBackupJobFailed records a terminal Failed phase on the BackupJob.
//
// Coupling note: a failure that fires before reconcileCNPG's StartedAt block
// has set backupJob.Status.StartedAt leaves StartedAt nil. The CNPG path's
// cnpgDefaultBackupDeadline check only fires once StartedAt is set, so an
// early-failure retry that reaches the StartedAt block restarts the deadline
// budget from the retry's StartedAt - intentional, since retries are
// user-driven and a fresh budget is what users expect after correcting the
// cause of the previous failure.
func (r *BackupJobReconciler) markBackupJobFailed(ctx context.Context, backupJob *backupsv1alpha1.BackupJob, message string) (ctrl.Result, error) {
logger := getLogger(ctx)
now := metav1.Now()
@@ -32,13 +32,13 @@ const ConditionReasonRestoreInProgress = "RestoreInProgress"
//
// A backup must not start against an application a restore is rewriting. It
// would capture a half-restored state at best, and on the CNPG path it does
// worse: every BackupJob reconcile re-applies the Cluster's barman plugin, and
// one landing between the restore re-rendering the Cluster with its new
// serverName and the plugin being attached can put the restored cluster back
// on the source's WAL prefix, where its recovery fails with "Expected empty
// archive". Plan-created runs are the ones that hit this, since nothing else
// looks at restores before creating a BackupJob.
func activeRestoreTargeting(ctx context.Context, c client.Client, namespace string, app corev1.TypedLocalObjectReference) (string, error) {
// worse: a BackupJob applies the Cluster's barman plugin before it creates
// its Backup, and an apply landing between the restore re-rendering the
// Cluster with its new serverName and the plugin being attached can put the
// restored cluster back on the source's WAL prefix, where its recovery fails
// with "Expected empty archive". Plan-created runs are the ones that hit
// this, since nothing else looks at restores before creating a BackupJob.
func activeRestoreTargeting(ctx context.Context, c client.Reader, namespace string, app corev1.TypedLocalObjectReference) (string, error) {
app = NormalizeApplicationRef(app)
list := &backupsv1alpha1.RestoreJobList{}
if err := c.List(ctx, list, client.InNamespace(namespace)); err != nil {
@@ -80,7 +80,7 @@ func activeRestoreTargeting(ctx context.Context, c client.Client, namespace stri
// reads options.targetNamespace, so the option is looked at only when the
// Backup's strategy is Velero — the CNPG and Job-based drivers ignore it and
// restore in place.
func restoreJobTarget(ctx context.Context, c client.Client, rj *backupsv1alpha1.RestoreJob) (corev1.TypedLocalObjectReference, bool, error) {
func restoreJobTarget(ctx context.Context, c client.Reader, rj *backupsv1alpha1.RestoreJob) (corev1.TypedLocalObjectReference, bool, error) {
backup := &backupsv1alpha1.Backup{}
if err := c.Get(ctx, client.ObjectKey{Namespace: rj.Namespace, Name: rj.Spec.BackupRef.Name}, backup); err != nil {
if client.IgnoreNotFound(err) == nil {
@@ -130,13 +130,14 @@ func ptrString(s *string) string {
return *s
}
// holdForRestore keeps a BackupJob that has not started yet waiting while a
// restore writes into its application. It stays non-terminal, so a scheduled
// run still happens once the restore ends, only later than its slot.
// holdForRestore keeps a BackupJob waiting while a restore writes into its
// application: one that has not started yet, or a CNPG one about to write
// its WAL archive destination. It stays non-terminal, so a scheduled run
// still happens once the restore ends, only later than its slot.
//
// A failed status write is returned so the manager retries it with backoff.
// The job is held either way, since nothing past the gate ran, but a hold
// nobody can see reads as a run stuck for no reason.
// The job is held either way, since it writes nothing past the hold, but a
// hold nobody can see reads as a run stuck for no reason.
func (r *BackupJobReconciler) holdForRestore(ctx context.Context, j *backupsv1alpha1.BackupJob, restore string) (ctrl.Result, error) {
if j.Status.Phase == backupsv1alpha1.BackupJobPhaseEmpty {
j.Status.Phase = backupsv1alpha1.BackupJobPhasePending
@@ -284,7 +284,7 @@ func TestCreateCNPGBackupArtifact_SizeBytes(t *testing.T) {
r := reconcilerWithLister(t, l)
r.Client = newCNPGStrategyTestClient(t, s3CredsSecret("tenant", "creds"))
got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant"))
got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "srv", "postgresql", cnpgDestinationOf(rendered), rendered, newPostgresApp("pg", "tenant"))
if err != nil {
t.Fatalf("createCNPGBackupArtifact: %v", err)
}
@@ -311,7 +311,7 @@ func TestCreateCNPGBackupArtifact_ExistingBackupSkipsProbe(t *testing.T) {
cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}}
rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()}
if _, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")); err != nil {
if _, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", cnpgDestinationOf(rendered), rendered, newPostgresApp("pg", "tenant")); err != nil {
t.Fatalf("createCNPGBackupArtifact: %v", err)
}
if l.calls != 0 {
@@ -331,7 +331,7 @@ func TestCreateCNPGBackupArtifact_AlreadyExistsRefetchReadsPastTheCache(t *testi
store := newCNPGStrategyTestClient(t, backup)
r := &BackupJobReconciler{Client: staleCache(t, store), APIReader: notFoundOnce(t, store)}
rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://b/"}}
got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}}, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant"))
got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}}, "postgres-pg", "srv", "postgresql", cnpgDestinationOf(rendered), rendered, newPostgresApp("pg", "tenant"))
if err != nil || got == nil || got.Name != "bj-race" {
t.Fatalf("createCNPGBackupArtifact = %v, %v; want the Backup that already exists", got, err)
}
@@ -349,7 +349,7 @@ func TestCreateCNPGBackupArtifact_UnreadableExistingBackupIsRetried(t *testing.T
store := newCNPGStrategyTestClient(t, backup)
r := &BackupJobReconciler{Client: staleCache(t, store), APIReader: failingReads(t, store)}
rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://b/"}}
_, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}}, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant"))
_, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}}, "postgres-pg", "srv", "postgresql", cnpgDestinationOf(rendered), rendered, newPostgresApp("pg", "tenant"))
if !errors.Is(err, errBackupNotReadBack) {
t.Errorf("err = %v, want errBackupNotReadBack", err)
}
@@ -409,7 +409,7 @@ func TestCreateCNPGBackupArtifact_FailedExistenceCheckSkipsProbe(t *testing.T) {
cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}}
rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()}
got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant"))
got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", cnpgDestinationOf(rendered), rendered, newPostgresApp("pg", "tenant"))
if err != nil || got == nil {
t.Fatalf("createCNPGBackupArtifact = %v, %v; a transient read must not fail the backup", got, err)
}
@@ -451,3 +451,128 @@ func TestReconcileCNPG_RejectedCreateFailsTheJob(t *testing.T) {
t.Errorf("phase = %q, want Failed for a rejected Create", persisted.Status.Phase)
}
}
// The probe measures the base backup where it was written: the destination
// recorded on the cnpg.io/Backup when it was created, not the strategy as
// it reads by the time the backup completes.
func TestCreateCNPGBackupArtifact_SizeProbeReadsTheRecordedDestination(t *testing.T) {
apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup
j := &backupsv1alpha1.BackupJob{
ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-moved"},
Spec: backupsv1alpha1.BackupJobSpec{
ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: "pg"},
},
}
cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}}
rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()}
oldCreds := "creds-before-edit"
written := cnpgDestination{DestinationPath: "s3://old-bkt/before/edit/", EndpointURL: "https://old.example:9000", S3SecretRef: &oldCreds}
l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: "before/edit/srv/base/20261021T100000/data.tar.gz", Size: 42}}}
r := reconcilerWithLister(t, l)
r.Client = newCNPGStrategyTestClient(t, s3CredsSecret("tenant", oldCreds))
got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", written, rendered, newPostgresApp("pg", "tenant"))
if err != nil {
t.Fatalf("createCNPGBackupArtifact: %v", err)
}
if l.bucket != "old-bkt" || l.prefix != "before/edit/srv/base/20261021T100000/" {
t.Errorf("listed %s/%s, want the recorded destination old-bkt/before/edit/srv/base/20261021T100000/", l.bucket, l.prefix)
}
if l.conn.Endpoint != "old.example:9000" {
t.Errorf("probed endpoint %q, want the recorded old.example:9000", l.conn.Endpoint)
}
if got.Status.Artifact == nil || got.Status.Artifact.SizeBytes != 42 {
t.Errorf("Artifact = %#v, want SizeBytes 42", got.Status.Artifact)
}
}
// A backup recorded without a credentials Secret is not measured with the
// Secret the strategy names now: that Secret belongs to another location.
func TestCreateCNPGBackupArtifact_SizeProbeSkipsARecordWithoutCredentials(t *testing.T) {
apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup
j := &backupsv1alpha1.BackupJob{
ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-nocreds"},
Spec: backupsv1alpha1.BackupJobSpec{
ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: "pg"},
},
}
cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}}
rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()}
written := cnpgDestination{DestinationPath: "s3://bkt/tenant/pg/", EndpointURL: "https://s3.example:8443"}
l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: "x", Size: 1}}}
r := reconcilerWithLister(t, l)
r.Client = newCNPGStrategyTestClient(t, s3CredsSecret("tenant", "creds"))
got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", written, rendered, newPostgresApp("pg", "tenant"))
if err != nil {
t.Fatalf("createCNPGBackupArtifact: %v", err)
}
if l.calls != 0 {
t.Errorf("S3 listed %d times with the strategy's Secret for a backup recorded without one", l.calls)
}
if got.Status.Artifact == nil || got.Status.Artifact.SizeBytes != 0 {
t.Errorf("Artifact = %#v, want no size", got.Status.Artifact)
}
}
// probeRecordedBackup runs createCNPGBackupArtifact for a backup recorded as
// written with credentials Secret recordedSecret, against rendered, and
// returns the size it measured and the connection it measured through.
func probeRecordedBackup(t *testing.T, rendered *strategyv1alpha1.CNPGTemplate, recordedSecret *corev1.Secret) (int64, s3Connection) {
t.Helper()
apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup
j := &backupsv1alpha1.BackupJob{
ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-probe"},
Spec: backupsv1alpha1.BackupJobSpec{
ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: "pg"},
},
}
cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}}
secretName := recordedSecret.Name
written := cnpgDestination{DestinationPath: "s3://bkt/tenant/pg/", EndpointURL: "https://s3.example:8443", S3SecretRef: &secretName}
l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: "tenant/pg/srv/base/20261021T100000/data.tar.gz", Size: 42}}}
r := reconcilerWithLister(t, l)
r.Client = newCNPGStrategyTestClient(t, recordedSecret)
got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", written, rendered, newPostgresApp("pg", "tenant"))
if err != nil {
t.Fatalf("createCNPGBackupArtifact: %v", err)
}
if got.Status.Artifact == nil {
t.Fatalf("no artifact")
}
return got.Status.Artifact.SizeBytes, l.conn
}
// A strategy that has since dropped its credentials still measures a backup
// recorded with a Secret, through that Secret and its default keys.
func TestCreateCNPGBackupArtifact_SizeProbeOfARecordedSecretUnderAStrategyWithoutCredentials(t *testing.T) {
store := newSizedBarmanObjectStore()
store.S3Credentials = nil
size, conn := probeRecordedBackup(t, &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *store}, s3CredsSecret("tenant", "creds-recorded"))
if size != 42 {
t.Errorf("SizeBytes = %d, want 42 measured through the recorded Secret", size)
}
if conn.AccessKey != "AKID" {
t.Errorf("probed with access key %q, want the recorded Secret's AKID", conn.AccessKey)
}
}
// The record names the Secret, not its keys: the strategy's key names stay,
// so a strategy with its own key names still reads the recorded Secret.
func TestCreateCNPGBackupArtifact_SizeProbeKeepsTheStrategysSecretKeys(t *testing.T) {
store := newSizedBarmanObjectStore()
store.S3Credentials.AccessKeyIDKey = "id"
store.S3Credentials.SecretAccessKeyKey = "key"
secret := &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "creds-recorded"},
Data: map[string][]byte{"id": []byte("CUSTOMID"), "key": []byte("custom-secret")},
}
size, conn := probeRecordedBackup(t, &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *store}, secret)
if size != 42 {
t.Errorf("SizeBytes = %d, want 42 measured with the strategy's key names", size)
}
if conn.AccessKey != "CUSTOMID" {
t.Errorf("probed with access key %q, want CUSTOMID read under the strategy's key name", conn.AccessKey)
}
}
@@ -15,6 +15,7 @@ import (
"time"
corev1 "k8s.io/api/core/v1"
apiequality "k8s.io/apimachinery/pkg/api/equality"
apierrors "k8s.io/apimachinery/pkg/api/errors"
apimeta "k8s.io/apimachinery/pkg/api/meta"
"k8s.io/apimachinery/pkg/api/resource"
@@ -175,11 +176,17 @@ const (
// against an unreachable target reproduces the same FATAL.
cnpgRecoveryMaxInspectPods = 5
// Cap on the wall-clock time a BackupJob waits for the barman-cloud
// plugin rollout before the cnpg.io/Backup exists. A Backup that exists
// is not capped: CloudNativePG keeps a healthy one pending while another
// backup of the same Cluster runs, or while it restarts one on another
// instance, so no fixed deadline tells those apart from a stuck one.
// Cap on the wall-clock time a BackupJob spends before its
// cnpg.io/Backup exists on each of two waits. The plugin rollout is
// counted from the attach, the later of the PluginRolledOut condition and
// the Cluster's cnpgPluginAttachedAtAnnotation, or from the release
// recorded in DestinationWriteHeld when that is later. Retrying the WAL
// archive destination write is counted from the first failure since the
// last successful write or the last wait for another backup or a
// restore. A Backup that exists is not capped: CloudNativePG keeps a
// healthy one pending while another backup of the same Cluster runs, or
// while it restarts one on another instance, so no fixed deadline tells
// those apart from a stuck one.
cnpgDefaultBackupDeadline = 30 * time.Minute
cnpgClusterHealthyPhase = "Cluster in healthy state"
@@ -190,11 +197,69 @@ const (
// instance pod spec (the plugin injects its sidecar), so CNPG rolls every
// instance, and a cnpg.io/Backup created meanwhile fails with "requested
// plugin is not available" or "instance manager was restarted during
// backup" and never recovers. False holds the Backup back; its
// LastTransitionTime, written before the plugin is attached, is what the
// instance pods must postdate (see cnpgPluginRolloutPending).
// backup" and never recovers. False holds the Backup back; the instance
// pods must postdate its LastTransitionTime and the Cluster's
// cnpgPluginAttachedAtAnnotation (see cnpgReasonAttachingPlugin and
// cnpgPluginRolloutPending).
backupCondPluginRolledOut = "PluginRolledOut"
// Reason of the PluginRolledOut latch recorded before the write that
// attaches the plugin. Every write of this job that attaches it replaces
// the condition with its own time, which retries can put long after the
// latch.
cnpgReasonAttachingPlugin = "AttachingPlugin"
// Condition Type recorded while writing the WAL archive destination keeps
// failing. Its LastTransitionTime marks the first failure, so the retry
// budget is not spent by time the job waited for another backup of the
// app, or for a restore of it, before it tried.
backupCondDestinationApplied = "DestinationApplied"
// Condition Type recorded while the job is held off writing the WAL
// archive destination, by another backup of the app in flight or by a
// restore of it, and set False when released. The release time restarts
// the plugin rollout deadline, which that wait must not spend; the
// instance pods still have to postdate the rollout's own start.
backupCondDestinationWriteHeld = "DestinationWriteHeld"
// Annotation on a cnpg.io/Backup carrying, as JSON, the WAL archive
// destination written by the pass that created it: the destination
// path, the endpoint URL, the name of the credentials Secret and the
// serverName the Cluster archived under.
cnpgWrittenDestinationAnnotation = "backups.cozystack.io/cnpg-written-destination"
// Annotation on a cnpg.io/Cluster carrying, as RFC 3339, when a backup
// job attached the barman-cloud plugin. It is written by the same apply
// that attaches the plugin, so unlike the PluginRolledOut condition it
// cannot be lost to a failed BackupJob status write after the attach.
cnpgPluginAttachedAtAnnotation = "backups.cozystack.io/barman-plugin-attached-at"
// Annotation on the ObjectStore carrying, as RFC 3339, when a backup job
// last changed its configuration. It is written by the apply that makes
// the change, so it cannot be lost apart from it.
cnpgDestinationChangedAtAnnotation = "backups.cozystack.io/destination-changed-at"
// Annotation on the ObjectStore naming the BackupJob whose apply last
// wrote it. Written by that apply, it holds the destination for that job
// while it runs and has no cnpg.io/Backup of its own yet; from then on
// the queue sees that Backup instead.
cnpgDestinationWrittenByAnnotation = "backups.cozystack.io/destination-written-by"
// How long after an ObjectStore configuration change a Backup is held.
// The plugin-barman-cloud sidecar reads the ObjectStore through its own
// cache, kept for DefaultTTLSeconds = 10 (plugin-barman-cloud v0.13.0,
// internal/cnpgi/instance/internal/client/client.go) and invalidated
// only by the sidecar's own writes, so a Backup taken sooner can use the
// previous configuration and land in the previous bucket. The margin
// covers the cache's whole-second expiry and the record's whole-second
// timestamp.
cnpgObjectStoreCacheSettle = 15 * time.Second
// How far ahead of this controller's clock a recorded time, of a plugin
// attach or of a destination change, may be before it is taken as
// corrupt rather than as written by a leader whose clock ran ahead.
cnpgAttachClockSkewTolerance = 5 * time.Minute
cnpgPodRoleLabel = "cnpg.io/podRole"
cnpgPodRoleInstance = "instance"
)
@@ -321,7 +386,53 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp
rollout := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut)
awaitingRollout := existingBackup == nil && rollout != nil && rollout.Status == metav1.ConditionFalse
attachment, err := r.applyClusterPluginBackup(ctx, j.Namespace, clusterName, rendered, serverName, existingBackup != nil || awaitingRollout)
// The ObjectStore is the Cluster's WAL archive destination, shared by
// every backup of the app, so it is written only before this job's
// Backup exists, and only while no other job's Backup is in flight and
// no other running job without a Backup yet holds the destination it
// last wrote.
writeDestination := existingBackup == nil
if writeDestination {
// The gate in Reconcile sees only a job that has not started, and
// this one may have waited in the queue for as long as a restore
// takes to begin. Read from the apiserver: the cache can still miss
// a RestoreJob created a moment ago.
restore, err := activeRestoreTargeting(ctx, r.apiReader(), j.Namespace, j.Spec.ApplicationRef)
if err != nil {
return ctrl.Result{}, err
}
if restore != "" {
apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondDestinationApplied)
apimeta.SetStatusCondition(&j.Status.Conditions, cnpgDestinationWriteHeld(ConditionReasonRestoreInProgress))
return r.holdForRestore(ctx, j, restore)
}
holder, err := r.otherCNPGBackupInFlight(ctx, j, clusterName)
if err != nil {
return ctrl.Result{}, err
}
if holder == "" {
if holder, err = r.destinationClaimHolder(ctx, j, clusterName); err != nil {
return ctrl.Result{}, err
}
}
if holder != "" {
apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondDestinationApplied)
apimeta.SetStatusCondition(&j.Status.Conditions, cnpgDestinationWriteHeld("WaitingForOtherBackup"))
return r.waitCNPGBackupJob(ctx, j, "WaitingForOtherBackup", fmt.Sprintf(
"waiting for BackupJob %s of the same application to finish its backup, which writes to the shared WAL archive destination", holder))
}
if apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationWriteHeld) != nil {
apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{
Type: backupCondDestinationWriteHeld,
Status: metav1.ConditionFalse,
Reason: "Released",
Message: "nothing holds the WAL archive destination write any more",
})
}
}
attemptedAt := metav1.Now()
attachment, err := r.applyClusterPluginBackup(ctx, j.Namespace, clusterName, rendered, serverName, existingBackup != nil || awaitingRollout, writeDestination, j.Name)
if err != nil {
if apierrors.IsNotFound(err) {
// HelmRelease has not yet rendered the Cluster (fresh app, or
@@ -340,7 +451,29 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp
}
return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil
}
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to attach barman-cloud plugin to Cluster: %v", err))
if !writeDestination {
// Only the live Cluster was read for a Backup already running;
// a failed read says nothing about that Backup.
return ctrl.Result{}, fmt.Errorf("read cnpg.io/Cluster %s/%s: %w", j.Namespace, clusterName, err)
}
message := fmt.Sprintf("failed to attach barman-cloud plugin to Cluster: %v", err)
if apierrors.IsInvalid(err) || apierrors.IsBadRequest(err) || apierrors.IsForbidden(err) {
return r.markBackupJobFailed(ctx, j, message)
}
apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{
Type: backupCondDestinationApplied,
Status: metav1.ConditionFalse,
Reason: "ClusterPluginApplyRetrying",
Message: message,
})
failingSince := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationApplied).LastTransitionTime
if cnpgBackupDeadlineExceeded(&failingSince) {
return r.markBackupJobFailed(ctx, j, message)
}
return r.waitCNPGBackupJob(ctx, j, "ClusterPluginApplyRetrying", message)
}
if writeDestination {
apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondDestinationApplied)
}
if attachment.serverName != serverName {
logger.Info("preserving the live Cluster's barman serverName over the strategy template's",
@@ -358,7 +491,7 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp
apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{
Type: backupCondPluginRolledOut,
Status: metav1.ConditionFalse,
Reason: "WaitingForPluginRollout",
Reason: cnpgReasonAttachingPlugin,
Message: message,
})
apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{
@@ -372,12 +505,54 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp
}
return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil
}
pending, err := r.cnpgPluginRolloutPending(ctx, j.Namespace, clusterName, attachment, rollout.LastTransitionTime)
if attachment.missing {
// This write attached the plugin, so its rollout starts here:
// instances recreated since the latch, or by a restore that
// re-rendered the Cluster without the plugin, run without it. A
// plugin someone else attached since the latch keeps it: the
// rollout that attach started may already be over.
message := fmt.Sprintf("attached the barman-cloud plugin to cnpg.io/Cluster %s/%s, waiting for its instances to roll", j.Namespace, clusterName)
apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondPluginRolledOut)
apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{
Type: backupCondPluginRolledOut,
Status: metav1.ConditionFalse,
Reason: "WaitingForPluginRollout",
Message: message,
LastTransitionTime: attemptedAt,
})
apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{
Type: "Ready",
Status: metav1.ConditionFalse,
Reason: "WaitingForPluginRollout",
Message: message,
})
if err := r.Status().Update(ctx, j); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil
}
// The Cluster's record covers an attach whose status write was lost.
attachedAt := rollout.LastTransitionTime
recorded, ok, err := cnpgRecordedTime(attachment.attachedAt)
if err != nil {
return r.markBackupJobFailed(ctx, j, fmt.Sprintf(
"cannot tell when the barman-cloud plugin was attached: the %s annotation of cnpg.io/Cluster %s/%s: %v",
cnpgPluginAttachedAtAnnotation, j.Namespace, clusterName, err))
}
if ok && recorded.After(attachedAt.Time) {
attachedAt = metav1.NewTime(recorded)
}
pending, err := r.cnpgPluginRolloutPending(ctx, j.Namespace, clusterName, attachment, attachedAt)
if err != nil {
return ctrl.Result{}, err
}
if pending != "" {
if cnpgBackupDeadlineExceeded(j.Status.StartedAt) {
deadlineFrom := attachedAt
if held := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationWriteHeld); held != nil &&
held.LastTransitionTime.After(deadlineFrom.Time) {
deadlineFrom = held.LastTransitionTime
}
if cnpgBackupDeadlineExceeded(&deadlineFrom) {
return r.markBackupJobFailed(ctx, j, fmt.Sprintf(
"barman-cloud plugin rollout did not finish within %s: %s", cnpgDefaultBackupDeadline, pending))
}
@@ -401,6 +576,24 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp
})
apimeta.RemoveStatusCondition(&j.Status.Conditions, "Ready")
rolledOut = true
} else if existingBackup == nil && attachment.attachedAt != "" {
// Another backup job attached the plugin, and its rollout can still
// be running while the attach is recent; past the rollout deadline
// this job stops waiting. The record is that job's: one this job
// cannot use does not fail it.
recorded, _, err := cnpgRecordedTime(attachment.attachedAt)
if err != nil {
logger.Info("ignoring an unusable attach record another backup job left",
"cluster", clusterName, "annotation", cnpgPluginAttachedAtAnnotation, "error", err)
} else if time.Since(recorded) < cnpgDefaultBackupDeadline {
pending, err := r.cnpgPluginRolloutPending(ctx, j.Namespace, clusterName, attachment, metav1.NewTime(recorded))
if err != nil {
return ctrl.Result{}, err
}
if pending != "" {
return r.waitCNPGBackupJob(ctx, j, "WaitingForPluginRollout", pending)
}
}
}
// The chart refuses a flavor change on a live Cluster, but the app keeps
@@ -411,7 +604,15 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp
"cluster", clusterName, "live", attachment.flavor, "application", specFlavor)
}
cnpgBackup, err := r.ensureCNPGBackup(ctx, j, clusterName)
if changedAt := attachment.destinationChangedAt; !changedAt.IsZero() && time.Since(changedAt) < cnpgObjectStoreCacheSettle {
return r.waitCNPGBackupJob(ctx, j, "WaitingForDestinationToSettle", fmt.Sprintf(
"the WAL archive destination of cnpg.io/Cluster %s/%s changed at %s; the barman-cloud plugin can read the previous one for up to %s",
j.Namespace, clusterName, changedAt.UTC().Format(time.RFC3339), cnpgObjectStoreCacheSettle))
}
written := cnpgDestinationOf(rendered)
written.ServerName = serverName
cnpgBackup, err := r.ensureCNPGBackup(ctx, j, clusterName, existingBackup, written)
if err != nil {
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to ensure cnpg.io/Backup: %v", err))
}
@@ -431,7 +632,16 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp
if j.Status.BackupRef != nil {
return ctrl.Result{}, nil
}
artifact, err := r.createCNPGBackupArtifact(ctx, j, resolved, cnpgBackup, clusterName, serverName, attachment.flavor, rendered, app)
destination, err := cnpgArtifactDestination(cnpgBackup, rendered)
if err != nil {
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("refusing to create the Backup artifact: %v", err))
}
// A restore can re-render the Cluster under another serverName
// between the backup and this pass.
if destination.ServerName != "" {
serverName = destination.ServerName
}
artifact, err := r.createCNPGBackupArtifact(ctx, j, resolved, cnpgBackup, clusterName, serverName, attachment.flavor, destination, rendered, app)
if err != nil {
return r.failBackupArtifact(ctx, j, err)
}
@@ -486,16 +696,15 @@ func cnpgBackupFailedMessage(b *cnpgtypes.Backup, errText string) string {
return message
}
// cnpgBackupDeadlineExceeded reports whether enough wall-clock time has
// elapsed since the BackupJob started that we should stop waiting for the
// barman-cloud plugin rollout and fail the run. Returns false when
// StartedAt is nil so the very first reconcile (which sets StartedAt) does
// not trip the gate.
func cnpgBackupDeadlineExceeded(startedAt *metav1.Time) bool {
if startedAt == nil {
// cnpgBackupDeadlineExceeded reports whether more than the backup deadline
// has passed since the BackupJob began the step it is waiting on: the plugin
// rollout or retrying the destination write (see cnpgDefaultBackupDeadline).
// Returns false for a nil time.
func cnpgBackupDeadlineExceeded(since *metav1.Time) bool {
if since == nil {
return false
}
return time.Since(startedAt.Time) > cnpgDefaultBackupDeadline
return time.Since(since.Time) > cnpgDefaultBackupDeadline
}
// cnpgPurgeNeeded decides whether the controller should delete the live
@@ -601,7 +810,10 @@ var cnpgClusterGVR = schema.GroupVersionResource{
// With attach unset, a Cluster that has no barman-cloud plugin yet is left
// untouched and only reported missing, so the caller can record that it is
// about to roll the Cluster before it does.
func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, namespace, clusterName string, t *strategyv1alpha1.CNPGTemplate, serverName string, attach bool) (cnpgPluginAttachment, error) {
//
// With write unset nothing is written: only what the live Cluster carries is
// reported.
func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, namespace, clusterName string, t *strategyv1alpha1.CNPGTemplate, serverName string, attach, write bool, jobName string) (cnpgPluginAttachment, error) {
// Live read: the cached Cluster can predate a restore re-render, and the
// serverName, the flavor and the UID below must come from the current
// object.
@@ -618,6 +830,7 @@ func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, name
flavor: cnpgImageFlavor(existing.Spec.ImageName),
clusterStatus: existing.Status,
}
attachment.attachedAt = existing.Annotations[cnpgPluginAttachedAtAnnotation]
if live := currentBarmanServerName(existing); live != "" {
attachment.serverName = live
} else {
@@ -626,6 +839,9 @@ func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, name
return attachment, nil
}
}
if !write {
return attachment, nil
}
serverName = attachment.serverName
// The ObjectStore is named after the Cluster (distinct kind, same
@@ -653,18 +869,116 @@ func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, name
RetentionPolicy: t.BarmanObjectStore.RetentionPolicy,
InstanceSidecarConfiguration: barmanSidecarConfiguration(),
}
changedAt, err := r.objectStoreChangeRecord(ctx, objStore)
if err != nil {
return cnpgPluginAttachment{}, err
}
// Only a job without a Backup of its own writes, and the queue let it
// through, so it holds the destination from this write while it runs
// and has not created its Backup.
objStore.Annotations = map[string]string{cnpgDestinationWrittenByAnnotation: jobName}
if !changedAt.IsZero() {
objStore.Annotations[cnpgDestinationChangedAtAnnotation] = changedAt.UTC().Format(time.RFC3339)
}
if err := r.Patch(ctx, objStore, client.Apply, client.FieldOwner(cnpgFieldManager), client.ForceOwnership); err != nil {
return cnpgPluginAttachment{}, fmt.Errorf("apply ObjectStore %s/%s: %w", namespace, objStoreName, err)
}
attachment.destinationChangedAt = changedAt
patch := newCNPGClusterPatch(namespace, clusterName)
patch.Spec.Plugins = []cnpgtypes.PluginConfiguration{buildBarmanPlugin(objStoreName, serverName)}
// Re-sent unchanged on later applies: the field manager owns it, and an
// apply that left it out would delete it.
stamp := existing.Annotations[cnpgPluginAttachedAtAnnotation]
if attachment.missing {
stamp = metav1.Now().UTC().Format(time.RFC3339)
}
if stamp != "" {
patch.Annotations = map[string]string{cnpgPluginAttachedAtAnnotation: stamp}
}
if err := r.Patch(ctx, patch, client.Apply, client.FieldOwner(cnpgFieldManager), client.ForceOwnership); err != nil {
return cnpgPluginAttachment{}, err
}
return attachment, nil
}
// objectStoreChangeRecord returns the cnpgDestinationChangedAtAnnotation the
// apply of desired should carry: the time of this write when it changes the
// configuration, otherwise the record the live ObjectStore already carries,
// re-sent so the apply does not delete it, and zero when there is none.
// "Changes" is judged on the configuration a server-side dry run of the same
// apply returns, not on desired, so values the apiserver defaults or another
// field manager owns do not read as a change on every pass. Creating the
// ObjectStore stamps no time: the plugin cannot hold a stale copy of an
// object that did not exist, short of one deleted and recreated by hand
// within the cache's lifetime.
func (r *BackupJobReconciler) objectStoreChangeRecord(ctx context.Context, desired *cnpgtypes.ObjectStore) (time.Time, error) {
live := &cnpgtypes.ObjectStore{}
if err := r.apiReader().Get(ctx, client.ObjectKeyFromObject(desired), live); err != nil {
if apierrors.IsNotFound(err) {
return time.Time{}, nil
}
return time.Time{}, fmt.Errorf("read ObjectStore %s/%s: %w", desired.Namespace, desired.Name, err)
}
dry := desired.DeepCopy()
if err := r.Patch(ctx, dry, client.Apply, client.FieldOwner(cnpgFieldManager), client.ForceOwnership, client.DryRunAll); err != nil {
return time.Time{}, fmt.Errorf("dry-run apply ObjectStore %s/%s: %w", desired.Namespace, desired.Name, err)
}
if !apiequality.Semantic.DeepEqual(live.Spec.Configuration, dry.Spec.Configuration) {
return time.Now(), nil
}
recorded, _, err := cnpgRecordedTime(live.Annotations[cnpgDestinationChangedAtAnnotation])
if err != nil {
// Nothing says when the cache expires; this write's time is the
// latest it can be.
return time.Now(), nil
}
return recorded, nil
}
// destinationClaimHolder returns the BackupJob other than j that the
// ObjectStore of clusterName names as the last to write it, while that job
// is still running and has no cnpg.io/Backup of its own, or "" when there is
// none. Such a job wrote the destination before otherCNPGBackupInFlight can
// see it: it can still be waiting for the plugin cache, or for the rollout of
// the plugin it attached. Once it has a Backup, that Backup holds the
// destination while CloudNativePG runs it, and the claim stops holding, so a
// job stuck after its Backup finished does not hold every later one.
func (r *BackupJobReconciler) destinationClaimHolder(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string) (string, error) {
store := &cnpgtypes.ObjectStore{}
if err := r.apiReader().Get(ctx, client.ObjectKey{Namespace: j.Namespace, Name: clusterName}, store); err != nil {
if apierrors.IsNotFound(err) {
return "", nil
}
return "", fmt.Errorf("read ObjectStore %s/%s: %w", j.Namespace, clusterName, err)
}
holder := store.Annotations[cnpgDestinationWrittenByAnnotation]
if holder == "" || holder == j.Name {
return "", nil
}
holderJob := &backupsv1alpha1.BackupJob{}
if err := r.Get(ctx, client.ObjectKey{Namespace: j.Namespace, Name: holder}, holderJob); err != nil {
if apierrors.IsNotFound(err) {
return "", nil
}
return "", err
}
if holderJob.Status.Phase == backupsv1alpha1.BackupJobPhaseSucceeded || holderJob.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed {
return "", nil
}
holderBackups := &cnpgtypes.BackupList{}
if err := r.apiReader().List(ctx, holderBackups, client.InNamespace(j.Namespace), client.MatchingLabels{
backupsv1alpha1.OwningJobNameLabel: holder,
backupsv1alpha1.OwningJobNamespaceLabel: j.Namespace,
}); err != nil {
return "", fmt.Errorf("list cnpg.io/Backups of BackupJob %s/%s: %w", j.Namespace, holder, err)
}
if len(holderBackups.Items) > 0 {
return "", nil
}
return holder, nil
}
// cnpgPluginAttachment is what applyClusterPluginBackup read off the live
// Cluster before applying the plugin.
type cnpgPluginAttachment struct {
@@ -673,8 +987,15 @@ type cnpgPluginAttachment struct {
// flavor is the image flavor the live Cluster runs.
flavor string
// missing reports a Cluster that carried no barman-cloud plugin.
missing bool
clusterStatus cnpgtypes.ClusterStatus
missing bool
// attachedAt is the cnpgPluginAttachedAtAnnotation the Cluster carried,
// "" when it carries no record: the plugin was attached by the chart, or
// before the record existed.
attachedAt string
// destinationChangedAt is the cnpgDestinationChangedAtAnnotation this
// pass applied to the ObjectStore, zero when it wrote none.
destinationChangedAt time.Time
clusterStatus cnpgtypes.ClusterStatus
}
// cnpgPluginRolloutPending returns why the rollout that attaching the
@@ -720,6 +1041,26 @@ func (r *BackupJobReconciler) cnpgPluginRolloutPending(ctx context.Context, name
return "", nil
}
// cnpgRecordedTime parses a recorded time annotation, the plugin attach
// record on the Cluster or the destination change record on the ObjectStore;
// ok is false for "", an object that carries no record. A record further
// ahead of this controller's clock than cnpgAttachClockSkewTolerance is an
// error: every instance would predate it, and a deadline counted from it
// would not run out for as long as it lies ahead.
func cnpgRecordedTime(raw string) (recorded time.Time, ok bool, err error) {
if raw == "" {
return time.Time{}, false, nil
}
recorded, err = time.Parse(time.RFC3339, raw)
if err != nil {
return time.Time{}, false, err
}
if time.Until(recorded) > cnpgAttachClockSkewTolerance {
return time.Time{}, false, fmt.Errorf("%s is in the future", raw)
}
return recorded, true, nil
}
func podReady(pod *corev1.Pod) bool {
for _, c := range pod.Status.Conditions {
if c.Type == corev1.PodReady {
@@ -747,16 +1088,23 @@ func currentBarmanServerName(c *cnpgtypes.Cluster) string {
}
// ensureCNPGBackup creates a one-shot postgresql.cnpg.io/Backup CR labelled
// with the BackupJob, or returns the existing one if a previous reconcile
// already created it. Idempotency relies on the OwningJob labels.
func (r *BackupJobReconciler) ensureCNPGBackup(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string) (*cnpgtypes.Backup, error) {
existing, err := r.findCNPGBackupForJob(ctx, j)
if err != nil {
return nil, err
}
// with the BackupJob, or returns existing, the job's Backup as looked up at
// the start of the pass. It does not look again: a reconcile of the same
// job cannot run in between, and a failed lookup here would come after the
// destination write and fail the job.
//
// written is the destination this pass wrote to the ObjectStore. A pass
// reaches the create only after that write: a job without a Backup writes
// the destination on every pass, and a pass that wrote nothing, or wrote it
// and waits for the plugin rollout, returns before this point.
func (r *BackupJobReconciler) ensureCNPGBackup(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string, existing *cnpgtypes.Backup, written cnpgDestination) (*cnpgtypes.Backup, error) {
if existing != nil {
return existing, nil
}
record, err := json.Marshal(written)
if err != nil {
return nil, fmt.Errorf("encode the WAL archive destination: %w", err)
}
obj := &cnpgtypes.Backup{
ObjectMeta: metav1.ObjectMeta{
@@ -767,6 +1115,7 @@ func (r *BackupJobReconciler) ensureCNPGBackup(ctx context.Context, j *backupsv1
backupsv1alpha1.OwningJobNamespaceLabel: j.Namespace,
cnpgClusterLabel: clusterName,
},
Annotations: map[string]string{cnpgWrittenDestinationAnnotation: string(record)},
},
Spec: cnpgtypes.BackupSpec{
Method: cnpgtypes.BackupMethodPlugin,
@@ -784,23 +1133,167 @@ func (r *BackupJobReconciler) ensureCNPGBackup(ctx context.Context, j *backupsv1
// findCNPGBackupForJob returns the cnpg.io/Backup labelled with the
// BackupJob's OwningJob{Name,Namespace}, if any. Returns (nil, nil) when
// no match is found.
//
// A miss in the cache is checked against the apiserver: the cache can lag
// behind a Backup the previous pass created, and taking that miss as no
// Backup would write the destination under it and create a second one.
func (r *BackupJobReconciler) findCNPGBackupForJob(ctx context.Context, j *backupsv1alpha1.BackupJob) (*cnpgtypes.Backup, error) {
list := &cnpgtypes.BackupList{}
if err := r.List(ctx, list,
opts := []client.ListOption{
client.InNamespace(j.Namespace),
client.MatchingLabels{
backupsv1alpha1.OwningJobNameLabel: j.Name,
backupsv1alpha1.OwningJobNamespaceLabel: j.Namespace,
},
); err != nil {
}
list := &cnpgtypes.BackupList{}
if err := r.List(ctx, list, opts...); err != nil {
return nil, err
}
if len(list.Items) == 0 {
if err := r.apiReader().List(ctx, list, opts...); err != nil {
return nil, err
}
}
if len(list.Items) == 0 {
return nil, nil
}
return &list.Items[0], nil
}
// otherCNPGBackupInFlight returns the name of another BackupJob that is still
// running and whose cnpg.io/Backup of clusterName has not finished, or "" when
// there is none. A Backup left behind by a finished or deleted job does not
// hold the queue.
//
// The Backups are read from the apiserver, not the cache, which can still
// miss a Backup another job's earlier pass created. Only the ones this
// controller created for the Cluster are asked for, by the labels
// ensureCNPGBackup has always set, so a waiting job's poll does not fetch
// the namespace's whole Backup history.
func (r *BackupJobReconciler) otherCNPGBackupInFlight(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string) (string, error) {
list := &cnpgtypes.BackupList{}
if err := r.apiReader().List(ctx, list, client.InNamespace(j.Namespace),
client.MatchingLabels{cnpgClusterLabel: clusterName},
client.HasLabels{backupsv1alpha1.OwningJobNameLabel}); err != nil {
return "", err
}
var holders []string
for i := range list.Items {
b := &list.Items[i]
owner := b.Labels[backupsv1alpha1.OwningJobNameLabel]
if b.Spec.Cluster.Name != clusterName || owner == "" || owner == j.Name {
continue
}
if b.Status.Phase == cnpgBackupPhaseComplete || b.Status.Phase == cnpgBackupPhaseFailed {
continue
}
ownerJob := &backupsv1alpha1.BackupJob{}
if err := r.Get(ctx, client.ObjectKey{Namespace: j.Namespace, Name: owner}, ownerJob); err != nil {
if apierrors.IsNotFound(err) {
continue
}
return "", err
}
if ownerJob.Status.Phase == backupsv1alpha1.BackupJobPhaseSucceeded || ownerJob.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed {
continue
}
holders = append(holders, owner)
}
if len(holders) == 0 {
return "", nil
}
sort.Strings(holders)
return holders[0], nil
}
func cnpgDestinationWriteHeld(reason string) metav1.Condition {
return metav1.Condition{
Type: backupCondDestinationWriteHeld,
Status: metav1.ConditionTrue,
Reason: reason,
Message: "held off writing the WAL archive destination",
}
}
// waitCNPGBackupJob requeues the BackupJob and records why it waits in its
// Ready condition, writing the status only when the conditions changed.
func (r *BackupJobReconciler) waitCNPGBackupJob(ctx context.Context, j *backupsv1alpha1.BackupJob, reason, message string) (ctrl.Result, error) {
if apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{
Type: "Ready",
Status: metav1.ConditionFalse,
Reason: reason,
Message: message,
}) {
if err := r.Status().Update(ctx, j); err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil
}
// cnpgDestination is the WAL archive destination recorded in
// cnpgWrittenDestinationAnnotation. S3SecretRef is nil when the strategy
// names no credentials Secret. ServerName is empty in a record written
// before it carried one.
type cnpgDestination struct {
DestinationPath string `json:"destinationPath"`
EndpointURL string `json:"endpointURL,omitempty"`
S3SecretRef *string `json:"s3SecretRef,omitempty"`
ServerName string `json:"serverName,omitempty"`
}
func cnpgDestinationOf(t *strategyv1alpha1.CNPGTemplate) cnpgDestination {
d := cnpgDestination{
DestinationPath: t.BarmanObjectStore.DestinationPath,
EndpointURL: t.BarmanObjectStore.EndpointURL,
}
if t.BarmanObjectStore.S3Credentials != nil {
name := t.BarmanObjectStore.S3Credentials.SecretRef.Name
d.S3SecretRef = &name
}
return d
}
// cnpgArtifactDestination returns the destination the artifact of b names:
// the one recorded when b was created. A Backup without the record was
// created by a controller that kept none, and falls back to the strategy
// rendered now. A record that names no destination is an error: the
// strategy may have changed since the write, so it cannot stand in.
func cnpgArtifactDestination(b *cnpgtypes.Backup, rendered *strategyv1alpha1.CNPGTemplate) (cnpgDestination, error) {
raw, ok := b.Annotations[cnpgWrittenDestinationAnnotation]
if !ok {
return cnpgDestinationOf(rendered), nil
}
var d cnpgDestination
if err := json.Unmarshal([]byte(raw), &d); err != nil {
return cnpgDestination{}, fmt.Errorf("read the %s annotation of cnpg.io/Backup %s/%s: %w", cnpgWrittenDestinationAnnotation, b.Namespace, b.Name, err)
}
if d.DestinationPath == "" {
return cnpgDestination{}, fmt.Errorf("the %s annotation of cnpg.io/Backup %s/%s names no destination path", cnpgWrittenDestinationAnnotation, b.Namespace, b.Name)
}
return d, nil
}
// cnpgProbeStore is the object store the size probe reads: the location
// and the credentials Secret recorded for the backup, with the Secret keys
// and the endpoint CA the record does not carry taken from the strategy.
// After a strategy edit those may not fit the recorded location; the probe
// then reports no size rather than measuring somewhere else.
func cnpgProbeStore(rendered *strategyv1alpha1.BarmanObjectStoreTemplate, destination cnpgDestination) *strategyv1alpha1.BarmanObjectStoreTemplate {
store := rendered.DeepCopy()
store.DestinationPath = destination.DestinationPath
store.EndpointURL = destination.EndpointURL
if destination.S3SecretRef == nil {
store.S3Credentials = nil
} else {
if store.S3Credentials == nil {
store.S3Credentials = &strategyv1alpha1.S3CredentialsTemplate{}
}
store.S3Credentials.SecretRef.Name = *destination.S3SecretRef
}
return store
}
// createCNPGBackupArtifact materialises a Cozystack Backup resource carrying
// the metadata callers need to drive a future restore. The source app's
// spec.databases / spec.users are snapshotted into Status.UnderlyingResources
@@ -812,6 +1305,7 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact(
resolved *ResolvedBackupConfig,
cnpgBackup *cnpgtypes.Backup,
clusterName, serverName, flavor string,
destination cnpgDestination,
rendered *strategyv1alpha1.CNPGTemplate,
sourceApp *postgresapp.Postgres,
) (*backupsv1alpha1.Backup, error) {
@@ -827,12 +1321,12 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact(
driverMD := map[string]string{
cnpgBackupNameKey: cnpgBackup.Name,
cnpgServerNameKey: serverName,
cnpgDestinationPathKey: rendered.BarmanObjectStore.DestinationPath,
cnpgEndpointURLKey: rendered.BarmanObjectStore.EndpointURL,
cnpgDestinationPathKey: destination.DestinationPath,
cnpgEndpointURLKey: destination.EndpointURL,
cnpgClusterNameKey: clusterName,
}
if rendered.BarmanObjectStore.S3Credentials != nil {
driverMD[cnpgS3SecretRefKey] = rendered.BarmanObjectStore.S3Credentials.SecretRef.Name
if destination.S3SecretRef != nil {
driverMD[cnpgS3SecretRefKey] = *destination.S3SecretRef
}
// Copied rather than read back at restore time: barman retention deletes
// the cnpg.io/Backup while the artifact can still be restored from.
@@ -865,7 +1359,7 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact(
},
}
if probe {
backup.Status.Artifact.SizeBytes = r.cnpgArtifactSizeBytes(ctx, j.Namespace, &rendered.BarmanObjectStore, serverName, cnpgBackup.Status.BackupID)
backup.Status.Artifact.SizeBytes = r.cnpgArtifactSizeBytes(ctx, j.Namespace, cnpgProbeStore(&rendered.BarmanObjectStore, destination), serverName, cnpgBackup.Status.BackupID)
}
if j.Spec.PlanRef != nil {
backup.Spec.PlanRef = j.Spec.PlanRef
File diff suppressed because it is too large. Load diff