mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
fix(backups): delete a MariaDB dump from S3 when its Backup is deleted (#5233)
## What this PR does Deleting a MariaDB `Backup` removed only the Cozystack object: the dump stayed in the bucket and the `k8s.mariadb.com/Backup` stayed in the namespace. Nothing else prunes such a dump on the platform bucket. mariadb-operator's `maxRetention` runs only inside the next backup run of the same prefix, and a `Plan` never deletes a `Backup`. Deleting a MariaDB `Backup` now deletes its dump, then the operator `Backup` that wrote it, and only then lets the `Backup` go. Every run of an application writes under the same prefix, so the only key ever deleted is one proven to belong to this `Backup`: - a `Backup` taken with #5227 records its dump in `driverMetadata` (`k8s.mariadb.com/dump-file`), and that key is deleted unless another run of the same prefix, finished or not, could have written it. The size probe that records the name lets only runs finished by then contest it, so a run that uploaded its dump and pruned this one through `maxRetention` before completing can have its dump recorded here; the delete contests the recorded dump again against every run as it stands, keeps it when any could own it, and releases the `Backup` with a `Warning`; - a `Backup` taken before that is matched with the same ownership-proven attribution #5223 introduced for the size probe and #5227 uses for legacy restores: one dump in the operator `Backup`'s lifetime on both clocks (the timestamp in the name and `LastModified`), and in the lifetime of no other run of the same prefix, finished or not: a run that has not completed yet may already have uploaded its dump and pruned this one through `maxRetention`, so its window is treated as still open. The match is recorded on the `Backup` before the delete, so a retry deletes the same key by name; - when no dump is proven, the `Backup` is released with a `Warning` event and nothing is deleted, the operator `Backup` included, since it is what any later attribution starts from. The delete runs inside the controller through the minio client #5223 added, the same library mariadb-operator writes with, so TLS is verified the same way. Bucket, prefix and dump name come from the operator `Backup`. When the operator `Backup` writes to the platform bucket, which it names by the bucket and by the projected `cozy-backups-creds` Secret, the endpoint and keys are the platform's current ones (read from the Secret `cozy-backups-creds` is projected from, TLS from the scheme of the platform endpoint, as the default strategy decides it, still trusting a CA the strategy pinned), so a `Backup` taken before the endpoint moved is still deleted. Any other bucket is reached as the operator `Backup` recorded it. Other cases: - a restore from the `Backup` holds the deletion while it runs: a `RestoreJob` of the `Backup` that has not finished, or an operator `Restore` of its operator `Backup` that mariadb-operator has not marked `Complete` (it does so once the Restore's Job has completed or failed). The second check matters because a `RestoreJob` can fail on its deadline while its operator `Restore` keeps reading the dump. Both are listed from the API server rather than the informer cache, so a restore that started just before the delete is not missed; - a restore does not start from a `Backup` that is being deleted or is gone: the MariaDB restore reads the `Backup` from the API server before creating the operator `Restore` and fails the `RestoreJob` otherwise, since mariadb-operator restores the nearest other dump once the requested one is deleted. A failed lookup of an existing operator `Restore` is retried rather than failing the `RestoreJob`; - a failed dump delete keeps the `Backup` in `Terminating` and retries every 30 seconds with a `Warning` naming the `backups.cozystack.io/skip-artifact-cleanup` annotation, which releases the `Backup` and deletes nothing; a failed delete of the operator `Backup` is retried with the controller's backoff; - a missing or malformed platform credentials Secret falls back to the credentials the operator `Backup` names, so it never releases a `Backup` on its own: a delete that then fails holds it as above. A failed read of that Secret (an API error) is retried instead, because the recorded endpoint may be the one the platform moved off; - a recorded dump that is not in the listing is not deleted again: the operator `Backup` is deleted and a `Normal` `ArtifactAlreadyAbsent` event names the key and the endpoint that was listed, so a dump left on storage the platform moved off can still be found; - in a terminating namespace a failed delete releases the `Backup` with a `Warning`, so it does not hold the namespace; - an operator `Backup` that is gone, that stores on a PVC or volume, or whose Job failed (`Complete=True` with reason `JobFailed`, which wrote no dump) releases the `Backup` with a `Warning` and nothing is deleted, as does a cluster without the mariadb-operator CRDs. The controller gains the `delete` verb on `k8s.mariadb.com/backups`. No Job, image or flag is added. Known limitation: when a namespace is deleted with its `Backup`s, the namespace controller may delete the operator `Backup` first, and the `Backup` is then released with a `Warning` and its dump left behind, because the bucket and prefix are read from the operator `Backup`. Known limitation: an operator `Backup` that never completes, for example one whose Job pod never starts, contests every later dump its lifetime covers on that prefix, so those `Backup`s are released with a `Warning` and their dumps left behind. That leaks storage rather than deleting a dump another `Backup` may restore from. The contest reads the runs from the operator `Backup`s that still exist, so a run whose operator `Backup` was deleted by hand no longer contests a dump. Likewise, an operator `Restore` that never completes holds its `Backup`'s deletion until the skip annotation releases it. These and the namespace-teardown limitation above are tracked in #5291. Stacked on #5227 and targets its branch until that merges; only the commits after #5227's head belong here. Tests: controller tests on the fake client cover each path above, including the window bound (`takenAt`, not the operator's `Complete` transition) of the legacy match, ordering (the operator `Backup` survives a failed dump delete, and a legacy `Backup` retried after a failed operator `Backup` delete finishes by its recorded name) and the finalizer at `Reconcile` level. The minio adapter is tested against an HTTP server for the escaped key and an error response. A helm-unittest pins the RBAC verbs. An unfinished neighbour run is covered both ways: one that started inside the `Backup`'s lifetime keeps its candidate dump, one that started after it does not contest an earlier dump. The coordination with restores is covered from both sides: a `RestoreJob` or operator `Restore` that has not finished, including one only the API server has seen yet, holds the deletion, and a restore from a `Backup` being deleted or gone is refused without creating an operator `Restore`. A recorded dump is contested the same way, down to the full sequence of the size probe recording a dump an unfinished run uploaded and the `Backup` then being deleted, and a recorded dump already gone is not contested. Every condition was mutated, each mutation turns a test red, including the timeout that bounds an unreachable endpoint. Checked on a test cluster against SeaweedFS with the controller function and its real minio client: over the in-cluster `seaweedfs-s3.<ns>:8333` endpoint with its wildcard certificate a recorded dump and a legacy dump were deleted with their operator `Backup`s, a neighbouring dump was kept, and a recorded dump already gone completed; with a dead recorded endpoint the dump on the platform bucket was deleted through the platform's current endpoint; on a foreign bucket behind a dead endpoint the `Backup` was held with everything kept. The contest of a recorded dump and the coordination with restores were added after that run and are covered by the controller tests only. ### Screenshots Not a UI change. ### Downstream repositories Walked against the diff. It touches no CRD or API type, no app `values.schema.json`, no `packages/core/platform/values.yaml`, no `hack/` tooling the trigger map names, and no ApplicationDefinition. Only `cozystack/website` is affected: `content/en/docs/next/operations/services/backup-classes.md` describes deletion of the artifact on `Backup` removal for Rabbitmq, Redis, Kafka and the MongoDB system bucket but not for MariaDB. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: needed, not opened yet: document MariaDB dump deletion on `Backup` removal and the skip annotation - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): deleting a MariaDB Backup on S3 storage now deletes its dump and its k8s.mariadb.com/Backup before the Backup goes. Only a dump proven to be the Backup's own is deleted: the one it records, unless another run could have written it, or for older Backups the one that can be attributed to its run. Otherwise the Backup is released with a Warning event and nothing is deleted. A failed delete keeps the Backup Terminating and retries; the backups.cozystack.io/skip-artifact-cleanup annotation releases it and keeps the dump. Deletion waits while a RestoreJob or an operator Restore reads the Backup's dump, and a restore from a Backup being deleted is refused. ```
This commit is contained in:
10 files changed
+2002
-41
No files matched your search
@@ -38,6 +38,13 @@ type s3ObjectLister interface {
|
||||
ListObjects(ctx context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error)
|
||||
}
|
||||
|
||||
type s3ObjectRemover interface {
|
||||
// RemoveObject deletes key. S3 answers 204 for a key that does not
|
||||
// exist and minio-go takes 204 as success; a backend answering a
|
||||
// missing key with any other status makes it an error.
|
||||
RemoveObject(ctx context.Context, bucket, key string) error
|
||||
}
|
||||
|
||||
// s3Connection is what a driver resolves from its rendered strategy to
|
||||
// read the bucket with the same identity the backup was written with.
|
||||
type s3Connection struct {
|
||||
@@ -63,8 +70,10 @@ type objectStore struct {
|
||||
// reader reads Secrets, which the manager's client fetches uncached,
|
||||
// and lists operator Backups, which it serves from the cache.
|
||||
reader client.Reader
|
||||
// newLister overrides the S3 client; nil means a real minio client.
|
||||
newLister func(conn s3Connection) (s3ObjectLister, error)
|
||||
// newLister and newRemover override the S3 client; nil means a real
|
||||
// minio client.
|
||||
newLister func(conn s3Connection) (s3ObjectLister, error)
|
||||
newRemover func(conn s3Connection) (s3ObjectRemover, error)
|
||||
}
|
||||
|
||||
func (s objectStore) lister(conn s3Connection) (s3ObjectLister, error) {
|
||||
@@ -74,6 +83,17 @@ func (s objectStore) lister(conn s3Connection) (s3ObjectLister, error) {
|
||||
return newMinioObjectLister(conn)
|
||||
}
|
||||
|
||||
func (s objectStore) remover(conn s3Connection) (s3ObjectRemover, error) {
|
||||
if s.newRemover != nil {
|
||||
return s.newRemover(conn)
|
||||
}
|
||||
c, err := newMinioObjectClient(conn)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
func (s objectStore) secretValue(ctx context.Context, namespace, name, key string) (string, error) {
|
||||
secret := &corev1.Secret{}
|
||||
if err := s.reader.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, secret); err != nil {
|
||||
@@ -94,6 +114,10 @@ func (r *RestoreJobReconciler) objectStore() objectStore {
|
||||
return objectStore{reader: r.Client, newLister: r.newS3ObjectLister}
|
||||
}
|
||||
|
||||
func (r *BackupReconciler) objectStore() objectStore {
|
||||
return objectStore{reader: r.Client, newLister: r.newS3ObjectLister, newRemover: r.newS3ObjectRemover}
|
||||
}
|
||||
|
||||
func (r *BackupJobReconciler) s3Lister(conn s3Connection) (s3ObjectLister, error) {
|
||||
return r.objectStore().lister(conn)
|
||||
}
|
||||
@@ -103,6 +127,14 @@ type minioObjectLister struct {
|
||||
}
|
||||
|
||||
func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) {
|
||||
c, err := newMinioObjectClient(conn)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
func newMinioObjectClient(conn s3Connection) (*minioObjectLister, error) {
|
||||
opts := &minio.Options{
|
||||
Creds: credentials.NewStaticV4(conn.AccessKey, conn.SecretKey, conn.SessionToken),
|
||||
Region: conn.Region,
|
||||
@@ -152,6 +184,10 @@ func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix stri
|
||||
return collectObjects(ctx, l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: recursive}))
|
||||
}
|
||||
|
||||
func (l *minioObjectLister) RemoveObject(ctx context.Context, bucket, key string) error {
|
||||
return l.client.RemoveObject(ctx, bucket, key, minio.RemoveObjectOptions{})
|
||||
}
|
||||
|
||||
// collectObjects drains a minio listing. minio-go ends a listing whose
|
||||
// context is cancelled between pages by closing the channel without an
|
||||
// error item, so a cancelled context is checked once the channel closes:
|
||||
|
||||
@@ -39,9 +39,22 @@ type BackupReconciler struct {
|
||||
client.Client
|
||||
Scheme *runtime.Scheme
|
||||
Recorder record.EventRecorder
|
||||
// CredentialsConfig lets the Rabbitmq cleanup path project cozy-backups-creds
|
||||
// into the Backup's namespace before spawning its best-effort delete Job.
|
||||
// CredentialsConfig locates the platform backup credentials: the Job-based
|
||||
// cleanups project them into the Backup's namespace, and the MariaDB
|
||||
// cleanup reads them to reach the platform bucket.
|
||||
CredentialsConfig BackupCredentialsConfig
|
||||
// APIReader is the manager's uncached reader. The MariaDB cleanup lists
|
||||
// the RestoreJobs and operator Restores of a Backup, and the operator
|
||||
// Backups that may own its dump, through it: a restore that started
|
||||
// before the delete, or a run that has already uploaded its dump, may
|
||||
// not be in the cache yet. Wired in SetupWithManager.
|
||||
APIReader client.Reader
|
||||
|
||||
// newS3ObjectLister and newS3ObjectRemover override the S3 client the
|
||||
// MariaDB cleanup deletes a dump with; nil means a real minio client.
|
||||
// Set by unit tests.
|
||||
newS3ObjectLister func(conn s3Connection) (s3ObjectLister, error)
|
||||
newS3ObjectRemover func(conn s3Connection) (s3ObjectRemover, error)
|
||||
}
|
||||
|
||||
func (r *BackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
|
||||
@@ -126,22 +139,20 @@ func (r *BackupReconciler) cleanupOnDelete(ctx context.Context, backup *backupsv
|
||||
// orchestration" for tenant-facing guidance.
|
||||
return ctrl.Result{}, nil
|
||||
case strategyv1alpha1.MariaDBStrategyKind:
|
||||
// Cozystack Backup deletion does NOT delete the operator-side
|
||||
// k8s.mariadb.com/Backup CR or the backing S3/PVC archive.
|
||||
// Lifecycle of that CR is owned by mariadb-operator's
|
||||
// MaxRetention setting and by tenants / explicit teardown flows;
|
||||
// the controller's RBAC has no delete verb on
|
||||
// k8s.mariadb.com/backups for that reason. Surfacing this as an
|
||||
// explicit branch (rather than falling through to the Velero
|
||||
// default) keeps the contract honest: the MariaDB driver does
|
||||
// not own the archive, so it does not delete it on CR removal.
|
||||
// Mirrors the Altinity branch's "we do not own S3" stance.
|
||||
return ctrl.Result{}, nil
|
||||
// mariadb-operator's maxRetention prunes only from inside the next
|
||||
// backup run of the same prefix and a Plan never deletes a Backup, so
|
||||
// a dump on the platform bucket is deleted here or never. A dump in
|
||||
// a tenant's own bucket is deleted too: the Backup proves it owns
|
||||
// it, and tenants cannot delete Backups, so the delete comes from an
|
||||
// administrator or from namespace teardown. Teardown can remove the
|
||||
// operator Backup first, and the dump is then left behind with a
|
||||
// Warning, since its storage is read from it (see cleanupMariaDBBackup).
|
||||
return r.cleanupMariaDBBackup(ctx, backup)
|
||||
case strategyv1alpha1.MongoDBStrategyKind:
|
||||
// Split by flow (see cleanupMongoDBBackup). A legacy backup writes to the
|
||||
// tenant's own bucket and its lifecycle stays the tenant's / psmdb
|
||||
// operator's — that branch is a no-op, the same "we do not own the
|
||||
// archive" contract as Altinity / MariaDB / FDB. A useSystemBucket backup
|
||||
// archive" contract as Altinity / FDB. A useSystemBucket backup
|
||||
// lands in the shared cozy-backups bucket with no psmdb task retention to
|
||||
// prune it, so the driver OWNS it: this deletes the operator
|
||||
// PerconaServerMongoDBBackup CR and waits for its delete-backup finalizer
|
||||
@@ -154,7 +165,7 @@ func (r *BackupReconciler) cleanupOnDelete(ctx context.Context, backup *backupsv
|
||||
// is governed by the next BackupJob's stopOtherFoundationDBBackups
|
||||
// call, by manual operator action, or by
|
||||
// examples/backups/foundationdb/cleanup.sh. Same "we do not own
|
||||
// the archive" contract as Altinity / MariaDB — the explicit
|
||||
// the archive" contract as Altinity — the explicit
|
||||
// branch guards the seam against a future driver refactor that
|
||||
// might incidentally stamp velero.io/backup-name onto FDB
|
||||
// driverMetadata via a shared helper.
|
||||
@@ -291,7 +302,18 @@ func (r *BackupReconciler) cleanupCNPGBackup(ctx context.Context, backup *backup
|
||||
|
||||
// SetupWithManager registers the BackupReconciler with the Manager.
|
||||
func (r *BackupReconciler) SetupWithManager(mgr ctrl.Manager) error {
|
||||
r.APIReader = mgr.GetAPIReader()
|
||||
return ctrl.NewControllerManagedBy(mgr).
|
||||
For(&backupsv1alpha1.Backup{}).
|
||||
Complete(r)
|
||||
}
|
||||
|
||||
// apiReader returns the uncached reader. Reconcilers built by hand in unit
|
||||
// tests have none and fall back to their fake Client, which has no cache to
|
||||
// bypass.
|
||||
func (r *BackupReconciler) apiReader() client.Reader {
|
||||
if r.APIReader != nil {
|
||||
return r.APIReader
|
||||
}
|
||||
return r.Client
|
||||
}
|
||||
@@ -176,11 +176,8 @@ func TestBackupCleanup_Altinity_DoesNotFallThroughToVelero(t *testing.T) {
|
||||
|
||||
// TestBackupCleanup_MariaDB_DoesNotFallThroughToVelero locks in the same
|
||||
// invariant as the Altinity test above for the MariaDB strategy. The
|
||||
// dispatcher must explicitly route MariaDB-strategy Backups to a no-op
|
||||
// cleanup rather than fall through to the Velero default - the MariaDB
|
||||
// driver does not own the operator-side k8s.mariadb.com/Backup CR or the
|
||||
// S3/PVC archive behind it (rbac.yaml grants no delete verb on
|
||||
// k8s.mariadb.com/backups for that reason). The test seeds the Velero
|
||||
// dispatcher must route MariaDB-strategy Backups to their own cleanup
|
||||
// rather than fall through to the Velero default. The test seeds the Velero
|
||||
// metadata keys plus a matching velero.io/Backup; with the explicit branch
|
||||
// no DeleteBackupRequest is synthesised, and the Velero Backup survives.
|
||||
// Without the branch the default falls through to cleanupVeleroBackup,
|
||||
|
||||
@@ -60,11 +60,23 @@ func (r *BackupJobReconciler) attributedMariaDBDump(ctx context.Context, mdbBack
|
||||
// from can have been edited since, and pointing at another bucket now says
|
||||
// nothing about where this dump went.
|
||||
func findMariaDBDump(ctx context.Context, store objectStore, mdbBackup *mariadbtypes.Backup, completedAt time.Time) (mariadbDump, error) {
|
||||
lister, s3, err := mariadbStorage(ctx, store, mdbBackup)
|
||||
lister, _, err := mariadbStorage(ctx, store, mdbBackup)
|
||||
if err != nil {
|
||||
return mariadbDump{}, err
|
||||
}
|
||||
others, err := finishedMariaDBRuns(ctx, store.reader, mdbBackup)
|
||||
return findMariaDBDumpWith(ctx, lister, store.reader, mdbBackup, completedAt, false)
|
||||
}
|
||||
|
||||
// findMariaDBDumpWith is findMariaDBDump reading the bucket through lister,
|
||||
// for a caller that reaches the bucket by another endpoint than the one
|
||||
// mdbBackup names. withUnfinished lets runs that have not completed contest
|
||||
// the dump too, as neighbourMariaDBRuns describes.
|
||||
func findMariaDBDumpWith(ctx context.Context, lister s3ObjectLister, reader client.Reader, mdbBackup *mariadbtypes.Backup, completedAt time.Time, withUnfinished bool) (mariadbDump, error) {
|
||||
s3 := mdbBackup.Spec.Storage.S3
|
||||
if s3 == nil {
|
||||
return mariadbDump{}, errors.New("the operator Backup does not store to S3")
|
||||
}
|
||||
others, err := neighbourMariaDBRuns(ctx, reader, mdbBackup, withUnfinished)
|
||||
if err != nil {
|
||||
return mariadbDump{}, fmt.Errorf("list operator Backups: %w", err)
|
||||
}
|
||||
@@ -125,6 +137,10 @@ type mariadbRun struct {
|
||||
lower, upper time.Time
|
||||
}
|
||||
|
||||
// mariadbUnfinishedRunEnd closes the window of a run that has not completed,
|
||||
// which can still write its dump at any time.
|
||||
var mariadbUnfinishedRunEnd = time.Date(9999, time.December, 31, 0, 0, 0, 0, time.UTC)
|
||||
|
||||
func newMariaDBRun(b *mariadbtypes.Backup, created, completed time.Time) mariadbRun {
|
||||
return mariadbRun{
|
||||
name: b.Namespace + "/" + b.Name,
|
||||
@@ -150,8 +166,8 @@ func (r mariadbRun) placesOnEitherClock(d mariadbDump) bool {
|
||||
// dump in the window on one clock only may be this run's own, misplaced by
|
||||
// a skewed clock, which leaves any other candidate unproven. A neighbour's
|
||||
// dump is still taken if this run's own is missing from the listing, or
|
||||
// misplaced on both clocks, and the neighbour's operator Backup is gone or
|
||||
// has not completed.
|
||||
// misplaced on both clocks, and the neighbour's operator Backup is gone or,
|
||||
// unless others includes unfinished runs, has not completed.
|
||||
func matchMariaDBDump(objects []s3ObjectInfo, prefix string, own mariadbRun, others []mariadbRun) (mariadbDump, error) {
|
||||
var near []mariadbDump
|
||||
for _, obj := range objects {
|
||||
@@ -183,14 +199,17 @@ func matchMariaDBDump(objects []s3ObjectInfo, prefix string, own mariadbRun, oth
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// finishedMariaDBRuns returns the runs of the other operator Backups that
|
||||
// neighbourMariaDBRuns returns the runs of the other operator Backups that
|
||||
// write under the same bucket and prefix as b and have completed, whether
|
||||
// their Job succeeded or failed. One still running is left out: its window
|
||||
// has no end yet, and an operator Backup stuck short of Complete, as one
|
||||
// whose Job pod never starts is, would otherwise contest every later run of
|
||||
// the application for good. The endpoint is not compared, since one has
|
||||
// several spellings, and a false neighbour only leaves a size unset.
|
||||
func finishedMariaDBRuns(ctx context.Context, reader client.Reader, b *mariadbtypes.Backup) ([]mariadbRun, error) {
|
||||
// their Job succeeded or failed. One still running is left out unless
|
||||
// withUnfinished is set: its window has no end yet, and an operator Backup
|
||||
// stuck short of Complete, as one whose Job pod never starts is, would
|
||||
// otherwise contest every later run of the application for good. That is
|
||||
// the price of deleting a dump, since a run can upload its dump and prune
|
||||
// older ones through maxRetention before it completes. The endpoint is not
|
||||
// compared, since one has several spellings, and a false neighbour only
|
||||
// keeps a dump unattributed.
|
||||
func neighbourMariaDBRuns(ctx context.Context, reader client.Reader, b *mariadbtypes.Backup, withUnfinished bool) ([]mariadbRun, error) {
|
||||
list := &mariadbtypes.BackupList{}
|
||||
if err := reader.List(ctx, list); err != nil {
|
||||
return nil, err
|
||||
@@ -207,6 +226,9 @@ func finishedMariaDBRuns(ctx context.Context, reader client.Reader, b *mariadbty
|
||||
}
|
||||
cond := apimeta.FindStatusCondition(other.Status.Conditions, mariadbtypes.ConditionTypeComplete)
|
||||
if cond == nil || cond.Status != metav1.ConditionTrue {
|
||||
if withUnfinished {
|
||||
runs = append(runs, newMariaDBRun(other, other.CreationTimestamp.Time, mariadbUnfinishedRunEnd))
|
||||
}
|
||||
continue
|
||||
}
|
||||
runs = append(runs, newMariaDBRun(other, other.CreationTimestamp.Time, cond.LastTransitionTime.Time))
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"k8s.io/client-go/tools/record"
|
||||
ctrl "sigs.k8s.io/controller-runtime"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
|
||||
|
||||
backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
|
||||
"github.com/cozystack/cozystack/internal/backupcontroller/mariadbapp"
|
||||
@@ -184,6 +185,117 @@ func recordedDump(name string) map[string]string {
|
||||
return map[string]string{mariadbDumpFileKey: name, mariadbDumpTimeKey: mdbRestoreTime}
|
||||
}
|
||||
|
||||
// A Backup being deleted is deleting its dump, so a restore that starts
|
||||
// now can find it gone once the operator reads it, and the operator then
|
||||
// restores the nearest other dump. The Backup is read from the API server,
|
||||
// since the copy the reconcile was handed can predate the delete.
|
||||
func TestReconcileMariaDBRestore_RefusesABackupBeingDeleted(t *testing.T) {
|
||||
creds := s3CredsSecret("tenant", "creds")
|
||||
present := func() *fakeS3Lister {
|
||||
return &fakeS3Lister{objects: []s3ObjectInfo{{Key: mdbRestorePrefix + mdbRestoreDump, LastModified: mdbRestoreT0.Add(20 * time.Second)}}}
|
||||
}
|
||||
cases := map[string]map[string]string{"recorded dump": recordedDump(mdbRestoreDump), "legacy Backup": nil}
|
||||
for name, md := range cases {
|
||||
t.Run(name+", being deleted", func(t *testing.T) {
|
||||
f := newMariaDBRestoreFixture(t, md, present(), creds)
|
||||
ctx := context.Background()
|
||||
live := &backupsv1alpha1.Backup{}
|
||||
if err := f.c.Get(ctx, client.ObjectKeyFromObject(f.backup), live); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
live.Finalizers = []string{backupFinalizer}
|
||||
if err := f.c.Update(ctx, live); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.c.Delete(ctx, live); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.reconcile(t)
|
||||
f.wantFailed(t, "src-bk", "being deleted")
|
||||
})
|
||||
t.Run(name+", already gone", func(t *testing.T) {
|
||||
f := newMariaDBRestoreFixture(t, md, present(), creds)
|
||||
if err := f.c.Delete(context.Background(), f.backup.DeepCopy()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.reconcile(t)
|
||||
f.wantFailed(t, "src-bk", "is gone")
|
||||
})
|
||||
}
|
||||
// The refusal applies only before the operator Restore exists. One
|
||||
// created by an earlier reconcile but not yet in the cache must be
|
||||
// found and followed, or the RestoreJob fails under a running restore
|
||||
// and the cleanup deletes the dump that restore reads.
|
||||
t.Run("an operator Restore the cache has not seen yet is followed", func(t *testing.T) {
|
||||
f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), present(), creds)
|
||||
ctx := context.Background()
|
||||
live := &backupsv1alpha1.Backup{}
|
||||
if err := f.c.Get(ctx, client.ObjectKeyFromObject(f.backup), live); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
live.Finalizers = []string{backupFinalizer}
|
||||
if err := f.c.Update(ctx, live); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.c.Delete(ctx, live); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.c.Get(ctx, client.ObjectKeyFromObject(live), live); err != nil || live.DeletionTimestamp.IsZero() {
|
||||
t.Fatalf("Backup not left being deleted: %v", err)
|
||||
}
|
||||
live.ResourceVersion = ""
|
||||
apiServer := newMariaDBStrategyTestClient(t, live, &mariadbtypes.Restore{ObjectMeta: metav1.ObjectMeta{
|
||||
Namespace: "tenant", Name: "rj-abcde",
|
||||
Labels: map[string]string{
|
||||
backupsv1alpha1.OwningJobNameLabel: "rj",
|
||||
backupsv1alpha1.OwningJobNamespaceLabel: "tenant",
|
||||
},
|
||||
}})
|
||||
f.r.APIReader = apiServer
|
||||
f.reconcile(t)
|
||||
if f.rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseFailed {
|
||||
t.Fatalf("RestoreJob failed under a running restore: %s", f.rj.Status.Message)
|
||||
}
|
||||
if got := f.restores(t); len(got) != 0 {
|
||||
t.Errorf("created %d operator Restores; want the existing one followed", len(got))
|
||||
}
|
||||
})
|
||||
// A failed lookup says nothing about whether the operator Restore
|
||||
// exists, so the RestoreJob must not be failed under one that runs.
|
||||
t.Run("an operator Restore lookup that fails is retried", func(t *testing.T) {
|
||||
f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), present(), creds)
|
||||
f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{
|
||||
List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error {
|
||||
if _, ok := list.(*mariadbtypes.RestoreList); ok {
|
||||
return errors.New("etcdserver: request timed out")
|
||||
}
|
||||
return c.List(ctx, list, opts...)
|
||||
},
|
||||
})
|
||||
if _, err := f.r.reconcileMariaDBRestore(context.Background(), f.rj, f.backup); err == nil {
|
||||
t.Error("err = nil, want the failed lookup returned for a retry")
|
||||
}
|
||||
if f.rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseFailed {
|
||||
t.Fatalf("RestoreJob failed on a lookup error: %s", f.rj.Status.Message)
|
||||
}
|
||||
if got := f.restores(t); len(got) != 0 {
|
||||
t.Errorf("created %d operator Restores without knowing whether one exists", len(got))
|
||||
}
|
||||
})
|
||||
t.Run("a Backup that cannot be read keeps the RestoreJob waiting", func(t *testing.T) {
|
||||
f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), present(), creds)
|
||||
f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{
|
||||
Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error {
|
||||
if _, ok := obj.(*backupsv1alpha1.Backup); ok {
|
||||
return errors.New("apiserver unavailable")
|
||||
}
|
||||
return c.Get(ctx, key, obj, opts...)
|
||||
},
|
||||
})
|
||||
f.wantWaiting(t, f.reconcile(t), "src-bk", "apiserver unavailable")
|
||||
})
|
||||
}
|
||||
|
||||
func TestReconcileMariaDBRestore_RecordedDump(t *testing.T) {
|
||||
creds := s3CredsSecret("tenant", "creds")
|
||||
|
||||
|
||||
@@ -0,0 +1,354 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
package backupcontroller
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
apierrors "k8s.io/apimachinery/pkg/api/errors"
|
||||
apimeta "k8s.io/apimachinery/pkg/api/meta"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/apimachinery/pkg/types"
|
||||
ctrl "sigs.k8s.io/controller-runtime"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client"
|
||||
|
||||
backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
|
||||
"github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes"
|
||||
)
|
||||
|
||||
const (
|
||||
mariadbSkipArtifactCleanupAnnotation = "backups.cozystack.io/skip-artifact-cleanup"
|
||||
|
||||
// Retry cadence while a dump cannot be deleted or a restore from the
|
||||
// Backup is still running.
|
||||
mariadbCleanupRetryInterval = 30 * time.Second
|
||||
)
|
||||
|
||||
// mariadbDumpCleanupTimeout bounds the object-store round trips of one
|
||||
// cleanup attempt. The Backup controller has a single worker, so an
|
||||
// unreachable endpoint stalls every other Backup for this long. A variable
|
||||
// so tests can shorten it.
|
||||
var mariadbDumpCleanupTimeout = 10 * time.Second
|
||||
|
||||
// cleanupMariaDBBackup deletes the dump a MariaDB Backup was taken as, then
|
||||
// the operator Backup that wrote it. Every run of an application writes
|
||||
// under one prefix, so the only key deleted is one proven to be this
|
||||
// Backup's: the dump it records, unless contestedMariaDBDump finds another
|
||||
// run that could have written it, or, for a Backup taken before dumps were
|
||||
// recorded, the one findMariaDBDump attributes to its operator Backup. When
|
||||
// no dump is proven the Backup is released with a Warning and nothing is
|
||||
// deleted, the operator Backup included, since it is what any later
|
||||
// attribution would start from. A failed delete holds the Backup and
|
||||
// retries, as the Redis and Rabbitmq cleanups do.
|
||||
func (r *BackupReconciler) cleanupMariaDBBackup(ctx context.Context, backup *backupsv1alpha1.Backup) (ctrl.Result, error) {
|
||||
logger := getLogger(ctx)
|
||||
if backup.Annotations[mariadbSkipArtifactCleanupAnnotation] == "true" {
|
||||
logger.Debug("skipping MariaDB artifact cleanup per annotation", "backup", backup.Name, "annotation", mariadbSkipArtifactCleanupAnnotation)
|
||||
return ctrl.Result{}, nil
|
||||
}
|
||||
|
||||
opName := backup.Spec.DriverMetadata[mariadbBackupNameKey]
|
||||
if opName == "" {
|
||||
return r.releaseMariaDBCleanup(ctx, backup, "the dump", "the Backup records no k8s.mariadb.com/Backup"), nil
|
||||
}
|
||||
|
||||
// The operator Restore reads the dump through the operator Backup, so
|
||||
// neither may go while a restore from this Backup runs.
|
||||
restore, err := r.activeRestoreOf(ctx, backup, opName)
|
||||
if err != nil {
|
||||
return ctrl.Result{}, err
|
||||
}
|
||||
if restore != "" {
|
||||
logger.Debug("holding MariaDB Backup cleanup while a restore from it runs", "backup", backup.Name, "restore", restore)
|
||||
if r.Recorder != nil {
|
||||
r.Recorder.Eventf(backup, corev1.EventTypeNormal, "ArtifactCleanupWaiting",
|
||||
"%s is restoring from this Backup; its dump is deleted once the restore finishes. Annotate the Backup %s=true to release it now and leave the dump",
|
||||
restore, mariadbSkipArtifactCleanupAnnotation)
|
||||
}
|
||||
return ctrl.Result{RequeueAfter: mariadbCleanupRetryInterval}, nil
|
||||
}
|
||||
|
||||
op := &mariadbtypes.Backup{}
|
||||
if err := r.Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: opName}, op); err != nil {
|
||||
if apierrors.IsNotFound(err) || apimeta.IsNoMatchError(err) {
|
||||
return r.releaseMariaDBCleanup(ctx, backup, "the dump",
|
||||
fmt.Sprintf("k8s.mariadb.com/Backup %s is gone, so the storage and credentials its dump was written with cannot be read", opName)), nil
|
||||
}
|
||||
return ctrl.Result{}, err
|
||||
}
|
||||
|
||||
s3 := op.Spec.Storage.S3
|
||||
if s3 == nil {
|
||||
return r.releaseMariaDBCleanup(ctx, backup, fmt.Sprintf("the dump on the storage volume of k8s.mariadb.com/Backup %s", opName),
|
||||
"PVC and Volume storage is not reachable from the controller"), nil
|
||||
}
|
||||
prefix := mariadbOperatorS3Prefix(s3.Prefix)
|
||||
location := fmt.Sprintf("s3://%s/%s", s3.Bucket, prefix)
|
||||
|
||||
// A failed Job is Complete=True too, with reason JobFailed, and wrote no
|
||||
// dump: whatever its window holds is another run's.
|
||||
if cond := apimeta.FindStatusCondition(op.Status.Conditions, mariadbtypes.ConditionTypeComplete); cond == nil ||
|
||||
cond.Status != metav1.ConditionTrue || cond.Reason != mariadbtypes.ConditionReasonJobComplete {
|
||||
return r.releaseMariaDBCleanup(ctx, backup, location,
|
||||
fmt.Sprintf("k8s.mariadb.com/Backup %s did not complete successfully, so no dump is its own", opName)), nil
|
||||
}
|
||||
|
||||
name := backup.Spec.DriverMetadata[mariadbDumpFileKey]
|
||||
if name != "" {
|
||||
if _, ok := parseMariaDBLogicalBackupFile(name); !ok {
|
||||
return r.releaseMariaDBCleanup(ctx, backup, location,
|
||||
fmt.Sprintf("driverMetadata %s=%q is not the name of a mariadb-operator dump", mariadbDumpFileKey, name)), nil
|
||||
}
|
||||
}
|
||||
|
||||
conn, err := r.mariadbCleanupConnection(ctx, op)
|
||||
if err != nil {
|
||||
return r.retryMariaDBCleanup(ctx, backup, location, fmt.Errorf("resolve S3 credentials: %w", err))
|
||||
}
|
||||
|
||||
s3ctx, cancel := context.WithTimeout(ctx, mariadbDumpCleanupTimeout)
|
||||
defer cancel()
|
||||
store := r.objectStore()
|
||||
lister, err := store.lister(conn)
|
||||
if err != nil {
|
||||
return r.retryMariaDBCleanup(ctx, backup, location, fmt.Errorf("build S3 client: %w", err))
|
||||
}
|
||||
absent := false
|
||||
if name != "" {
|
||||
other, found, err := contestedMariaDBDump(s3ctx, lister, r.apiReader(), op, name)
|
||||
absent = !found
|
||||
if err != nil {
|
||||
return r.retryMariaDBCleanup(ctx, backup, location+name, err)
|
||||
}
|
||||
if other != "" {
|
||||
return r.releaseMariaDBCleanup(ctx, backup, location+name,
|
||||
fmt.Sprintf("the recorded dump also falls in the lifetime of k8s.mariadb.com/Backup %s, which may have written it; the operator Backup is kept", other)), nil
|
||||
}
|
||||
} else {
|
||||
dump, err := findMariaDBDumpWith(s3ctx, lister, r.apiReader(), op, backup.Spec.TakenAt.Time, true)
|
||||
if errors.Is(err, errMariaDBDumpNotFound) || errors.Is(err, errMariaDBDumpAmbiguous) {
|
||||
return r.releaseMariaDBCleanup(ctx, backup, location,
|
||||
fmt.Sprintf("the Backup records no dump and none is proven to be the one k8s.mariadb.com/Backup %s wrote (%v); the operator Backup is kept", opName, err)), nil
|
||||
}
|
||||
if err != nil {
|
||||
return r.retryMariaDBCleanup(ctx, backup, location, err)
|
||||
}
|
||||
name = dump.Name
|
||||
// Recorded first: once the dump is gone a retry could not match it
|
||||
// again, and could match a neighbour's instead.
|
||||
base := backup.DeepCopy()
|
||||
backup.Spec.DriverMetadata[mariadbDumpFileKey] = name
|
||||
backup.Spec.DriverMetadata[mariadbDumpTimeKey] = dump.Timestamp.UTC().Format(time.RFC3339)
|
||||
if err := r.Patch(ctx, backup, client.MergeFrom(base)); err != nil {
|
||||
return ctrl.Result{}, fmt.Errorf("record dump %s on Backup %s: %w", name, backup.Name, err)
|
||||
}
|
||||
}
|
||||
|
||||
remover, err := store.remover(conn)
|
||||
if err != nil {
|
||||
return r.retryMariaDBCleanup(ctx, backup, location, fmt.Errorf("build S3 client: %w", err))
|
||||
}
|
||||
if err := remover.RemoveObject(s3ctx, s3.Bucket, prefix+name); err != nil {
|
||||
return r.retryMariaDBCleanup(ctx, backup, location+name, err)
|
||||
}
|
||||
policy := metav1.DeletePropagationBackground
|
||||
if err := r.Delete(ctx, op, &client.DeleteOptions{PropagationPolicy: &policy}); err != nil && !apierrors.IsNotFound(err) {
|
||||
return ctrl.Result{}, fmt.Errorf("delete k8s.mariadb.com/Backup %s/%s: %w", op.Namespace, op.Name, err)
|
||||
}
|
||||
if absent {
|
||||
logger.Info("MariaDB dump already absent; deleted its operator Backup", "backup", backup.Name, "dump", location+name, "endpoint", conn.Endpoint, "operatorBackup", opName)
|
||||
if r.Recorder != nil {
|
||||
r.Recorder.Eventf(backup, corev1.EventTypeNormal, "ArtifactAlreadyAbsent",
|
||||
"%s was not found when listed through %s, so it was deleted earlier or never reached that storage; k8s.mariadb.com/Backup %s is deleted",
|
||||
location+name, conn.Endpoint, opName)
|
||||
}
|
||||
return ctrl.Result{}, nil
|
||||
}
|
||||
logger.Info("deleted MariaDB dump and its operator Backup", "backup", backup.Name, "dump", location+name, "operatorBackup", opName)
|
||||
return ctrl.Result{}, nil
|
||||
}
|
||||
|
||||
// contestedMariaDBDump returns the name of another operator Backup on op's
|
||||
// bucket and prefix whose run, finished or not, could have written the dump
|
||||
// name, or "" when none could, and whether the listing found the dump. The
|
||||
// size probe that recorded name let only runs finished by then contest it,
|
||||
// and a run can upload its dump and prune op's through maxRetention before
|
||||
// it completes. A dump no longer in the bucket is contested by nobody:
|
||||
// deleting it again removes nothing.
|
||||
func contestedMariaDBDump(ctx context.Context, lister s3ObjectLister, reader client.Reader, op *mariadbtypes.Backup, name string) (string, bool, error) {
|
||||
ts, ok := parseMariaDBLogicalBackupFile(name)
|
||||
if !ok {
|
||||
return "", false, fmt.Errorf("%q is not the name of a mariadb-operator dump", name)
|
||||
}
|
||||
s3 := op.Spec.Storage.S3
|
||||
key := mariadbOperatorS3Prefix(s3.Prefix) + name
|
||||
objects, err := lister.ListObjects(ctx, s3.Bucket, key, false)
|
||||
if err != nil {
|
||||
return "", false, fmt.Errorf("list s3://%s/%s: %w", s3.Bucket, key, err)
|
||||
}
|
||||
i := slices.IndexFunc(objects, func(obj s3ObjectInfo) bool { return obj.Key == key })
|
||||
if i < 0 {
|
||||
return "", false, nil
|
||||
}
|
||||
others, err := neighbourMariaDBRuns(ctx, reader, op, true)
|
||||
if err != nil {
|
||||
return "", true, fmt.Errorf("list operator Backups: %w", err)
|
||||
}
|
||||
d := mariadbDump{Key: key, Name: name, Timestamp: ts, LastModified: objects[i].LastModified}
|
||||
for _, other := range others {
|
||||
if other.placesOnEitherClock(d) {
|
||||
return other.name, true, nil
|
||||
}
|
||||
}
|
||||
return "", true, nil
|
||||
}
|
||||
|
||||
// activeRestoreOf names a restore from backup that has not finished: a
|
||||
// RestoreJob of it, or an operator Restore reading the dump of opName, the
|
||||
// operator Backup it was made of; "" when there is none. The operator
|
||||
// Restore is checked on its own because a RestoreJob can fail, on its
|
||||
// deadline, while the Restore it created keeps reading the dump.
|
||||
// mariadb-operator marks a Restore Complete=True once its Job has
|
||||
// completed or failed.
|
||||
func (r *BackupReconciler) activeRestoreOf(ctx context.Context, backup *backupsv1alpha1.Backup, opName string) (string, error) {
|
||||
list := &backupsv1alpha1.RestoreJobList{}
|
||||
if err := r.apiReader().List(ctx, list, client.InNamespace(backup.Namespace)); err != nil {
|
||||
return "", fmt.Errorf("list RestoreJobs: %w", err)
|
||||
}
|
||||
for i := range list.Items {
|
||||
rj := &list.Items[i]
|
||||
if rj.Spec.BackupRef.Name != backup.Name {
|
||||
continue
|
||||
}
|
||||
if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseSucceeded && rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed {
|
||||
return "RestoreJob " + rj.Name, nil
|
||||
}
|
||||
}
|
||||
restores := &mariadbtypes.RestoreList{}
|
||||
if err := r.apiReader().List(ctx, restores, client.InNamespace(backup.Namespace)); err != nil {
|
||||
// Without the operator's CRDs no Restore can run; the operator
|
||||
// Backup read that follows releases the Backup.
|
||||
if apimeta.IsNoMatchError(err) {
|
||||
return "", nil
|
||||
}
|
||||
return "", fmt.Errorf("list k8s.mariadb.com/Restores: %w", err)
|
||||
}
|
||||
for i := range restores.Items {
|
||||
restore := &restores.Items[i]
|
||||
if restore.Spec.BackupRef == nil || restore.Spec.BackupRef.Name != opName {
|
||||
continue
|
||||
}
|
||||
if !apimeta.IsStatusConditionTrue(restore.Status.Conditions, mariadbtypes.ConditionTypeComplete) {
|
||||
return "k8s.mariadb.com/Restore " + restore.Name, nil
|
||||
}
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// mariadbCleanupConnection returns how to reach the bucket op wrote to. A
|
||||
// dump on the platform bucket, which op names by its bucket and by the
|
||||
// Secret the platform credentials are projected to, is reached as the
|
||||
// platform reaches it now, so a Backup taken before the endpoint moved is
|
||||
// still deleted. Any other bucket, or the platform bucket while the
|
||||
// platform credentials are missing or malformed, is reached as op recorded
|
||||
// it; a delete that fails there is handled by retryMariaDBCleanup like any
|
||||
// other. A failed read of the platform credentials is retried instead: the
|
||||
// recorded endpoint may be the one the platform moved off, where the dump
|
||||
// would be found missing and reported deleted.
|
||||
func (r *BackupReconciler) mariadbCleanupConnection(ctx context.Context, op *mariadbtypes.Backup) (s3Connection, error) {
|
||||
if r.CredentialsConfig.IsEnabled() {
|
||||
conn, bucket, err := r.platformS3Connection(ctx)
|
||||
var projErr *ProjectionError
|
||||
switch {
|
||||
case errors.As(err, &projErr) && projErr.Reason == ReasonAPIError:
|
||||
return s3Connection{}, err
|
||||
case err != nil:
|
||||
getLogger(ctx).Info("cannot read the platform backup credentials; using the operator Backup's own", "operatorBackup", op.Name, "error", err)
|
||||
case bucket == op.Spec.Storage.S3.Bucket && op.Spec.Storage.S3.AccessKeyIdSecretKeyRef.Name == r.CredentialsConfig.TargetSecretName:
|
||||
// A strategy can pin a private CA for the platform bucket; the
|
||||
// platform credentials carry none.
|
||||
if tls := op.Spec.Storage.S3.TLS; conn.Secure && tls != nil && tls.CASecretKeyRef != nil {
|
||||
ca, err := r.objectStore().secretValue(ctx, op.Namespace, tls.CASecretKeyRef.Name, tls.CASecretKeyRef.Key)
|
||||
if err != nil {
|
||||
return s3Connection{}, err
|
||||
}
|
||||
conn.CAPEM = []byte(ca)
|
||||
}
|
||||
return conn, nil
|
||||
}
|
||||
}
|
||||
return mariadbS3Connection(ctx, r.objectStore(), op.Namespace, op.Spec.Storage.S3)
|
||||
}
|
||||
|
||||
// platformS3Connection reads the platform's backup bucket from the Secret
|
||||
// cozy-backups-creds is projected from, and its TLS from the scheme of
|
||||
// BACKUP_STORAGE_ENDPOINT, which is how the default strategy sets
|
||||
// tls.enabled. It reads the source rather than projecting it: projecting
|
||||
// writes a Secret, which a terminating namespace refuses.
|
||||
func (r *BackupReconciler) platformS3Connection(ctx context.Context) (s3Connection, string, error) {
|
||||
cfg := r.CredentialsConfig
|
||||
src := &corev1.Secret{}
|
||||
if err := r.Get(ctx, types.NamespacedName{Namespace: cfg.SourceNamespace, Name: cfg.SourceSecretName}, src); err != nil {
|
||||
if apierrors.IsNotFound(err) {
|
||||
return s3Connection{}, "", &ProjectionError{Reason: ReasonSourceMissing,
|
||||
Message: fmt.Sprintf("source credentials Secret %s/%s not found", cfg.SourceNamespace, cfg.SourceSecretName)}
|
||||
}
|
||||
return s3Connection{}, "", &ProjectionError{Reason: ReasonAPIError, Message: fmt.Sprintf("read source credentials Secret: %v", err)}
|
||||
}
|
||||
creds, err := parseSourceSecret(src)
|
||||
if err != nil {
|
||||
return s3Connection{}, "", err
|
||||
}
|
||||
if creds.endpoint == "" {
|
||||
creds.endpoint = stripScheme(cfg.Endpoint)
|
||||
}
|
||||
if creds.endpoint == "" {
|
||||
return s3Connection{}, "", &ProjectionError{Reason: ReasonSourceMalformed,
|
||||
Message: fmt.Sprintf("endpoint is empty in both source Secret %s/%s and BACKUP_STORAGE_ENDPOINT", cfg.SourceNamespace, cfg.SourceSecretName)}
|
||||
}
|
||||
if creds.region == "" {
|
||||
creds.region = cfg.Region
|
||||
}
|
||||
return s3Connection{
|
||||
Endpoint: creds.endpoint,
|
||||
Secure: strings.HasPrefix(cfg.Endpoint, "https://"),
|
||||
Region: creds.region,
|
||||
AccessKey: creds.accessKey,
|
||||
SecretKey: creds.secretKey,
|
||||
}, creds.bucket, nil
|
||||
}
|
||||
|
||||
// retryMariaDBCleanup holds the Backup after a failed attempt and says so,
|
||||
// unless its namespace is going away, where nothing it needs can be read
|
||||
// any longer and holding it would hold the namespace.
|
||||
func (r *BackupReconciler) retryMariaDBCleanup(ctx context.Context, backup *backupsv1alpha1.Backup, location string, cause error) (ctrl.Result, error) {
|
||||
terminating, err := r.namespaceTerminating(ctx, backup.Namespace)
|
||||
if err != nil {
|
||||
return ctrl.Result{}, err
|
||||
}
|
||||
if terminating {
|
||||
return r.releaseMariaDBCleanup(ctx, backup, location, fmt.Sprintf("namespace is terminating and the dump cannot be deleted: %v", cause)), nil
|
||||
}
|
||||
getLogger(ctx).Info("cannot delete MariaDB dump; retrying", "backup", backup.Name, "location", location, "error", cause)
|
||||
if r.Recorder != nil {
|
||||
r.Recorder.Eventf(backup, corev1.EventTypeWarning, "ArtifactCleanupFailed",
|
||||
"cannot delete the dump under %s: %v; retrying. Annotate the Backup %s=true to release it and leave the dump",
|
||||
location, cause, mariadbSkipArtifactCleanupAnnotation)
|
||||
}
|
||||
return ctrl.Result{RequeueAfter: mariadbCleanupRetryInterval}, nil
|
||||
}
|
||||
|
||||
// releaseMariaDBCleanup lets the Backup go without deleting anything further
|
||||
// and records a Warning naming what was left, so the give-up is visible.
|
||||
func (r *BackupReconciler) releaseMariaDBCleanup(ctx context.Context, backup *backupsv1alpha1.Backup, location, reason string) ctrl.Result {
|
||||
getLogger(ctx).Info("releasing Backup without deleting its dump", "backup", backup.Name, "location", location, "reason", reason)
|
||||
if r.Recorder != nil {
|
||||
r.Recorder.Eventf(backup, corev1.EventTypeWarning, "ArtifactNotDeleted", "left %s behind: %s", location, reason)
|
||||
}
|
||||
return ctrl.Result{}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -598,7 +598,9 @@ func (r *RestoreJobReconciler) reconcileMariaDBRestore(ctx context.Context, rest
|
||||
|
||||
mdbRestore, err := r.findMariaDBRestoreForJob(ctx, restoreJob)
|
||||
if err != nil {
|
||||
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("failed to ensure k8s.mariadb.com/Restore: %v", err))
|
||||
// Retried: failing the RestoreJob here could fail it under an
|
||||
// operator Restore that is running.
|
||||
return ctrl.Result{}, fmt.Errorf("find k8s.mariadb.com/Restore of RestoreJob %s/%s: %w", restoreJob.Namespace, restoreJob.Name, err)
|
||||
}
|
||||
// The dump is chosen once, as the operator Restore is created; one that
|
||||
// already exists has made its choice.
|
||||
@@ -669,6 +671,21 @@ func (e mariadbRestoreRefused) Error() string { return string(e) }
|
||||
// checked to still exist. A Backup that records no dump is resolved by
|
||||
// legacyMariaDBRestoreTargetTime.
|
||||
func (r *RestoreJobReconciler) mariadbRestoreTargetTime(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, src *mariadbtypes.Backup) (string, error) {
|
||||
// Deleting a Backup deletes its dump, and the operator restores the
|
||||
// nearest other dump once its own is gone, so a restore must not start
|
||||
// from a Backup being deleted. The copy in hand can predate the delete;
|
||||
// the cleanup holds the Backup only for restores that exist when it
|
||||
// lists them.
|
||||
live := &backupsv1alpha1.Backup{}
|
||||
if err := r.apiReader().Get(ctx, client.ObjectKeyFromObject(backup), live); err != nil {
|
||||
if apierrors.IsNotFound(err) {
|
||||
return "", mariadbRestoreRefused(fmt.Sprintf("Backup %s/%s is gone, and its dump may be gone with it", backup.Namespace, backup.Name))
|
||||
}
|
||||
return "", fmt.Errorf("read Backup %s/%s: %w", backup.Namespace, backup.Name, err)
|
||||
}
|
||||
if !live.DeletionTimestamp.IsZero() {
|
||||
return "", mariadbRestoreRefused(fmt.Sprintf("Backup %s/%s is being deleted, and its dump with it", backup.Namespace, backup.Name))
|
||||
}
|
||||
// A failed Job is Complete=True too, with reason JobFailed, and a
|
||||
// BackupJob that took Complete=True for success made a Backup of it.
|
||||
if cond := apimeta.FindStatusCondition(src.Status.Conditions, mariadbtypes.ConditionTypeComplete); cond != nil && cond.Reason == mariadbtypes.ConditionReasonJobFailed {
|
||||
@@ -742,10 +759,12 @@ func (r *RestoreJobReconciler) awaitMariaDBDumpCheck(ctx context.Context, rj *ba
|
||||
}
|
||||
|
||||
// findMariaDBRestoreForJob returns the k8s.mariadb.com/Restore labelled
|
||||
// with the RestoreJob, if a previous reconcile already created one.
|
||||
// with the RestoreJob, if a previous reconcile already created one. It is
|
||||
// read uncached: one the cache has not caught up with would be created
|
||||
// again, or, once its Backup is being deleted, refused while it runs.
|
||||
func (r *RestoreJobReconciler) findMariaDBRestoreForJob(ctx context.Context, rj *backupsv1alpha1.RestoreJob) (*mariadbtypes.Restore, error) {
|
||||
list := &mariadbtypes.RestoreList{}
|
||||
if err := r.List(ctx, list,
|
||||
if err := r.apiReader().List(ctx, list,
|
||||
client.InNamespace(rj.Namespace),
|
||||
client.MatchingLabels{
|
||||
backupsv1alpha1.OwningJobNameLabel: rj.Name,
|
||||
|
||||
@@ -146,15 +146,16 @@ rules:
|
||||
# MariaDB strategy: read the operator-side MariaDB CR to surface "not yet
|
||||
# rendered" preconditions, create one-shot k8s.mariadb.com/Backup CRs on
|
||||
# the BackupJob path, and create k8s.mariadb.com/Restore CRs on the
|
||||
# RestoreJob path. The driver only ever Creates/Lists/Gets these objects -
|
||||
# it does not patch or delete them; their lifecycle is owned by
|
||||
# mariadb-operator and (for cleanup) by tenants / explicit teardown flows,
|
||||
# never by this controller.
|
||||
# RestoreJob path. A Backup is deleted once the dump it wrote is, when its
|
||||
# Cozystack Backup is deleted; a Restore is never deleted.
|
||||
- apiGroups: ["k8s.mariadb.com"]
|
||||
resources: ["mariadbs"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: ["k8s.mariadb.com"]
|
||||
resources: ["backups", "restores"]
|
||||
resources: ["backups"]
|
||||
verbs: ["get", "list", "watch", "create", "delete"]
|
||||
- apiGroups: ["k8s.mariadb.com"]
|
||||
resources: ["restores"]
|
||||
verbs: ["get", "list", "watch", "create"]
|
||||
# MongoDB strategy: read the operator-side PerconaServerMongoDB CR to gate on
|
||||
# the backup preconditions, create one-shot PerconaServerMongoDBBackup CRs on the
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
suite: RBAC of the MariaDB dump cleanup
|
||||
|
||||
# Deleting a MariaDB Backup deletes its operator Backup once the dump is gone.
|
||||
# The Go unit tests run with no authorization, so a missing verb only shows up
|
||||
# on a cluster, where every such Backup would stay Terminating.
|
||||
|
||||
templates:
|
||||
- templates/rbac.yaml
|
||||
|
||||
release:
|
||||
name: backupstrategy-controller
|
||||
namespace: cozy-backup-controller
|
||||
|
||||
tests:
|
||||
- it: "ClusterRole can delete operator Backups but not Restores"
|
||||
asserts:
|
||||
- contains:
|
||||
path: rules
|
||||
content:
|
||||
apiGroups: ["k8s.mariadb.com"]
|
||||
resources: ["backups"]
|
||||
verbs: ["get", "list", "watch", "create", "delete"]
|
||||
- contains:
|
||||
path: rules
|
||||
content:
|
||||
apiGroups: ["k8s.mariadb.com"]
|
||||
resources: ["restores"]
|
||||
verbs: ["get", "list", "watch", "create"]
|
||||
Reference in new issue
Block a user