scooby87 a7240ba0ee fix(backups): delete a MariaDB dump from S3 when its Backup is deleted (#5233)
## What this PR does

Deleting a MariaDB `Backup` removed only the Cozystack object: the dump
stayed in the bucket and the `k8s.mariadb.com/Backup` stayed in the
namespace. Nothing else prunes such a dump on the platform bucket.
mariadb-operator's `maxRetention` runs only inside the next backup run
of the same prefix, and a `Plan` never deletes a `Backup`.

Deleting a MariaDB `Backup` now deletes its dump, then the operator
`Backup` that wrote it, and only then lets the `Backup` go. Every run of
an application writes under the same prefix, so the only key ever
deleted is one proven to belong to this `Backup`:

- a `Backup` taken with #5227 records its dump in `driverMetadata`
(`k8s.mariadb.com/dump-file`), and that key is deleted unless another
run of the same prefix, finished or not, could have written it. The size
probe that records the name lets only runs finished by then contest it,
so a run that uploaded its dump and pruned this one through
`maxRetention` before completing can have its dump recorded here; the
delete contests the recorded dump again against every run as it stands,
keeps it when any could own it, and releases the `Backup` with a
`Warning`;
- a `Backup` taken before that is matched with the same ownership-proven
attribution #5223 introduced for the size probe and #5227 uses for
legacy restores: one dump in the operator `Backup`'s lifetime on both
clocks (the timestamp in the name and `LastModified`), and in the
lifetime of no other run of the same prefix, finished or not: a run that
has not completed yet may already have uploaded its dump and pruned this
one through `maxRetention`, so its window is treated as still open. The
match is recorded on the `Backup` before the delete, so a retry deletes
the same key by name;
- when no dump is proven, the `Backup` is released with a `Warning`
event and nothing is deleted, the operator `Backup` included, since it
is what any later attribution starts from.

The delete runs inside the controller through the minio client #5223
added, the same library mariadb-operator writes with, so TLS is verified
the same way. Bucket, prefix and dump name come from the operator
`Backup`. When the operator `Backup` writes to the platform bucket,
which it names by the bucket and by the projected `cozy-backups-creds`
Secret, the endpoint and keys are the platform's current ones (read from
the Secret `cozy-backups-creds` is projected from, TLS from the scheme
of the platform endpoint, as the default strategy decides it, still
trusting a CA the strategy pinned), so a `Backup` taken before the
endpoint moved is still deleted. Any other bucket is reached as the
operator `Backup` recorded it.

Other cases:

- a restore from the `Backup` holds the deletion while it runs: a
`RestoreJob` of the `Backup` that has not finished, or an operator
`Restore` of its operator `Backup` that mariadb-operator has not marked
`Complete` (it does so once the Restore's Job has completed or failed).
The second check matters because a `RestoreJob` can fail on its deadline
while its operator `Restore` keeps reading the dump. Both are listed
from the API server rather than the informer cache, so a restore that
started just before the delete is not missed;
- a restore does not start from a `Backup` that is being deleted or is
gone: the MariaDB restore reads the `Backup` from the API server before
creating the operator `Restore` and fails the `RestoreJob` otherwise,
since mariadb-operator restores the nearest other dump once the
requested one is deleted. A failed lookup of an existing operator
`Restore` is retried rather than failing the `RestoreJob`;
- a failed dump delete keeps the `Backup` in `Terminating` and retries
every 30 seconds with a `Warning` naming the
`backups.cozystack.io/skip-artifact-cleanup` annotation, which releases
the `Backup` and deletes nothing; a failed delete of the operator
`Backup` is retried with the controller's backoff;
- a missing or malformed platform credentials Secret falls back to the
credentials the operator `Backup` names, so it never releases a `Backup`
on its own: a delete that then fails holds it as above. A failed read of
that Secret (an API error) is retried instead, because the recorded
endpoint may be the one the platform moved off;
- a recorded dump that is not in the listing is not deleted again: the
operator `Backup` is deleted and a `Normal` `ArtifactAlreadyAbsent`
event names the key and the endpoint that was listed, so a dump left on
storage the platform moved off can still be found;
- in a terminating namespace a failed delete releases the `Backup` with
a `Warning`, so it does not hold the namespace;
- an operator `Backup` that is gone, that stores on a PVC or volume, or
whose Job failed (`Complete=True` with reason `JobFailed`, which wrote
no dump) releases the `Backup` with a `Warning` and nothing is deleted,
as does a cluster without the mariadb-operator CRDs.

The controller gains the `delete` verb on `k8s.mariadb.com/backups`. No
Job, image or flag is added.

Known limitation: when a namespace is deleted with its `Backup`s, the
namespace controller may delete the operator `Backup` first, and the
`Backup` is then released with a `Warning` and its dump left behind,
because the bucket and prefix are read from the operator `Backup`.

Known limitation: an operator `Backup` that never completes, for example
one whose Job pod never starts, contests every later dump its lifetime
covers on that prefix, so those `Backup`s are released with a `Warning`
and their dumps left behind. That leaks storage rather than deleting a
dump another `Backup` may restore from. The contest reads the runs from
the operator `Backup`s that still exist, so a run whose operator
`Backup` was deleted by hand no longer contests a dump. Likewise, an
operator `Restore` that never completes holds its `Backup`'s deletion
until the skip annotation releases it. These and the namespace-teardown
limitation above are tracked in #5291.

Stacked on #5227 and targets its branch until that merges; only the
commits after #5227's head belong here.

Tests: controller tests on the fake client cover each path above,
including the window bound (`takenAt`, not the operator's `Complete`
transition) of the legacy match, ordering (the operator `Backup`
survives a failed dump delete, and a legacy `Backup` retried after a
failed operator `Backup` delete finishes by its recorded name) and the
finalizer at `Reconcile` level. The minio adapter is tested against an
HTTP server for the escaped key and an error response. A helm-unittest
pins the RBAC verbs. An unfinished neighbour run is covered both ways:
one that started inside the `Backup`'s lifetime keeps its candidate
dump, one that started after it does not contest an earlier dump. The
coordination with restores is covered from both sides: a `RestoreJob` or
operator `Restore` that has not finished, including one only the API
server has seen yet, holds the deletion, and a restore from a `Backup`
being deleted or gone is refused without creating an operator `Restore`.
A recorded dump is contested the same way, down to the full sequence of
the size probe recording a dump an unfinished run uploaded and the
`Backup` then being deleted, and a recorded dump already gone is not
contested. Every condition was mutated, each mutation turns a test red,
including the timeout that bounds an unreachable endpoint. Checked on a
test cluster against SeaweedFS with the controller function and its real
minio client: over the in-cluster `seaweedfs-s3.<ns>:8333` endpoint with
its wildcard certificate a recorded dump and a legacy dump were deleted
with their operator `Backup`s, a neighbouring dump was kept, and a
recorded dump already gone completed; with a dead recorded endpoint the
dump on the platform bucket was deleted through the platform's current
endpoint; on a foreign bucket behind a dead endpoint the `Backup` was
held with everything kept. The contest of a recorded dump and the
coordination with restores were added after that run and are covered by
the controller tests only.

### Screenshots

Not a UI change.

### Downstream repositories

Walked against the diff. It touches no CRD or API type, no app
`values.schema.json`, no `packages/core/platform/values.yaml`, no
`hack/` tooling the trigger map names, and no ApplicationDefinition.
Only `cozystack/website` is affected:
`content/en/docs/next/operations/services/backup-classes.md` describes
deletion of the artifact on `Backup` removal for Rabbitmq, Redis, Kafka
and the MongoDB system bucket but not for MariaDB.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: needed, not opened yet: document MariaDB dump deletion on
`Backup` removal and the skip annotation
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): deleting a MariaDB Backup on S3 storage now deletes its dump and its k8s.mariadb.com/Backup before the Backup goes. Only a dump proven to be the Backup's own is deleted: the one it records, unless another run could have written it, or for older Backups the one that can be attributed to its run. Otherwise the Backup is released with a Warning event and nothing is deleted. A failed delete keeps the Backup Terminating and retries; the backups.cozystack.io/skip-artifact-cleanup annotation releases it and keeps the dump. Deletion waits while a RestoreJob or an operator Restore reads the Backup's dump, and a restore from a Backup being deleted is refused.
```
2026-11-13 15:07:50 +03:00
2025-10-08 09:43:34 +05:00
2024-02-08 12:04:32 +01:00

Cozystack Cozystack

GitHub Release Apache-2.0 License Commit activity OpenSSF Best Practices OpenSSF Scorecard CLOMonitor Support

Cozystack

Cozystack is a free and open-source platform and framework for building clouds.

Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.

With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.

Use Cozystack to build your own cloud or provide a cost-effective development environment.

Cozystack user interface

Use-Cases

Documentation

The documentation is located on the cozystack.io website.

Read the Getting Started section for a quick start.

If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.

Versioning

Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.

Contributions

Contributions are highly appreciated and very welcomed!

In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.

You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.

If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.

Community

You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.

Project documents

Document What it covers
Governance How decisions are made and who makes them
Contributor Ladder How to grow from contributor to maintainer
Maintainers Who maintains the project
Code of Conduct What we expect of each other
Security Policy Reporting a vulnerability, and which releases get fixes
Adopters Who runs Cozystack in production
Roadmap Where the project is going

License

Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.

Commercial Support

A list of companies providing commercial support for this project can be found on official site.

Languages
Go 52%
Shell 34%
TypeScript 6.8%
Go Template 4.7%
Makefile 1.3%
Other 1.1%