## What this PR does Deleting a MariaDB `Backup` removed only the Cozystack object: the dump stayed in the bucket and the `k8s.mariadb.com/Backup` stayed in the namespace. Nothing else prunes such a dump on the platform bucket. mariadb-operator's `maxRetention` runs only inside the next backup run of the same prefix, and a `Plan` never deletes a `Backup`. Deleting a MariaDB `Backup` now deletes its dump, then the operator `Backup` that wrote it, and only then lets the `Backup` go. Every run of an application writes under the same prefix, so the only key ever deleted is one proven to belong to this `Backup`: - a `Backup` taken with #5227 records its dump in `driverMetadata` (`k8s.mariadb.com/dump-file`), and that key is deleted unless another run of the same prefix, finished or not, could have written it. The size probe that records the name lets only runs finished by then contest it, so a run that uploaded its dump and pruned this one through `maxRetention` before completing can have its dump recorded here; the delete contests the recorded dump again against every run as it stands, keeps it when any could own it, and releases the `Backup` with a `Warning`; - a `Backup` taken before that is matched with the same ownership-proven attribution #5223 introduced for the size probe and #5227 uses for legacy restores: one dump in the operator `Backup`'s lifetime on both clocks (the timestamp in the name and `LastModified`), and in the lifetime of no other run of the same prefix, finished or not: a run that has not completed yet may already have uploaded its dump and pruned this one through `maxRetention`, so its window is treated as still open. The match is recorded on the `Backup` before the delete, so a retry deletes the same key by name; - when no dump is proven, the `Backup` is released with a `Warning` event and nothing is deleted, the operator `Backup` included, since it is what any later attribution starts from. The delete runs inside the controller through the minio client #5223 added, the same library mariadb-operator writes with, so TLS is verified the same way. Bucket, prefix and dump name come from the operator `Backup`. When the operator `Backup` writes to the platform bucket, which it names by the bucket and by the projected `cozy-backups-creds` Secret, the endpoint and keys are the platform's current ones (read from the Secret `cozy-backups-creds` is projected from, TLS from the scheme of the platform endpoint, as the default strategy decides it, still trusting a CA the strategy pinned), so a `Backup` taken before the endpoint moved is still deleted. Any other bucket is reached as the operator `Backup` recorded it. Other cases: - a restore from the `Backup` holds the deletion while it runs: a `RestoreJob` of the `Backup` that has not finished, or an operator `Restore` of its operator `Backup` that mariadb-operator has not marked `Complete` (it does so once the Restore's Job has completed or failed). The second check matters because a `RestoreJob` can fail on its deadline while its operator `Restore` keeps reading the dump. Both are listed from the API server rather than the informer cache, so a restore that started just before the delete is not missed; - a restore does not start from a `Backup` that is being deleted or is gone: the MariaDB restore reads the `Backup` from the API server before creating the operator `Restore` and fails the `RestoreJob` otherwise, since mariadb-operator restores the nearest other dump once the requested one is deleted. A failed lookup of an existing operator `Restore` is retried rather than failing the `RestoreJob`; - a failed dump delete keeps the `Backup` in `Terminating` and retries every 30 seconds with a `Warning` naming the `backups.cozystack.io/skip-artifact-cleanup` annotation, which releases the `Backup` and deletes nothing; a failed delete of the operator `Backup` is retried with the controller's backoff; - a missing or malformed platform credentials Secret falls back to the credentials the operator `Backup` names, so it never releases a `Backup` on its own: a delete that then fails holds it as above. A failed read of that Secret (an API error) is retried instead, because the recorded endpoint may be the one the platform moved off; - a recorded dump that is not in the listing is not deleted again: the operator `Backup` is deleted and a `Normal` `ArtifactAlreadyAbsent` event names the key and the endpoint that was listed, so a dump left on storage the platform moved off can still be found; - in a terminating namespace a failed delete releases the `Backup` with a `Warning`, so it does not hold the namespace; - an operator `Backup` that is gone, that stores on a PVC or volume, or whose Job failed (`Complete=True` with reason `JobFailed`, which wrote no dump) releases the `Backup` with a `Warning` and nothing is deleted, as does a cluster without the mariadb-operator CRDs. The controller gains the `delete` verb on `k8s.mariadb.com/backups`. No Job, image or flag is added. Known limitation: when a namespace is deleted with its `Backup`s, the namespace controller may delete the operator `Backup` first, and the `Backup` is then released with a `Warning` and its dump left behind, because the bucket and prefix are read from the operator `Backup`. Known limitation: an operator `Backup` that never completes, for example one whose Job pod never starts, contests every later dump its lifetime covers on that prefix, so those `Backup`s are released with a `Warning` and their dumps left behind. That leaks storage rather than deleting a dump another `Backup` may restore from. The contest reads the runs from the operator `Backup`s that still exist, so a run whose operator `Backup` was deleted by hand no longer contests a dump. Likewise, an operator `Restore` that never completes holds its `Backup`'s deletion until the skip annotation releases it. These and the namespace-teardown limitation above are tracked in #5291. Stacked on #5227 and targets its branch until that merges; only the commits after #5227's head belong here. Tests: controller tests on the fake client cover each path above, including the window bound (`takenAt`, not the operator's `Complete` transition) of the legacy match, ordering (the operator `Backup` survives a failed dump delete, and a legacy `Backup` retried after a failed operator `Backup` delete finishes by its recorded name) and the finalizer at `Reconcile` level. The minio adapter is tested against an HTTP server for the escaped key and an error response. A helm-unittest pins the RBAC verbs. An unfinished neighbour run is covered both ways: one that started inside the `Backup`'s lifetime keeps its candidate dump, one that started after it does not contest an earlier dump. The coordination with restores is covered from both sides: a `RestoreJob` or operator `Restore` that has not finished, including one only the API server has seen yet, holds the deletion, and a restore from a `Backup` being deleted or gone is refused without creating an operator `Restore`. A recorded dump is contested the same way, down to the full sequence of the size probe recording a dump an unfinished run uploaded and the `Backup` then being deleted, and a recorded dump already gone is not contested. Every condition was mutated, each mutation turns a test red, including the timeout that bounds an unreachable endpoint. Checked on a test cluster against SeaweedFS with the controller function and its real minio client: over the in-cluster `seaweedfs-s3.<ns>:8333` endpoint with its wildcard certificate a recorded dump and a legacy dump were deleted with their operator `Backup`s, a neighbouring dump was kept, and a recorded dump already gone completed; with a dead recorded endpoint the dump on the platform bucket was deleted through the platform's current endpoint; on a foreign bucket behind a dead endpoint the `Backup` was held with everything kept. The contest of a recorded dump and the coordination with restores were added after that run and are covered by the controller tests only. ### Screenshots Not a UI change. ### Downstream repositories Walked against the diff. It touches no CRD or API type, no app `values.schema.json`, no `packages/core/platform/values.yaml`, no `hack/` tooling the trigger map names, and no ApplicationDefinition. Only `cozystack/website` is affected: `content/en/docs/next/operations/services/backup-classes.md` describes deletion of the artifact on `Backup` removal for Rabbitmq, Redis, Kafka and the MongoDB system bucket but not for MariaDB. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: needed, not opened yet: document MariaDB dump deletion on `Backup` removal and the skip annotation - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): deleting a MariaDB Backup on S3 storage now deletes its dump and its k8s.mariadb.com/Backup before the Backup goes. Only a dump proven to be the Backup's own is deleted: the one it records, unless another run could have written it, or for older Backups the one that can be attributed to its run. Otherwise the Backup is released with a Warning event and nothing is deleted. A failed delete keeps the Backup Terminating and retries; the backups.cozystack.io/skip-artifact-cleanup annotation releases it and keeps the dump. Deletion waits while a RestoreJob or an operator Restore reads the Backup's dump, and a restore from a Backup being deleted is refused. ```
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack to build your own platform
You can install only the Cozystack components you need on an existing Kubernetes cluster
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.
