myasnikovdaniil
18779dddfc
## What this PR does Fixes https://github.com/cozystack/cozystack/issues/4316. Continues https://github.com/cozystack/cozystack/pull/5015 by androndo on a separate branch. All ten of their commits are kept as they were (authorship, messages and order, only rebased on current `main`). MariaDB and CNPG backups stored in S3 report `Backup.status.artifact.sizeBytes`, the size of the dump mariadb-operator uploaded or the total of the barman base backup, and the field stays unset when the size cannot be attributed to that one backup. Four commits are added on top: - `test(backups)`: operator Backup fixtures carry the condition mariadb-operator writes on success, `Complete=True` with reason `JobComplete` and message `Success`. Operator sets `Complete=True` for a failed Job too (reason `JobFailed`), so a fixture without reason describes a state that never happens. https://github.com/cozystack/cozystack/pull/5207 makes the driver check that reason, the tests pass with and without it. - `refactor(backups)`: the MariaDB matcher returns the matched dump (key, file name, timestamp from the name, size, LastModified) and tells "not found" from "ambiguous", and the Secret lookup and S3 client live on a small value any reconciler can build from its own client. A MariaDB restore needs this to restore the dump of the Backup it was given and not the newest one under the prefix. `sizeBytes` does not change here. - `fix(backups)`: the probe reads bucket, prefix, endpoint, credentials and TLS from the operator Backup's own `spec.storage.s3`, not from the strategy as it renders now, so editing the strategy during a backup does not send the probe somewhere else. - `fix(backups)`: a dump is attributed only when its ownership is proven. Before, a backup pod with a clock more than a minute behind, right after a previous run, got that run's dump size (666 bytes for a 123-byte dump in a test). Now the dump has to be the only one that either clock places in the operator Backup's lifetime, both clocks have to agree, and neither may place it in the lifetime of another finished operator Backup with the same bucket and prefix. Otherwise the size stays unset. Two cases are still not covered. First, this run's own dump is missing from the listing while the operator Backup of a neighbour was deleted or has not finished, then the neighbour's dump can be taken. Second, backup pod clock and object store clock are both off the same way by more than the minute of slack, which moves a finished neighbour's dump into this run's window, and nothing the clocks record tells them apart. A run whose Job failed has no dump of its own and reaches the probe only because the driver takes `Complete=True` for success, https://github.com/cozystack/cozystack/pull/5207 fixes that. Of the open threads on the original PR, the cache reads, the dump file name layout and the default AWS endpoint test were already handled by its later commits, the clock skew one is the last commit here and the failed Job case is https://github.com/cozystack/cozystack/pull/5207. `minio-go` is new to this tree, this PR adds it as a direct dependency and `internal/backupcontroller/artifact_size.go` is its only importer. The open Bucket strategy PR https://github.com/cozystack/cozystack/pull/4235 picks the same client at the same version. Each clock skew case, and a run whose own dump is gone while the neighbour's operator Backup still exists and has finished, have a unit test that fails on the old matcher, and every new guard has a mutant that turns a test red. I also ran the matcher against a real bucket after four MariaDB backups in a row. Runs that finished more than a minute apart mapped to the dump their operator Job logged, the two back to back came back unset instead of taking each other's dump, and with a run's own dump hidden from the listing the matcher refused the neighbour's dump that the old one returned. ### Screenshots N/A, no UI changes. ### Downstream repositories The only API change is the description of the existing `Backup.status.artifact.sizeBytes` field, no new field, value, default or enum, no package, values or tooling change, and neither cozystack/website nor cozystack/terraform-provider-cozystack references `sizeBytes`. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): MariaDB and CNPG (Postgres) backups stored in S3 now report `status.artifact.sizeBytes`: the size of the dump mariadb-operator uploaded, or the total size of the barman base backup (WAL archive excluded). A MariaDB backup takes the size of the one dump that falls inside its run on both the backup pod's and the object store's clock and inside no other finished run, and the size is left unset rather than set to 0 whenever it cannot be measured or attributed. Velero-based backups intentionally keep it unset. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup artifacts now report the stored size attributable to an individual MariaDB or CNPG backup when it can be determined. Shared or ambiguous storage may leave the size unset. * **Bug Fixes** * Improved handling of backup artifact creation and lookup errors, including retrying when an existing backup cannot be read and failing jobs when artifact creation is rejected. * **Documentation** * Clarified that an unset size means unknown; zero does not indicate an unknown size. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack to build your own platform
You can install only the Cozystack components you need on an existing Kubernetes cluster
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.