<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A ClickHouse BackupJob attempt that fails during upload leaves its local backup on the data volume, and with the upstream defaults the clickhouse-backup sidecar never prunes local backups and resumes every unfinished upload when it starts. On the next sidecar restart the failed attempt is uploaded to the bucket as an archive no Backup refers to, and while that resumed upload runs the sidecar answers new operations with 423, which burns the strategy Job's retries in seconds. The strategy itself only waited for the sidecar API to respond and then started its operation regardless. The sidecar now runs with `API_COMPLETE_RESUMABLE_AFTER_RESTART=false`, so a failed attempt is never published, and `BACKUPS_TO_KEEP_LOCAL=-1`, so at most the newest local backup is kept and it is removed once its upload succeeds. Backups already left on existing volumes are cleaned up by the first backup after the upgrade. The Altinity strategy now waits until the sidecar has no operation in progress before it starts a backup or a restore, and fails the attempt with a clear message instead of proceeding when the sidecar stays busy or unreachable. Waiting for an idle sidecar changes what a retried strategy Pod does. The Job retries a failed Pod up to twice, and a Pod can die after its operation was accepted (a dropped poll under errexit, an eviction, a node drain) while the sidecar keeps running it. Before, the retry hit 423 and failed; with the idle gate it would wait for the running operation to finish and start it again. For a restore that is destructive, because `restore_remote` drops every table it restores before recreating it, so the second run discards whatever was written after the first one finished. A retried restore therefore takes over a `restore_remote` of the same archive that is still running and waits for its result instead of starting another. Once an operation is accepted, polls tolerate up to six failures in a row before the attempt fails, and the sidecar losing the operation from its in-memory action log, or ending it cancelled, fails the Pod instead of leaving it polling. A retried backup works from the archive name the controller records for its BackupJob. That name is now `<application>-<YYYYMMDDHHMMSS>-<first 12 hex digits of the BackupJob UID>`, so two runs of one application started in the same second, as two Plans on one schedule are, never share an archive. It is exactly as long as the previous `clickhouse-<application>-<YYYYMMDDTHHMMSSZ>`, so an `s3PathOverride` prefix that fit before still keeps every object key within the 1024-byte limit. A retry takes over a `create_remote --resume` of its archive that is still running, counts an upload of its archive whose latest run ended in `success` as its own result instead of uploading the archive again, and otherwise resumes the upload. Restoring a Backup that records no archive still picks an archive by timestamp, and that path accepts only `clickhouse-<application>-<timestamp>` names, so it never picks a per-run archive. One limit remains. A finished `restore_remote` row is not taken over: the sidecar never trims its action log, so a `success` row of the same archive may be left by an earlier RestoreJob, and counting it as this one would report a restore that never ran. A retry that arrives after the restore it replaces had already finished, but before that Pod saw the result, therefore still runs the restore a second time. Upgrading changes the ClickHouseInstallation pod template, so ClickHouse pods with `backup.enabled` are rolled once. A backup Job started before the upgrade by a controller that named archives the previous way is not re-rendered and keeps writing that name, so the controller records the previous name when the new one is not on the Job. The local copy of the latest failed attempt stays on the volume until the next backup. Objects partially written to the bucket by an upload that fails midway are not removed by clickhouse-backup and are left for a follow-up. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(clickhouse): failed ClickHouse backup attempts are no longer uploaded to the bucket when the backup sidecar restarts, local backups no longer accumulate on the data volume, the backup strategy waits for the sidecar to be idle instead of failing with HTTP 423, a retried ClickHouse restore takes over the restore that is still running instead of starting it a second time, and a retried ClickHouse backup takes over or reuses its own upload instead of starting a new one. Every ClickHouse backup run now writes an archive of its own, even when two runs start in the same second, under a name no longer than before. ClickHouse pods with backups enabled are restarted once on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup and restore operations wait for the backup service to become idle and join a matching operation already in progress rather than starting a duplicate. * Restore operations use the archive the Backup records and fail when it is not in remote storage; only a Backup that records no archive falls back to the newest eligible archive by timestamp. * New backups use distinct archive names, including for runs started in the same second. * Local backup copies are discarded after upload, and completed uploads are not resumed after a service restart. * **Bug Fixes** * Operations now fail when the service cannot be queried, an operation disappears from its activity log, or it ends with an error or cancellation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack to build your own platform
You can install only the Cozystack components you need on an existing Kubernetes cluster
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.
