Files
cozystack/packages/system
Aleksei Sviridkin 2cf80c02a4 fix(clickhouse): keep failed backup attempts out of the bucket (#5229)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse BackupJob attempt that fails during upload leaves its local
backup on the data volume, and with the upstream defaults the
clickhouse-backup sidecar never prunes local backups and resumes every
unfinished upload when it starts. On the next sidecar restart the failed
attempt is uploaded to the bucket as an archive no Backup refers to, and
while that resumed upload runs the sidecar answers new operations with
423, which burns the strategy Job's retries in seconds. The strategy
itself only waited for the sidecar API to respond and then started its
operation regardless.

The sidecar now runs with `API_COMPLETE_RESUMABLE_AFTER_RESTART=false`,
so a failed attempt is never published, and `BACKUPS_TO_KEEP_LOCAL=-1`,
so at most the newest local backup is kept and it is removed once its
upload succeeds. Backups already left on existing volumes are cleaned up
by the first backup after the upgrade. The Altinity strategy now waits
until the sidecar has no operation in progress before it starts a backup
or a restore, and fails the attempt with a clear message instead of
proceeding when the sidecar stays busy or unreachable.

Waiting for an idle sidecar changes what a retried strategy Pod does.
The Job retries a failed Pod up to twice, and a Pod can die after its
operation was accepted (a dropped poll under errexit, an eviction, a
node drain) while the sidecar keeps running it. Before, the retry hit
423 and failed; with the idle gate it would wait for the running
operation to finish and start it again. For a restore that is
destructive, because `restore_remote` drops every table it restores
before recreating it, so the second run discards whatever was written
after the first one finished. A retried restore therefore takes over a
`restore_remote` of the same archive that is still running and waits for
its result instead of starting another. Once an operation is accepted,
polls tolerate up to six failures in a row before the attempt fails, and
the sidecar losing the operation from its in-memory action log, or
ending it cancelled, fails the Pod instead of leaving it polling.

A retried backup works from the archive name the controller records for
its BackupJob. That name is now `<application>-<YYYYMMDDHHMMSS>-<first
12 hex digits of the BackupJob UID>`, so two runs of one application
started in the same second, as two Plans on one schedule are, never
share an archive. It is exactly as long as the previous
`clickhouse-<application>-<YYYYMMDDTHHMMSSZ>`, so an `s3PathOverride`
prefix that fit before still keeps every object key within the 1024-byte
limit. A retry takes over a `create_remote --resume` of its archive that
is still running, counts an upload of its archive whose latest run ended
in `success` as its own result instead of uploading the archive again,
and otherwise resumes the upload. Restoring a Backup that records no
archive still picks an archive by timestamp, and that path accepts only
`clickhouse-<application>-<timestamp>` names, so it never picks a
per-run archive.

One limit remains. A finished `restore_remote` row is not taken over:
the sidecar never trims its action log, so a `success` row of the same
archive may be left by an earlier RestoreJob, and counting it as this
one would report a restore that never ran. A retry that arrives after
the restore it replaces had already finished, but before that Pod saw
the result, therefore still runs the restore a second time.

Upgrading changes the ClickHouseInstallation pod template, so ClickHouse
pods with `backup.enabled` are rolled once. A backup Job started before
the upgrade by a controller that named archives the previous way is not
re-rendered and keeps writing that name, so the controller records the
previous name when the new one is not on the Job. The local copy of the
latest failed attempt stays on the volume until the next backup. Objects
partially written to the bucket by an upload that fails midway are not
removed by clickhouse-backup and are left for a follow-up.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): failed ClickHouse backup attempts are no longer uploaded to the bucket when the backup sidecar restarts, local backups no longer accumulate on the data volume, the backup strategy waits for the sidecar to be idle instead of failing with HTTP 423, a retried ClickHouse restore takes over the restore that is still running instead of starting it a second time, and a retried ClickHouse backup takes over or reuses its own upload instead of starting a new one. Every ClickHouse backup run now writes an archive of its own, even when two runs start in the same second, under a name no longer than before. ClickHouse pods with backups enabled are restarted once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup and restore operations wait for the backup service to become
idle and join a matching operation already in progress rather than
starting a duplicate.
* Restore operations use the archive the Backup records and fail when it
is not in remote storage; only a Backup that records no archive falls
back to the newest eligible archive by timestamp.
* New backups use distinct archive names, including for runs started in
the same second.
* Local backup copies are discarded after upload, and completed uploads
are not resumed after a service restart.
* **Bug Fixes**
* Operations now fail when the service cannot be queried, an operation
disappears from its activity log, or it ends with an error or
cancellation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 08:58:33 +03:00
..
…
…
…