Aleksei Sviridkin
abc07a3927
## What this PR does Nothing on the backup path looks at restores. A Plan creates its BackupJobs on schedule (`plan_controller.go`), and `BackupJobReconciler` dispatches them to their driver whatever else is happening to the application. A backup taken during a restore captures a half-restored state at best, and for CNPG it does worse. We hit it on a v1.6.3 cluster. helm-controller applied the restored Cluster with `bootstrap.newServerName`, and four seconds later a BackupJob of the same application re-applied the barman plugin with the source's `serverName`. The restored cluster archived onto the source's WAL prefix, and its recovery job died with `WAL archive check failed for server <cluster>: Expected empty archive`. Correcting `serverName` by hand let the same restore finish, which is what identified the writer. `currentBarmanServerName` preserves the live value on most paths, but not in the window where the restored Cluster is being re-rendered. An on-demand run can be refused by whoever creates it, but a scheduled one cannot, and there is no `suspend` on a Plan. With an hourly plan and a CNPG restore allowed thirty minutes, the overlap happens about half the time. A BackupJob that has not started yet (`status.startedAt` unset) now waits while a non-terminal RestoreJob in its namespace writes into the same application. The target is resolved as the drivers resolve it (`resolveCNPGRestoreTarget` and the Job-based paths): the referenced Backup's `applicationRef`, with the target's `kind` and `name` when a `targetApplicationRef` is set, and the target's API group only when it names one, so a target without one stays in the Backup's group. A Velero restore always stays in the Backup's group, since the Velero driver drops the target's API group and restores the Backup's resources. A RestoreJob whose Backup is gone is skipped, since the restore controller fails it on its next pass. `options.targetNamespace` is honoured only when the Backup was taken by the Velero strategy, the one driver that reads it: the CNPG and Job-based drivers ignore the option and restore in place, so their restores hold whatever it says. The held job stays `Pending` with a `Ready=False` condition, reason `RestoreInProgress`, naming the restore, and is requeued every 30s. It is not failed: a scheduled run should still happen once the restore ends. A failed write of that status is returned, so the manager retries it, and once the restore ends the condition is removed before the job starts, since it would otherwise read "waiting" until the driver next writes `Ready`, which for a healthy run is when the backup completes. The gate sits after BackupClass resolution and strategy validation, so third-party strategies are unaffected, and before credentials projection, so a held job projects nothing. What happens to a delayed scheduled run: the Plan controller only decides when to *create* a BackupJob (a slot within `startingDeadline`, 300s, of now). Once created, the job is not subject to that deadline, so a held run keeps its `<plan>-<unix-minutes>` name and runs once the restore ends, later than its slot. If the restore outlasts the next slot, that slot's job is created and held too, and both run after the restore. Nothing is skipped, and nothing is deduplicated. Not covered, deliberately: - A backup already running when a restore starts is not interrupted. Its driver owns state the gate would strand. The reverse guard, a RestoreJob waiting for a running BackupJob, would belong in the restore path, together with a per-application claim that closes the race between the two checks, and is tracked in #4559. - A Velero RestoreJob in another namespace that targets this one through `options.targetNamespace` does not hold this namespace's backups. The lookup is namespaced. - The Velero restore path has no deadline, so a stuck VM restore holds that application's backups until the restore is ended or deleted. That matches the restore already blocking the application. Tests: `TestActiveRestoreTargeting` covers the resolution matrix (in place, to-copy into and out of the application, terminal phases, another kind or another API group under the same name, a to-copy target inheriting the Backup's API group, a Velero target naming another API group, a Velero restore sent to another namespace or naming its own, a Postgres restore carrying `targetNamespace`, a missing Backup with or without a target). `TestReconcile_RetriesAHoldItCouldNotRecord` and `TestReleaseRestoreHold` cover the status writes, and the held-job test reconciles again after the restore ends to check the condition is gone. `TestReconcile_HoldsABackupJobWhileItsApplicationIsRestored` is the behavioural regression: it fails with the gate removed (the job reaches projection and dispatch). `TestReconcile_DoesNotHoldARunningBackupJob` checks that a started job is not held. Backport: a candidate for `release-1.6`, where the reproduction was seen. `backupjob_controller.go` differs from `main` only in the dispatch switch and the Go 1.27 modernizers, so the cherry-pick should be clean or close to it. ### Screenshots N/A, no UI change. ### Downstream repositories The diff touches only `internal/backupcontroller/` Go code. It changes no CRD, API type, chart, values file, hack script or naming, so the trigger map comes out empty. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): a BackupJob that has not started yet, scheduled or on demand, now waits while a RestoreJob is restoring the same application, instead of running against it. It stays Pending with a Ready=False RestoreInProgress condition and runs once the restore ends. This stops a CNPG backup that has not started yet from re-pointing an in-place restored cluster at the source's WAL archive; a backup already running when a restore starts is not held. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Unstarted backup jobs now wait when an active restore targets the same application. They remain pending and are retried after a short delay, then can proceed once the restore finishes. * Backup jobs that have already started continue without being held by an active restore. Completed or failed restores do not hold pending backup jobs. * Restore matching accounts for the specified target and, for Velero restores, the target namespace. A backup job’s restore hold is cleared when no matching active restore remains. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.