Aleksei Sviridkin 5dbfa1cfad fix(linstor): restore snapshots on a schedulable node (#5309)
## What this PR does

When linstor-csi restores a snapshot (a PVC with a `VolumeSnapshot` data
source, as the Velero data mover creates for every backup), it restores
it on exactly one node, and the new volume is pinned there. Two things
go wrong in how that node is chosen.

- The candidate filter is `slices.Contains(preferredNodes, snapNode)`,
but `preferredNodes` is the output of `SortByPreferred`, which holds
every node and only orders them. The filter therefore never filters, and
the node the scheduler selected for a `WaitForFirstConsumer` volume
(`preferredNodes[0]`) gets no priority. When the snapshot is not on that
node, the pod's `PreBind` fails with `node affinity doesn't match` and
needs another scheduling round. That happens even when the selected node
does hold the snapshot.
- `NodeAvailable` only checks that the LINSTOR satellite is `ONLINE`. A
cordoned Kubernetes node passes. A volume restored there can never be
used, because no new pod can be scheduled on that node: the pod stays
`Pending` until the node is uncordoned. On a production cluster with two
storage nodes cordoned ahead of a reboot, this failed 42 of 44 restores
that landed on those nodes, while 0 of 777 restores on the other nodes
failed. Each failure surfaced as a Velero `DataUpload` ending in
`timeout on preparing data upload`, although the snapshot also existed
on two schedulable replicas.

Patch `003-snapshot-restore-node-selection.diff` makes the controller:

- restore on the scheduler-selected node when it holds the snapshot;
- otherwise pick a random snapshot node, keeping the spreading that
`001-relocate-after-clone-restore.diff` introduced, but skip cordoned
nodes as long as another node holds the snapshot. When only cordoned
nodes hold it, behaviour is unchanged.

Cordon state is read from the Kubernetes `Node` object
(`spec.unschedulable`), through the in-cluster typed client that
`utils.KubernetesClient` already provides. The `linstor-csi-controller`
ServiceAccount already has `get` on `nodes` for the
external-provisioner. Outside Kubernetes, or when the lookup fails,
every node counts as schedulable, as before.

Tests: `TestLinstor_reconcileSnapshotResourcesSkipsCordonedNodes`
reproduces the production case through `reconcileSnapshotResources` with
mocked LINSTOR providers. Without the cordon check, 16 of 50 restores
landed on the cordoned node; with it, none do. `TestSelectRestoreNode`
covers the selected node, the cordoned selected node, only-cordoned
snapshot nodes, the preferred-node fallback and the spreading. The
Dockerfile runs `go test` in `pkg/client` as part of the image build.

### Downstream repositories

The diff only adds a patch under
`packages/system/linstor/images/linstor-csi/patches/`. I walked the
trigger map against it and found no entry that it matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(linstor): restore a snapshot on the node the scheduler selected when it holds the snapshot, and never on a cordoned node while another node holds it, so pods using a volume restored from a snapshot (such as Velero data mover backups) no longer stay Pending while a storage node is cordoned
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Snapshot restores now avoid cordoned nodes when another usable node is
available and prioritize preferred nodes when possible.
* Restore selection remains available when Kubernetes node information
is missing or lookup errors occur.
* When no preferred or schedulable node is available, restores can fall
back to any available node.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-14 00:13:49 +03:00
2025-10-08 09:43:34 +05:00
2024-02-08 12:04:32 +01:00

Cozystack Cozystack

GitHub Release Apache-2.0 License Commit activity OpenSSF Best Practices OpenSSF Scorecard CLOMonitor Support

Cozystack

Cozystack is a free and open-source platform and framework for building clouds.

Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.

With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.

Use Cozystack to build your own cloud or provide a cost-effective development environment.

Cozystack user interface

Use-Cases

Documentation

The documentation is located on the cozystack.io website.

Read the Getting Started section for a quick start.

If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.

Versioning

Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.

Contributions

Contributions are highly appreciated and very welcomed!

In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.

You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.

If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.

Community

You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.

Project documents

Document What it covers
Governance How decisions are made and who makes them
Contributor Ladder How to grow from contributor to maintainer
Maintainers Who maintains the project
Code of Conduct What we expect of each other
Security Policy Reporting a vulnerability, and which releases get fixes
Adopters Who runs Cozystack in production
Roadmap Where the project is going

License

Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.

Commercial Support

A list of companies providing commercial support for this project can be found on official site.

Languages
Go 52%
Shell 34%
TypeScript 6.8%
Go Template 4.7%
Makefile 1.3%
Other 1.1%