mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
## What this PR does When linstor-csi restores a snapshot (a PVC with a `VolumeSnapshot` data source, as the Velero data mover creates for every backup), it restores it on exactly one node, and the new volume is pinned there. Two things go wrong in how that node is chosen. - The candidate filter is `slices.Contains(preferredNodes, snapNode)`, but `preferredNodes` is the output of `SortByPreferred`, which holds every node and only orders them. The filter therefore never filters, and the node the scheduler selected for a `WaitForFirstConsumer` volume (`preferredNodes[0]`) gets no priority. When the snapshot is not on that node, the pod's `PreBind` fails with `node affinity doesn't match` and needs another scheduling round. That happens even when the selected node does hold the snapshot. - `NodeAvailable` only checks that the LINSTOR satellite is `ONLINE`. A cordoned Kubernetes node passes. A volume restored there can never be used, because no new pod can be scheduled on that node: the pod stays `Pending` until the node is uncordoned. On a production cluster with two storage nodes cordoned ahead of a reboot, this failed 42 of 44 restores that landed on those nodes, while 0 of 777 restores on the other nodes failed. Each failure surfaced as a Velero `DataUpload` ending in `timeout on preparing data upload`, although the snapshot also existed on two schedulable replicas. Patch `003-snapshot-restore-node-selection.diff` makes the controller: - restore on the scheduler-selected node when it holds the snapshot; - otherwise pick a random snapshot node, keeping the spreading that `001-relocate-after-clone-restore.diff` introduced, but skip cordoned nodes as long as another node holds the snapshot. When only cordoned nodes hold it, behaviour is unchanged. Cordon state is read from the Kubernetes `Node` object (`spec.unschedulable`), through the in-cluster typed client that `utils.KubernetesClient` already provides. The `linstor-csi-controller` ServiceAccount already has `get` on `nodes` for the external-provisioner. Outside Kubernetes, or when the lookup fails, every node counts as schedulable, as before. Tests: `TestLinstor_reconcileSnapshotResourcesSkipsCordonedNodes` reproduces the production case through `reconcileSnapshotResources` with mocked LINSTOR providers. Without the cordon check, 16 of 50 restores landed on the cordoned node; with it, none do. `TestSelectRestoreNode` covers the selected node, the cordoned selected node, only-cordoned snapshot nodes, the preferred-node fallback and the spreading. The Dockerfile runs `go test` in `pkg/client` as part of the image build. ### Downstream repositories The diff only adds a patch under `packages/system/linstor/images/linstor-csi/patches/`. I walked the trigger map against it and found no entry that it matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(linstor): restore a snapshot on the node the scheduler selected when it holds the snapshot, and never on a cordoned node while another node holds it, so pods using a volume restored from a snapshot (such as Velero data mover backups) no longer stay Pending while a storage node is cordoned ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Snapshot restores now avoid cordoned nodes when another usable node is available and prioritize preferred nodes when possible. * Restore selection remains available when Kubernetes node information is missing or lookup errors occur. * When no preferred or schedulable node is available, restores can fall back to any available node. <!-- end of auto-generated comment: release notes by coderabbit.ai -->