fix(linstor): restore snapshots on a schedulable node (#5309)

## What this PR does

When linstor-csi restores a snapshot (a PVC with a `VolumeSnapshot` data
source, as the Velero data mover creates for every backup), it restores
it on exactly one node, and the new volume is pinned there. Two things
go wrong in how that node is chosen.

- The candidate filter is `slices.Contains(preferredNodes, snapNode)`,
but `preferredNodes` is the output of `SortByPreferred`, which holds
every node and only orders them. The filter therefore never filters, and
the node the scheduler selected for a `WaitForFirstConsumer` volume
(`preferredNodes[0]`) gets no priority. When the snapshot is not on that
node, the pod's `PreBind` fails with `node affinity doesn't match` and
needs another scheduling round. That happens even when the selected node
does hold the snapshot.
- `NodeAvailable` only checks that the LINSTOR satellite is `ONLINE`. A
cordoned Kubernetes node passes. A volume restored there can never be
used, because no new pod can be scheduled on that node: the pod stays
`Pending` until the node is uncordoned. On a production cluster with two
storage nodes cordoned ahead of a reboot, this failed 42 of 44 restores
that landed on those nodes, while 0 of 777 restores on the other nodes
failed. Each failure surfaced as a Velero `DataUpload` ending in
`timeout on preparing data upload`, although the snapshot also existed
on two schedulable replicas.

Patch `003-snapshot-restore-node-selection.diff` makes the controller:

- restore on the scheduler-selected node when it holds the snapshot;
- otherwise pick a random snapshot node, keeping the spreading that
`001-relocate-after-clone-restore.diff` introduced, but skip cordoned
nodes as long as another node holds the snapshot. When only cordoned
nodes hold it, behaviour is unchanged.

Cordon state is read from the Kubernetes `Node` object
(`spec.unschedulable`), through the in-cluster typed client that
`utils.KubernetesClient` already provides. The `linstor-csi-controller`
ServiceAccount already has `get` on `nodes` for the
external-provisioner. Outside Kubernetes, or when the lookup fails,
every node counts as schedulable, as before.

Tests: `TestLinstor_reconcileSnapshotResourcesSkipsCordonedNodes`
reproduces the production case through `reconcileSnapshotResources` with
mocked LINSTOR providers. Without the cordon check, 16 of 50 restores
landed on the cordoned node; with it, none do. `TestSelectRestoreNode`
covers the selected node, the cordoned selected node, only-cordoned
snapshot nodes, the preferred-node fallback and the spreading. The
Dockerfile runs `go test` in `pkg/client` as part of the image build.

### Downstream repositories

The diff only adds a patch under
`packages/system/linstor/images/linstor-csi/patches/`. I walked the
trigger map against it and found no entry that it matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(linstor): restore a snapshot on the node the scheduler selected when it holds the snapshot, and never on a cordoned node while another node holds it, so pods using a volume restored from a snapshot (such as Velero data mover backups) no longer stay Pending while a storage node is cordoned
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Snapshot restores now avoid cordoned nodes when another usable node is
available and prioritize preferred nodes when possible.
* Restore selection remains available when Kubernetes node information
is missing or lookup errors occur.
* When no preferred or schedulable node is available, restores can fall
back to any available node.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-14 00:13:49 +03:00
commit 5dbfa1cfad
1 file changed
+259
@@ -0,0 +1,259 @@
diff --git a/cmd/linstor-csi/linstor-csi.go b/cmd/linstor-csi/linstor-csi.go
index e260d7b..2062787 100644
--- a/cmd/linstor-csi/linstor-csi.go
+++ b/cmd/linstor-csi/linstor-csi.go
@@ -148,6 +148,7 @@ func main() {
client.LogOut(logOut),
client.PropertyNamespace(*propNs),
client.LabelBySP(*labelBySP),
+ client.ConfigureKubernetesIfAvailable(),
)
if err != nil {
log.Fatal(err)
diff --git a/pkg/client/linstor.go b/pkg/client/linstor.go
index 5b52dd4..82f10d0 100644
--- a/pkg/client/linstor.go
+++ b/pkg/client/linstor.go
@@ -46,6 +46,8 @@ import (
"github.com/pborman/uuid"
"github.com/sirupsen/logrus"
"golang.org/x/sys/unix"
+ apierrors "k8s.io/apimachinery/pkg/api/errors"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/mount-utils"
utilexec "k8s.io/utils/exec"
@@ -73,6 +75,9 @@ type Linstor struct {
mounter mount.Interface
resizer *mount.ResizeFs
labelBySP bool
+ // nodeSchedulable reports whether new pods can be scheduled on a node.
+ // It is nil when not running in Kubernetes: every node then counts as schedulable.
+ nodeSchedulable func(ctx context.Context, node string) (bool, error)
}
// NewLinstor returns a high-level linstor client for CSI applications to interact with
@@ -126,6 +131,34 @@ func APIClient(c *lc.HighLevelClient) func(*Linstor) error {
}
}
+// ConfigureKubernetesIfAvailable lets the client read Kubernetes Node objects, so that a snapshot is
+// not restored on a cordoned node when another node holds it. Outside of Kubernetes it does nothing.
+func ConfigureKubernetesIfAvailable() func(*Linstor) error {
+ return func(l *Linstor) error {
+ cl, _, err := utils.KubernetesClient()
+ if err != nil {
+ // Not running in kubernetes
+ return nil
+ }
+
+ l.nodeSchedulable = func(ctx context.Context, node string) (bool, error) {
+ n, err := cl.CoreV1().Nodes().Get(ctx, node, metav1.GetOptions{})
+ if err != nil {
+ if apierrors.IsNotFound(err) {
+ // A LINSTOR node without a Kubernetes Node: nothing says it is cordoned.
+ return true, nil
+ }
+
+ return true, err
+ }
+
+ return !n.Spec.Unschedulable, nil
+ }
+
+ return nil
+ }
+}
+
// LogOut sets the Linstor client to write logs to the provided io.Writer
// instead of discarding logs.
func LogOut(out io.Writer) func(*Linstor) error {
@@ -1834,34 +1867,29 @@ func (s *Linstor) reconcileSnapshotResources(ctx context.Context, snapshot *volu
return fmt.Errorf("snapshot '%s' not deployed on any node", snap.Name)
}
- // Collect available snapshot nodes, preferring those matching the topology hints.
- var preferred, available []string
+ // Collect the snapshot nodes LINSTOR can restore on, and whether Kubernetes can schedule pods there.
+ var available []string
+ schedulable := make(map[string]bool, len(snap.Nodes))
for _, snapNode := range snap.Nodes {
if err := s.NodeAvailable(ctx, snapNode); err != nil {
logger.WithField("selected node candidate", snapNode).WithError(err).Debug("node is not available")
continue
}
- if slices.Contains(preferredNodes, snapNode) {
- preferred = append(preferred, snapNode)
- }
-
available = append(available, snapNode)
- }
+ schedulable[snapNode] = true
- // Pick a random node from preferred candidates, falling back to any available node.
- // Randomization distributes restore load across snapshot nodes, preventing all
- // clones of the same source from concentrating on a single node.
- candidates := preferred
- if len(candidates) == 0 {
- candidates = available
- }
+ if s.nodeSchedulable != nil {
+ ok, err := s.nodeSchedulable(ctx, snapNode)
+ if err != nil {
+ logger.WithField("selected node candidate", snapNode).WithError(err).Warn("could not check if node is cordoned, assuming it is not")
+ }
- var selectedNode string
- if len(candidates) > 0 {
- selectedNode = candidates[rand.Intn(len(candidates))]
+ schedulable[snapNode] = ok
+ }
}
+ selectedNode := selectRestoreNode(available, preferredNodes, func(node string) bool { return schedulable[node] })
if selectedNode == "" {
return fmt.Errorf("no node [%s] available for snapshot '%s'", strings.Join(snap.Nodes, ","), snap.Name)
}
@@ -2168,6 +2196,43 @@ func (s *Linstor) relocateResources(ctx context.Context, volID, rgName string) e
return nil
}
+// selectRestoreNode picks the node a snapshot is restored on, among the available nodes holding it.
+//
+// The restored volume is pinned to that node, so the consumer pod has to run there:
+// - preferredNodes[0] is the node the scheduler selected for a WaitForFirstConsumer volume. If it holds the
+// snapshot, restoring there lets the pod start without another scheduling round.
+// - Otherwise a random node is picked, which spreads restores of the same source across its snapshot nodes.
+// Cordoned nodes are skipped as long as another node holds the snapshot: a pod can never be scheduled on
+// them, so a volume restored there stays unusable until the node is uncordoned.
+//
+// Nodes in preferredNodes come first, but a schedulable node outside them wins over a cordoned one inside them.
+func selectRestoreNode(available, preferredNodes []string, schedulable func(node string) bool) string {
+ isPreferred := func(node string) bool { return slices.Contains(preferredNodes, node) }
+ both := func(node string) bool { return isPreferred(node) && schedulable(node) }
+
+ var candidates []string
+ for _, keep := range []func(string) bool{both, schedulable, isPreferred} {
+ candidates = slices.DeleteFunc(slices.Clone(available), func(node string) bool { return !keep(node) })
+ if len(candidates) > 0 {
+ break
+ }
+ }
+
+ if len(candidates) == 0 {
+ candidates = available
+ }
+
+ if len(candidates) == 0 {
+ return ""
+ }
+
+ if len(preferredNodes) > 0 && slices.Contains(candidates, preferredNodes[0]) {
+ return preferredNodes[0]
+ }
+
+ return candidates[rand.Intn(len(candidates))]
+}
+
// FindSnapsByID searches the snapshot in the backend
func (s *Linstor) FindSnapsByID(ctx context.Context, id string) ([]*volume.Snapshot, error) {
snapshotId, err := volume.ParseSnapshotId(id)
diff --git a/pkg/client/linstor_test.go b/pkg/client/linstor_test.go
index 54c5242..8da6cad 100644
--- a/pkg/client/linstor_test.go
+++ b/pkg/client/linstor_test.go
@@ -994,3 +994,93 @@ func TestReconcileResourceDefinitionRGConflict(t *testing.T) {
})
}
}
+
+func TestLinstor_reconcileSnapshotResourcesSkipsCordonedNodes(t *testing.T) {
+ // A replicated volume has a snapshot on three nodes, one of them cordoned. The scheduler selected a node
+ // without the snapshot. Restoring on the cordoned node pins the new volume to a node no pod can be scheduled
+ // on, so the restore must always go to one of the other two.
+ const (
+ sourceRD = "pvc-source"
+ snapName = "snapshot-1"
+ targetRD = "pvc-target"
+ cordoned = "node-3"
+ )
+
+ snapNodes := []string{"node-1", "node-2", cordoned}
+ preferred := []string{"node-9", "node-1", "node-2", cordoned}
+
+ restoredOn := map[string]int{}
+
+ for range 50 {
+ res := mocks.ResourceProvider{}
+ res.On("GetAll", mock.Anything, targetRD).Return([]lapi.Resource{}, nil)
+ res.On("GetSnapshot", mock.Anything, sourceRD, snapName).Return(lapi.Snapshot{Name: snapName, Nodes: snapNodes}, nil)
+ res.On("RestoreSnapshot", mock.Anything, sourceRD, snapName, mock.Anything).Run(func(args mock.Arguments) {
+ restoredOn[args.Get(3).(lapi.SnapshotRestore).Nodes[0]]++
+ }).Return(nil)
+
+ nodes := mocks.NodeProvider{}
+ for _, n := range snapNodes {
+ nodes.On("Get", mock.Anything, n).Return(lapi.Node{Name: n, ConnectionStatus: "ONLINE"}, nil)
+ }
+
+ cl := Linstor{
+ client: &lc.HighLevelClient{Client: &lapi.Client{Resources: &res, Nodes: &nodes}},
+ log: logrus.WithField("test", t.Name()),
+ nodeSchedulable: func(_ context.Context, node string) (bool, error) {
+ return node != cordoned, nil
+ },
+ }
+
+ snap := &volume.Snapshot{SnapshotId: volume.SnapshotId{SourceName: sourceRD, SnapshotName: snapName}}
+ assert.NoError(t, cl.reconcileSnapshotResources(context.Background(), snap, targetRD, preferred))
+ }
+
+ assert.Zero(t, restoredOn[cordoned], "restored on the cordoned node: %v", restoredOn)
+ assert.Equal(t, 50, restoredOn["node-1"]+restoredOn["node-2"])
+}
+
+func TestSelectRestoreNode(t *testing.T) {
+ all := func(string) bool { return true }
+ notNode3 := func(n string) bool { return n != "node-3" }
+
+ t.Run("selected node holds the snapshot", func(t *testing.T) {
+ for range 20 {
+ assert.Equal(t, "node-2", selectRestoreNode([]string{"node-1", "node-2", "node-3"}, []string{"node-2", "node-1", "node-3"}, all))
+ }
+ })
+
+ t.Run("selected node is cordoned", func(t *testing.T) {
+ for range 20 {
+ assert.NotEqual(t, "node-3", selectRestoreNode([]string{"node-1", "node-3"}, []string{"node-3", "node-1"}, notNode3))
+ }
+ })
+
+ t.Run("schedulable node outside the preferred nodes wins over a cordoned preferred node", func(t *testing.T) {
+ for range 20 {
+ assert.Equal(t, "node-2", selectRestoreNode([]string{"node-3", "node-2"}, []string{"node-3"}, notNode3))
+ }
+ })
+
+ t.Run("only cordoned nodes hold the snapshot", func(t *testing.T) {
+ assert.Equal(t, "node-3", selectRestoreNode([]string{"node-3"}, []string{"node-1", "node-3"}, notNode3))
+ })
+
+ t.Run("snapshot nodes outside the preferred nodes are a fallback", func(t *testing.T) {
+ assert.Equal(t, "node-2", selectRestoreNode([]string{"node-1", "node-2"}, []string{"node-2"}, all))
+ assert.Equal(t, "node-1", selectRestoreNode([]string{"node-1"}, []string{"node-2"}, all))
+ })
+
+ t.Run("no available node", func(t *testing.T) {
+ assert.Empty(t, selectRestoreNode(nil, []string{"node-1"}, all))
+ })
+
+ t.Run("restores are spread when the selected node lacks the snapshot", func(t *testing.T) {
+ seen := map[string]bool{}
+ for range 200 {
+ seen[selectRestoreNode([]string{"node-1", "node-2"}, []string{"node-9", "node-1", "node-2"}, all)] = true
+ }
+
+ assert.Equal(t, map[string]bool{"node-1": true, "node-2": true}, seen)
+ })
+}