mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
fix(linstor): constrain CSI provisioning to the scheduler's chosen node
Without --strict-topology, external-provisioner passes EVERY topology segment as `requisite` and merely *prefers* the node the scheduler picked, so when that node is out of space linstor-csi may legally provision the volume somewhere else. For a node-pinned volume (allowRemoteVolumeAccess: false, the `local` class that is the cluster default) the result is a PV whose node affinity contradicts the pod's other volumes, and the pod is then unschedulable forever against pv "pvc-..." node affinity doesn't match node "srv1": no matching NodeSelectorTerms with nothing erroring anywhere. It bites hardest on CDI imports, because an importer pod mounts two volumes: the disk, and a ~21 GiB scratch volume CDI creates only AFTER the pod is scheduled. The scheduler therefore sizes the node against the disk alone, can place two importers where both disks plus both scratch volumes do not fit, and the second scratch lands elsewhere. Capacity tracking cannot help -- the scratch volume does not exist when placement is decided, and CSIStorageCapacity agreed with LINSTOR to within 5% when this was measured. Reproduced on 2026-08-25 by cordoning two of three nodes so both importers landed on one constrained to 45 GiB: enough for two 20 GiB disks, not for their scratch. With the flag the scratch PVC stays Pending and the pod reports `1 node(s) did not have enough free storage`, then binds and completes once space frees. A silent permanent deadlock becomes a legible, self-correcting wait. The full argument list is restated because `args` has no patchMergeKey, so a strategic merge replaces rather than appends. That makes the list a drift hazard on every piraeus-operator bump, and the note beside it says so -- the count assertion in the test guards this chart's own list and cannot see a flag the operator adds upstream. Verified for node-pinned volumes. For `replicated` the PV is remotely accessible so the constraint is satisfiable wherever the pod lands, and that case is reasoned rather than measured -- hence the switch. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
This commit is contained in:
3 files changed
+103
No files matched your search
@@ -83,6 +83,30 @@ spec:
|
||||
containers:
|
||||
- name: linstor-csi
|
||||
image: {{ .Values.linstorCSI.image.repository }}:{{ .Values.linstorCSI.image.tag }}
|
||||
{{- if .Values.linstorCSI.strictTopology }}
|
||||
# See values.yaml linstorCSI.strictTopology for what this prevents.
|
||||
#
|
||||
# The full argument list is restated because `args` has no patchMergeKey,
|
||||
# so a strategic merge REPLACES it rather than appending. That makes this
|
||||
# list a drift hazard: re-check it against the operator's defaults on
|
||||
# every piraeus-operator bump, or a newly added upstream flag is silently
|
||||
# dropped. The $(VAR) references are resolved from the container's env by
|
||||
# Kubernetes and are stable across versions.
|
||||
- name: csi-provisioner
|
||||
args:
|
||||
- --v=$(VERBOSE)
|
||||
- --csi-address=$(ADDRESS)
|
||||
- --timeout=$(TIMEOUT)
|
||||
- --leader-election=$(LEADER_ELECTION)
|
||||
- --leader-election-namespace=$(NAMESPACE)
|
||||
- --worker-threads=$(WORKER_THREADS)
|
||||
- --http-endpoint=$(HTTP_ENDPOINT)
|
||||
- --default-fstype=$(DEFAULT_FS_TYPE)
|
||||
- --enable-capacity=$(ENABLE_CAPACITY)
|
||||
- --extra-create-metadata=$(EXTRA_CREATE_METADATA)
|
||||
- --capacity-ownerref-level=$(CAPACITY_OWNERREF_LEVEL)
|
||||
- --strict-topology
|
||||
{{- end }}
|
||||
csiNode:
|
||||
podTemplate:
|
||||
spec:
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
suite: CSI provisioner strict topology
|
||||
|
||||
# Without --strict-topology, external-provisioner passes every topology segment
|
||||
# as `requisite` and only *prefers* the node the scheduler picked, so linstor-csi
|
||||
# may legally place a node-pinned volume elsewhere when that node is full. The
|
||||
# pod is then unschedulable forever against a PV whose affinity contradicts its
|
||||
# other volumes -- a silent, permanent deadlock on a healthy-looking cluster.
|
||||
#
|
||||
# Reproduced and fixed by measurement 2026-08-25; see values.yaml for the full
|
||||
# mechanism and what was measured versus reasoned.
|
||||
|
||||
templates:
|
||||
- templates/cluster.yaml
|
||||
|
||||
tests:
|
||||
- it: the provisioner is constrained to the scheduler's chosen node by default
|
||||
template: templates/cluster.yaml
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.csiController.podTemplate.spec.containers[?(@.name=="csi-provisioner")].args
|
||||
content: --strict-topology
|
||||
|
||||
# The whole argument list is restated by the patch because `args` has no
|
||||
# patchMergeKey and a strategic merge replaces it. If a piraeus-operator bump
|
||||
# adds an upstream flag, this list silently drops it -- so pin the count as a
|
||||
# tripwire that forces a re-check rather than letting the loss go unnoticed.
|
||||
- it: the restated argument list is complete
|
||||
template: templates/cluster.yaml
|
||||
asserts:
|
||||
- lengthEqual:
|
||||
path: spec.csiController.podTemplate.spec.containers[?(@.name=="csi-provisioner")].args
|
||||
count: 12
|
||||
- contains:
|
||||
path: spec.csiController.podTemplate.spec.containers[?(@.name=="csi-provisioner")].args
|
||||
content: --enable-capacity=$(ENABLE_CAPACITY)
|
||||
|
||||
# Asserted by container count, not a negated `contains`: a negation against a
|
||||
# path that no longer resolves passes vacuously.
|
||||
- it: the override can be switched off
|
||||
template: templates/cluster.yaml
|
||||
set:
|
||||
linstorCSI.strictTopology: false
|
||||
asserts:
|
||||
- lengthEqual:
|
||||
path: spec.csiController.podTemplate.spec.containers
|
||||
count: 1
|
||||
- equal:
|
||||
path: spec.csiController.podTemplate.spec.containers[0].name
|
||||
value: linstor-csi
|
||||
@@ -42,6 +42,36 @@ linstor:
|
||||
minutes: 30
|
||||
allowCleanup: true
|
||||
linstorCSI:
|
||||
## Constrain provisioning to the node the scheduler actually picked.
|
||||
##
|
||||
## Without it, external-provisioner passes EVERY topology segment as
|
||||
## `requisite` and merely *prefers* the selected node, so when that node is out
|
||||
## of space linstor-csi may legally provision the volume somewhere else. For a
|
||||
## node-pinned volume (`allowRemoteVolumeAccess: false`, the `local` class that
|
||||
## is the cluster default) the result is a PV whose node affinity contradicts
|
||||
## the pod's other volumes, and the pod is then unschedulable FOREVER:
|
||||
##
|
||||
## pv "pvc-..." node affinity doesn't match node "srv1": no matching NodeSelectorTerms
|
||||
##
|
||||
## It bites hardest on CDI imports, because an importer pod mounts two volumes
|
||||
## -- the disk and a ~21 GiB scratch volume that CDI only creates AFTER the pod
|
||||
## is scheduled. The scheduler therefore sizes the node against the disk alone,
|
||||
## can place two importers on a node that cannot hold both disks plus both
|
||||
## scratch volumes, and the second scratch is then placed elsewhere.
|
||||
##
|
||||
## Reproduced and fixed by measurement 2026-08-25 on a three-node cluster: with
|
||||
## the flag absent the scratch PV landed on a different node than its disk and
|
||||
## the import hung indefinitely at ImportScheduled with no error anywhere; with
|
||||
## it set the scratch PVC stays Pending and the pod reports the actionable
|
||||
## `1 node(s) did not have enough free storage`, then binds and completes on
|
||||
## its own once space frees up. A silent permanent deadlock becomes a legible,
|
||||
## self-correcting wait.
|
||||
##
|
||||
## Verified for node-pinned (`local`) volumes, which is what the cluster
|
||||
## default uses. For `replicated` volumes the PV is remotely accessible, so the
|
||||
## constraint is satisfiable wherever the pod lands; that case is reasoned
|
||||
## rather than measured, hence the switch.
|
||||
strictTopology: true
|
||||
image:
|
||||
repository: ghcr.io/cozystack/cozystack/linstor-csi
|
||||
tag: v1.6.0@sha256:ae854423de62e94fb89b59d92c20da8939196e643e83b2d60d9f6915d5b0e55c
|
||||
|
||||
Reference in new issue
Block a user