fix(kubernetes): keep maxUnavailable 1 on GPU worker pools

A replacement VM in a pool with passed-through GPUs may need the very
GPU the old node holds. With maxUnavailable 0 it would stay Pending
and the rollout would never finish, so those pools keep removing a
node first.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
This commit is contained in:
Mattia Eleuteri committed 2026-10-29 10:45:21 +01:00
1 parent 55066b981d
commit 392bbb3c96
3 files changed
+22 -3

No files matched your search

@@ -1027,12 +1027,14 @@ spec:
{{- /* maxUnavailable 0 makes a rollout (instance type, image, version)
wait for each replacement Machine to be up before draining an old
one; with 1, a one-node pool lost its only node until the new one
joined. CAPI refuses maxSurge and maxUnavailable both at 0, and
joined. A pool with passed-through GPUs keeps 1: its replacement VM
may need the very GPU the old node holds, and would wait for it
forever. CAPI refuses maxSurge and maxUnavailable both at 0, and
maxReplicas may be 0, hence the floor of 1. */}}
strategy:
rollingUpdate:
maxSurge: {{ max 1 ($group.maxReplicas | int) }}
maxUnavailable: 0
maxUnavailable: {{ ternary 1 0 (not (empty $group.gpus)) }}
type: RollingUpdate
selector:
matchLabels:
@@ -97,7 +97,7 @@ matches the golden render of a GPU pool:
strategy:
rollingUpdate:
maxSurge: 3
maxUnavailable: 0
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
@@ -65,3 +65,20 @@ tests:
- equal:
path: spec.strategy.rollingUpdate.maxUnavailable
value: 0
# A replacement VM in a pool with passed-through GPUs may need the very GPU
# the old node holds; with maxUnavailable 0 it would wait for it forever.
- it: lets a GPU pool remove a node before its replacement is up
set:
gpus:
- name: nvidia.com/AD102GL_L40S
documentSelector:
path: kind
value: MachineDeployment
asserts:
- equal:
path: spec.strategy.rollingUpdate.maxUnavailable
value: 1
- equal:
path: spec.strategy.rollingUpdate.maxSurge
value: 1