fix(kubernetes): bring up a replacement worker before draining the old one

A worker pool rollout (instance type, image or version change) ran with
maxUnavailable 1, so CAPI drained and deleted an old Machine as soon as
its replacement had been created rather than once it had joined. A
one-node pool was left with no node for the few minutes the new VM
took to boot and register, and every pool lost one node of capacity
for each step of the rollout.

With maxUnavailable 0 a node is only drained once its replacement is
up. The surge stays at the pool ceiling, with a floor of 1 because
CAPI refuses a rolling update where both values are 0 and maxReplicas
may be 0.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
This commit is contained in:
Mattia Eleuteri committed 2026-10-29 10:06:40 +01:00
1 parent 048c029e29
commit 55066b981d
3 files changed
+77 -5

No files matched your search

@@ -1024,10 +1024,15 @@ spec:
# action until the autoscaler notices unschedulable Pods and
# scales back up — a real disruption window on every routine
# platform bump.
{{- /* maxUnavailable 0 makes a rollout (instance type, image, version)
wait for each replacement Machine to be up before draining an old
one; with 1, a one-node pool lost its only node until the new one
joined. CAPI refuses maxSurge and maxUnavailable both at 0, and
maxReplicas may be 0, hence the floor of 1. */}}
strategy:
rollingUpdate:
maxSurge: {{ $group.maxReplicas }}
maxUnavailable: 1
maxSurge: {{ max 1 ($group.maxReplicas | int) }}
maxUnavailable: 0
type: RollingUpdate
selector:
matchLabels:
@@ -97,7 +97,7 @@ matches the golden render of a GPU pool:
strategy:
rollingUpdate:
maxSurge: 3
maxUnavailable: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
@@ -250,7 +250,7 @@ matches the golden render of a kubelet-override pool:
strategy:
rollingUpdate:
maxSurge: 3
maxUnavailable: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
@@ -405,7 +405,7 @@ matches the golden render of the pool objects:
strategy:
rollingUpdate:
maxSurge: 3
maxUnavailable: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
@@ -0,0 +1,67 @@
suite: worker pool rollout keeps capacity
templates:
- templates/nodegroup.yaml
release:
name: kubernetes-nodes-myk8s-md0
namespace: tenant-test
set:
cluster: myk8s
_cluster:
cluster-domain: cozy.local
version: "v1.35"
minReplicas: 1
maxReplicas: 1
instanceType: ""
diskSize: 20Gi
storageClass: replicated
resources:
cpu: "2"
memory: 4Gi
tests:
# A rollout replaces every Machine. With maxUnavailable 1 a one-node pool
# drained and deleted its only node as soon as the replacement Machine was
# created, leaving the pool empty until the new node joined.
- it: never drains a node before its replacement is ready
documentSelector:
path: kind
value: MachineDeployment
asserts:
- equal:
path: spec.strategy.type
value: RollingUpdate
- equal:
path: spec.strategy.rollingUpdate.maxUnavailable
value: 0
- equal:
path: spec.strategy.rollingUpdate.maxSurge
value: 1
- it: surges up to the pool ceiling
set:
maxReplicas: 5
documentSelector:
path: kind
value: MachineDeployment
asserts:
- equal:
path: spec.strategy.rollingUpdate.maxSurge
value: 5
- equal:
path: spec.strategy.rollingUpdate.maxUnavailable
value: 0
# CAPI rejects a RollingUpdate with maxSurge and maxUnavailable both 0.
- it: keeps a surge of one on a pool capped at zero
set:
minReplicas: 0
maxReplicas: 0
documentSelector:
path: kind
value: MachineDeployment
asserts:
- equal:
path: spec.strategy.rollingUpdate.maxSurge
value: 1
- equal:
path: spec.strategy.rollingUpdate.maxUnavailable
value: 0