Files
cozystack/hack
Aleksei Sviridkin da17254bbc fix(kubernetes): bring up a replacement worker before draining the old one (#5153)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Worker pool MachineDeployments roll out with `maxUnavailable: 1`. CAPI
therefore drains and deletes an old Machine as soon as its replacement
Machine has been created, not once the replacement has joined. Observed
on a one-node pool changing instance type (`s1.large` to `s1.medium`):
the new Machine was created at 13:07:19, the old node was drained and
deleted 17 s later, and the new node was Ready around 13:11, so the pool
had no node for about four minutes. Larger pools lose one node of
capacity at every step of the rollout for the same reason.

This sets `maxUnavailable: 0`, so a node is drained only once its
replacement is up. `maxSurge` stays at the pool ceiling (`maxReplicas`),
with a floor of 1: CAPI rejects a RollingUpdate where both values are 0,
and `maxReplicas: 0` is accepted by the schema.

Exception: a pool with `gpus` keeps `maxUnavailable: 1`. Its replacement
VM may need the very passed-through GPU the old node holds, and with
`maxUnavailable: 0` it would stay Pending and the rollout would never
finish. A Proxmox pool that passes GPUs through its template carries no
`gpus` value, so the chart cannot recognise it: the new optional
`maxUnavailable` value lets such a pool (or any pool) set the strategy
explicitly, and set to 1 restores the drain-first behaviour.

The value is bounded to 0..2147483647, since CAPI decodes it as an
int32. ComputePlane pools render through the same chart, so
`packages/extra/computeplane` exposes `nodeGroups.<pool>.maxUnavailable`
and its allowlist forwards it.

Trade-off: a rollout now needs room for at least one extra VM. A tenant
at its quota stalls the rollout (the new Machine stays Pending) instead
of temporarily shrinking the pool, which is the safer failure.

Tests: `tests/rollout_strategy_test.yaml` (7 cases: the default, the
ceiling, a zero ceiling, a GPU pool, and the explicit override) fails on
`main` without the template change; the render snapshot is updated. Two
more cases cover the int32 maximum, accepted at 2147483647 and refused
one above. `helm unittest packages/apps/kubernetes-nodes`: 252 passed.
`packages/extra/computeplane/tests/cluster_test.yaml` checks that the
wrapper forwards `maxUnavailable` (it fails without the allowlist entry)
and refuses a value above the int32 range: 15 passed.

Backport: on `release-1.6` the same strategy is also rendered by
`packages/apps/kubernetes/templates/cluster.yaml` (the in-chart
`nodeGroups`), and the snapshot file does not exist there, so the
automatic backport cannot apply. A manual backport PR to `release-1.6`
covering both charts follows. `release-1.7` keeps `kind/backport`.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#79 (tracking issue:
the provider change is out of scope here). `cozystack_kubernetes_nodes`
has no attribute for the new `maxUnavailable`, so a Terraform-managed
pool cannot drain first; the provider has no ComputePlane resource yet.
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): worker pool rollouts (instance type, image or version changes) now bring each replacement node up before draining the old one, instead of removing a node first. A rollout needs room for one extra VM. Pools with GPUs keep the previous behaviour.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added a pool-level `maxUnavailable` setting to control how many
workers may be removed before replacements are ready. It defaults to 0
for non-GPU pools and 1 for GPU pools; configured values override these
defaults.
* Proxmox pools passing through GPUs can set `maxUnavailable` to 1 to
help prevent rollouts from waiting indefinitely.
* **Bug Fixes**
* Rolling updates now use at least one surge node, even when the replica
limit is zero.
* **Documentation**
  * Documented rollout settings, defaults, and capacity considerations.
* **Tests**
* Added coverage for defaults, overrides, surge limits, and invalid
values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-15 02:48:13 +03:00
..