mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
fix(kubernetes): bring up a replacement worker before draining the old one (#5153)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Worker pool MachineDeployments roll out with `maxUnavailable: 1`. CAPI therefore drains and deletes an old Machine as soon as its replacement Machine has been created, not once the replacement has joined. Observed on a one-node pool changing instance type (`s1.large` to `s1.medium`): the new Machine was created at 13:07:19, the old node was drained and deleted 17 s later, and the new node was Ready around 13:11, so the pool had no node for about four minutes. Larger pools lose one node of capacity at every step of the rollout for the same reason. This sets `maxUnavailable: 0`, so a node is drained only once its replacement is up. `maxSurge` stays at the pool ceiling (`maxReplicas`), with a floor of 1: CAPI rejects a RollingUpdate where both values are 0, and `maxReplicas: 0` is accepted by the schema. Exception: a pool with `gpus` keeps `maxUnavailable: 1`. Its replacement VM may need the very passed-through GPU the old node holds, and with `maxUnavailable: 0` it would stay Pending and the rollout would never finish. A Proxmox pool that passes GPUs through its template carries no `gpus` value, so the chart cannot recognise it: the new optional `maxUnavailable` value lets such a pool (or any pool) set the strategy explicitly, and set to 1 restores the drain-first behaviour. The value is bounded to 0..2147483647, since CAPI decodes it as an int32. ComputePlane pools render through the same chart, so `packages/extra/computeplane` exposes `nodeGroups.<pool>.maxUnavailable` and its allowlist forwards it. Trade-off: a rollout now needs room for at least one extra VM. A tenant at its quota stalls the rollout (the new Machine stays Pending) instead of temporarily shrinking the pool, which is the safer failure. Tests: `tests/rollout_strategy_test.yaml` (7 cases: the default, the ceiling, a zero ceiling, a GPU pool, and the explicit override) fails on `main` without the template change; the render snapshot is updated. Two more cases cover the int32 maximum, accepted at 2147483647 and refused one above. `helm unittest packages/apps/kubernetes-nodes`: 252 passed. `packages/extra/computeplane/tests/cluster_test.yaml` checks that the wrapper forwards `maxUnavailable` (it fails without the allowlist entry) and refuses a value above the int32 range: 15 passed. Backport: on `release-1.6` the same strategy is also rendered by `packages/apps/kubernetes/templates/cluster.yaml` (the in-chart `nodeGroups`), and the snapshot file does not exist there, so the automatic backport cannot apply. A manual backport PR to `release-1.6` covering both charts follows. `release-1.7` keeps `kind/backport`. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: cozystack/terraform-provider-cozystack#79 (tracking issue: the provider change is out of scope here). `cozystack_kubernetes_nodes` has no attribute for the new `maxUnavailable`, so a Terraform-managed pool cannot drain first; the provider has no ComputePlane resource yet. - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubernetes): worker pool rollouts (instance type, image or version changes) now bring each replacement node up before draining the old one, instead of removing a node first. A rollout needs room for one extra VM. Pools with GPUs keep the previous behaviour. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added a pool-level `maxUnavailable` setting to control how many workers may be removed before replacements are ready. It defaults to 0 for non-GPU pools and 1 for GPU pools; configured values override these defaults. * Proxmox pools passing through GPUs can set `maxUnavailable` to 1 to help prevent rollouts from waiting indefinitely. * **Bug Fixes** * Rolling updates now use at least one surge node, even when the replica limit is zero. * **Documentation** * Documented rollout settings, defaults, and capacity considerations. * **Tests** * Added coverage for defaults, overrides, surge limits, and invalid values. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
16 files changed
+281
-12
No files matched your search
+4
@@ -51,6 +51,10 @@ type ConfigSpec struct {
|
||||
// List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
|
||||
// +kubebuilder:default:={}
|
||||
Gpus []GPU `json:"gpus,omitempty"`
|
||||
// Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.
|
||||
// +kubebuilder:validation:Minimum=0
|
||||
// +kubebuilder:validation:Maximum=2.147483647e+09
|
||||
MaxUnavailable *int `json:"maxUnavailable,omitempty"`
|
||||
// Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.
|
||||
KernelModules []KernelModule `json:"kernelModules,omitempty"`
|
||||
// Kubelet resource reservations for this pool. An empty reservation is computed from the worker's size: `resources` when set, otherwise the `instanceType`.
|
||||
|
||||
@@ -67,6 +67,11 @@ func (in *ConfigSpec) DeepCopyInto(out *ConfigSpec) {
|
||||
*out = make([]GPU, len(*in))
|
||||
copy(*out, *in)
|
||||
}
|
||||
if in.MaxUnavailable != nil {
|
||||
in, out := &in.MaxUnavailable, &out.MaxUnavailable
|
||||
*out = new(int)
|
||||
**out = **in
|
||||
}
|
||||
if in.KernelModules != nil {
|
||||
in, out := &in.KernelModules, &out.KernelModules
|
||||
*out = make([]KernelModule, len(*in))
|
||||
|
||||
@@ -102,7 +102,9 @@ spec:
|
||||
#
|
||||
# nameOverride is outside the module's per-pool allowlist. The ComputePlane
|
||||
# schema accepts it, so it reaches the module and the pool check below can
|
||||
# see whether the module dropped it.
|
||||
# see whether the module dropped it. maxUnavailable is set to 0 because a
|
||||
# zero lost anywhere on the way would let a GPU pool fall back to the
|
||||
# chart's default of 1; only this path crosses the aggregated API.
|
||||
- script:
|
||||
timeout: 3m
|
||||
content: |
|
||||
@@ -118,6 +120,7 @@ spec:
|
||||
diskSize: 20Gi
|
||||
instanceType: u1.medium
|
||||
logSerialConsole: true
|
||||
maxUnavailable: 0
|
||||
minReplicas: 2
|
||||
maxReplicas: 2
|
||||
podCpuLimit: 2
|
||||
@@ -175,8 +178,9 @@ spec:
|
||||
enabled: true
|
||||
exposeMethod: Proxied
|
||||
|
||||
# The pool takes the patched shape, and the nameOverride the patch sent
|
||||
# does not reach it, nor does any other key outside the allowlist.
|
||||
# The pool takes the patched shape, the explicit maxUnavailable of 0
|
||||
# included, and the nameOverride the patch sent does not reach it, nor does
|
||||
# any other key outside the allowlist.
|
||||
- script:
|
||||
timeout: 1m
|
||||
content: |
|
||||
@@ -186,8 +190,8 @@ spec:
|
||||
sc=$(cozy_e2e_storage_class)
|
||||
values=$(timeout -k 5 30 kubectl -n tenant-cplane get helmrelease kubernetes-nodes-computeplane-cluster-md0 -o json | jq -c '.spec.values')
|
||||
echo "pool values: $values"
|
||||
echo "$values" | jq -e --arg sc "$sc" '.storageClass == $sc and .minReplicas == 2 and .cluster == "computeplane-cluster" and .clusterReleaseName == "computeplane-cluster"'
|
||||
extra=$(echo "$values" | jq -r 'keys - ["cluster","clusterReleaseName","minReplicas","maxReplicas","instanceType","diskSize","storageClass","roles","resources","gpus","kubelet","logSerialConsole","maxUnhealthy","nodeStartupTimeout","readyUnknownTimeout","readyFalseTimeout","podCpuLimit","podCpuRequest","maxNodeProvisionTime"] | .[]')
|
||||
echo "$values" | jq -e --arg sc "$sc" '.storageClass == $sc and .minReplicas == 2 and .maxUnavailable == 0 and .cluster == "computeplane-cluster" and .clusterReleaseName == "computeplane-cluster"'
|
||||
extra=$(echo "$values" | jq -r 'keys - ["cluster","clusterReleaseName","minReplicas","maxReplicas","instanceType","diskSize","storageClass","roles","resources","gpus","kubelet","logSerialConsole","maxUnhealthy","nodeStartupTimeout","readyUnknownTimeout","readyFalseTimeout","podCpuLimit","podCpuRequest","maxNodeProvisionTime","maxUnavailable"] | .[]')
|
||||
if [ -n "$extra" ]; then
|
||||
echo "pool HelmRelease carries keys outside the module allowlist: $extra" >&2
|
||||
exit 1
|
||||
|
||||
@@ -44,6 +44,7 @@ The pool's Kubernetes minor (`version`) may lag the cluster's during an upgrade
|
||||
| `podCpuRequest` | CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. | `quantity` | `""` |
|
||||
| `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
|
||||
| `gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` |
|
||||
| `maxUnavailable` | Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. | `*int` | `null` |
|
||||
| `kernelModules` | Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. | `[]object` | `[]` |
|
||||
| `kernelModules[i].name` | Module name as `modprobe` takes it, e.g. `nvidia_uvm`. | `string` | `""` |
|
||||
| `kernelModules[i].parameters` | Module parameters, each as a bare `key=value` string. | `[]string` | `[]` |
|
||||
|
||||
@@ -1021,10 +1021,22 @@ spec:
|
||||
# action until the autoscaler notices unschedulable Pods and
|
||||
# scales back up — a real disruption window on every routine
|
||||
# platform bump.
|
||||
{{- /* maxUnavailable 0 makes a rollout (instance type, image, version)
|
||||
wait for each replacement Machine to be up before draining an old
|
||||
one; with 1, a one-node pool lost its only node until the new one
|
||||
joined. A pool with passed-through GPUs keeps 1: its replacement VM
|
||||
may need the very GPU the old node holds, and would wait for it
|
||||
forever. A Proxmox template's GPU is invisible here, hence the
|
||||
maxUnavailable override. CAPI refuses maxSurge and maxUnavailable both
|
||||
at 0, and maxReplicas may be 0, hence the floor of 1. */}}
|
||||
{{- $maxUnavailable := ternary 1 0 (not (empty $group.gpus)) }}
|
||||
{{- if not (kindIs "invalid" $.Values.maxUnavailable) }}
|
||||
{{- $maxUnavailable = $.Values.maxUnavailable | int }}
|
||||
{{- end }}
|
||||
strategy:
|
||||
rollingUpdate:
|
||||
maxSurge: {{ $group.maxReplicas }}
|
||||
maxUnavailable: 1
|
||||
maxSurge: {{ max 1 ($group.maxReplicas | int) }}
|
||||
maxUnavailable: {{ $maxUnavailable }}
|
||||
type: RollingUpdate
|
||||
selector:
|
||||
matchLabels:
|
||||
|
||||
@@ -250,7 +250,7 @@ matches the golden render of a kubelet-override pool:
|
||||
strategy:
|
||||
rollingUpdate:
|
||||
maxSurge: 3
|
||||
maxUnavailable: 1
|
||||
maxUnavailable: 0
|
||||
type: RollingUpdate
|
||||
template:
|
||||
metadata:
|
||||
@@ -405,7 +405,7 @@ matches the golden render of the pool objects:
|
||||
strategy:
|
||||
rollingUpdate:
|
||||
maxSurge: 3
|
||||
maxUnavailable: 1
|
||||
maxUnavailable: 0
|
||||
type: RollingUpdate
|
||||
template:
|
||||
metadata:
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
suite: worker pool rollout keeps capacity
|
||||
templates:
|
||||
- templates/nodegroup.yaml
|
||||
release:
|
||||
name: kubernetes-nodes-myk8s-md0
|
||||
namespace: tenant-test
|
||||
set:
|
||||
cluster: myk8s
|
||||
_cluster:
|
||||
cluster-domain: cozy.local
|
||||
version: "v1.35"
|
||||
minReplicas: 1
|
||||
maxReplicas: 1
|
||||
instanceType: ""
|
||||
diskSize: 20Gi
|
||||
storageClass: replicated
|
||||
resources:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
tests:
|
||||
# A rollout replaces every Machine. With maxUnavailable 1 a one-node pool
|
||||
# drained and deleted its only node as soon as the replacement Machine was
|
||||
# created, leaving the pool empty until the new node joined.
|
||||
- it: never drains a node before its replacement is ready
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: MachineDeployment
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.strategy.type
|
||||
value: RollingUpdate
|
||||
- equal:
|
||||
path: spec.strategy.rollingUpdate.maxUnavailable
|
||||
value: 0
|
||||
- equal:
|
||||
path: spec.strategy.rollingUpdate.maxSurge
|
||||
value: 1
|
||||
|
||||
- it: surges up to the pool ceiling
|
||||
set:
|
||||
maxReplicas: 5
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: MachineDeployment
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.strategy.rollingUpdate.maxSurge
|
||||
value: 5
|
||||
- equal:
|
||||
path: spec.strategy.rollingUpdate.maxUnavailable
|
||||
value: 0
|
||||
|
||||
# CAPI rejects a RollingUpdate with maxSurge and maxUnavailable both 0.
|
||||
- it: keeps a surge of one on a pool capped at zero
|
||||
set:
|
||||
minReplicas: 0
|
||||
maxReplicas: 0
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: MachineDeployment
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.strategy.rollingUpdate.maxSurge
|
||||
value: 1
|
||||
- equal:
|
||||
path: spec.strategy.rollingUpdate.maxUnavailable
|
||||
value: 0
|
||||
|
||||
# A replacement VM in a pool with passed-through GPUs may need the very GPU
|
||||
# the old node holds; with maxUnavailable 0 it would wait for it forever.
|
||||
- it: lets a GPU pool remove a node before its replacement is up
|
||||
set:
|
||||
gpus:
|
||||
- name: nvidia.com/AD102GL_L40S
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: MachineDeployment
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.strategy.rollingUpdate.maxUnavailable
|
||||
value: 1
|
||||
- equal:
|
||||
path: spec.strategy.rollingUpdate.maxSurge
|
||||
value: 1
|
||||
|
||||
# A Proxmox template that passes a GPU through leaves gpus empty, so the
|
||||
# chart cannot tell; the pool says so itself.
|
||||
- it: lets a pool set maxUnavailable explicitly
|
||||
set:
|
||||
maxUnavailable: 1
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: MachineDeployment
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.strategy.rollingUpdate.maxUnavailable
|
||||
value: 1
|
||||
|
||||
- it: honours an explicit 0 on a GPU pool
|
||||
set:
|
||||
maxUnavailable: 0
|
||||
gpus:
|
||||
- name: nvidia.com/AD102GL_L40S
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: MachineDeployment
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.strategy.rollingUpdate.maxUnavailable
|
||||
value: 0
|
||||
|
||||
- it: refuses a negative maxUnavailable (schema minimum 0)
|
||||
set:
|
||||
maxUnavailable: -1
|
||||
asserts:
|
||||
- failedTemplate: {}
|
||||
|
||||
# CAPI decodes the field as an int32, so a larger value would render and
|
||||
# then be refused by the MachineDeployment decoder.
|
||||
- it: refuses a maxUnavailable above the int32 range (schema maximum)
|
||||
set:
|
||||
maxUnavailable: 2147483648
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: "maxUnavailable"
|
||||
|
||||
- it: accepts the int32 maximum
|
||||
set:
|
||||
maxUnavailable: 2147483647
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: MachineDeployment
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.strategy.rollingUpdate.maxUnavailable
|
||||
value: 2147483647
|
||||
@@ -146,6 +146,12 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"maxUnavailable": {
|
||||
"description": "Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.",
|
||||
"type": "integer",
|
||||
"maximum": 2147483647,
|
||||
"minimum": 0
|
||||
},
|
||||
"kernelModules": {
|
||||
"description": "Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.",
|
||||
"type": "array",
|
||||
|
||||
@@ -77,6 +77,10 @@ resources: {}
|
||||
## @param {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
|
||||
gpus: []
|
||||
|
||||
## @param {*int} [maxUnavailable] - Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.
|
||||
## @minimum 0
|
||||
## @maximum 2147483647
|
||||
|
||||
## @param {[]KernelModule} [kernelModules] - Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.
|
||||
|
||||
## @param {Kubelet} [kubelet] - Kubelet resource reservations for this pool. An empty reservation is computed from the worker's size: `resources` when set, otherwise the `instanceType`.
|
||||
|
||||
@@ -59,4 +59,5 @@ See the full design in
|
||||
| `nodeGroups[name].kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `nodeGroups[name].kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` |
|
||||
| `nodeGroups[name].kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` |
|
||||
| `nodeGroups[name].maxUnavailable` | Workers a rollout may remove before their replacement is up. Unset, the `kubernetes-nodes` pool chart uses 0, so each replacement joins before an old worker is drained (1 on a pool with `gpus`). Set 1 on a pool without room for one extra VM, whose rollout would otherwise stall with the new Machine Pending. | `*int` | `null` |
|
||||
|
||||
@@ -211,7 +211,7 @@ spec:
|
||||
clone or repoint itself at another Image Factory, both of which are
|
||||
operator decisions about the platform's own images. Keep this list in
|
||||
sync with the kubernetes-nodes values schema's pool-shape params. */}}
|
||||
{{- with pick $group "minReplicas" "maxReplicas" "instanceType" "diskSize" "storageClass" "roles" "resources" "gpus" "kubelet" "logSerialConsole" "maxUnhealthy" "nodeStartupTimeout" "readyUnknownTimeout" "readyFalseTimeout" "podCpuLimit" "podCpuRequest" "maxNodeProvisionTime" }}
|
||||
{{- with pick $group "minReplicas" "maxReplicas" "instanceType" "diskSize" "storageClass" "roles" "resources" "gpus" "kubelet" "logSerialConsole" "maxUnhealthy" "nodeStartupTimeout" "readyUnknownTimeout" "readyFalseTimeout" "podCpuLimit" "podCpuRequest" "maxNodeProvisionTime" "maxUnavailable" }}
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
valuesFrom:
|
||||
|
||||
@@ -280,6 +280,93 @@ tests:
|
||||
- notExists:
|
||||
path: spec.values.nameOverride
|
||||
|
||||
# kubernetes-nodes drains a worker only once its replacement is up; a pool
|
||||
# without room for one extra VM needs maxUnavailable 1 to finish a rollout,
|
||||
# so the allowlist must forward it.
|
||||
- it: forwards maxUnavailable to the pool chart
|
||||
documentSelector:
|
||||
path: metadata.name
|
||||
value: kubernetes-nodes-computeplane-cluster-md0
|
||||
set:
|
||||
nodeGroups:
|
||||
md0:
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: u1.large
|
||||
diskSize: 32Gi
|
||||
roles: ["ingress-nginx"]
|
||||
maxUnavailable: 1
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.values.maxUnavailable
|
||||
value: 1
|
||||
|
||||
# An explicit 0 must reach the pool chart: there it overrides the 1 a GPU
|
||||
# pool gets by default, so dropping it as empty would change the strategy.
|
||||
- it: forwards an explicit maxUnavailable of 0 to the pool chart
|
||||
documentSelector:
|
||||
path: metadata.name
|
||||
value: kubernetes-nodes-computeplane-cluster-md0
|
||||
set:
|
||||
nodeGroups:
|
||||
md0:
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: u1.large
|
||||
diskSize: 32Gi
|
||||
roles: ["ingress-nginx"]
|
||||
maxUnavailable: 0
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.values.maxUnavailable
|
||||
value: 0
|
||||
|
||||
- it: forwards the int32 maximum to the pool chart
|
||||
documentSelector:
|
||||
path: metadata.name
|
||||
value: kubernetes-nodes-computeplane-cluster-md0
|
||||
set:
|
||||
nodeGroups:
|
||||
md0:
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: u1.large
|
||||
diskSize: 32Gi
|
||||
roles: ["ingress-nginx"]
|
||||
maxUnavailable: 2147483647
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.values.maxUnavailable
|
||||
value: 2147483647
|
||||
|
||||
- it: refuses a maxUnavailable above the int32 range CAPI decodes
|
||||
set:
|
||||
nodeGroups:
|
||||
md0:
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: u1.large
|
||||
diskSize: 32Gi
|
||||
roles: ["ingress-nginx"]
|
||||
maxUnavailable: 2147483648
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: "maxUnavailable"
|
||||
|
||||
- it: refuses a negative maxUnavailable
|
||||
set:
|
||||
nodeGroups:
|
||||
md0:
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: u1.large
|
||||
diskSize: 32Gi
|
||||
roles: ["ingress-nginx"]
|
||||
maxUnavailable: -1
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: "maxUnavailable"
|
||||
|
||||
# Helm caps a HelmRelease name at 53 chars and each pool renders a 38-char fixed
|
||||
# prefix, so a pool key over 15 chars would create a release the apiserver
|
||||
# accepts but Flux never reconciles. The render must refuse instead (the same
|
||||
|
||||
@@ -93,6 +93,12 @@
|
||||
"type": "integer",
|
||||
"default": 10
|
||||
},
|
||||
"maxUnavailable": {
|
||||
"description": "Workers a rollout may remove before their replacement is up. Unset, the `kubernetes-nodes` pool chart uses 0, so each replacement joins before an old worker is drained (1 on a pool with `gpus`). Set 1 on a pool without room for one extra VM, whose rollout would otherwise stall with the new Machine Pending.",
|
||||
"type": "integer",
|
||||
"maximum": 2147483647,
|
||||
"minimum": 0
|
||||
},
|
||||
"minReplicas": {
|
||||
"description": "Minimum number of replicas.",
|
||||
"type": "integer",
|
||||
|
||||
@@ -30,6 +30,9 @@
|
||||
## @field {Resources} [resources] - Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Set both `cpu` and `memory` together or neither.
|
||||
## @field {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
|
||||
## @field {Kubelet} [kubelet] - Kubelet resource reservations for this node group.
|
||||
## @field {*int} [maxUnavailable] - Workers a rollout may remove before their replacement is up. Unset, the `kubernetes-nodes` pool chart uses 0, so each replacement joins before an old worker is drained (1 on a pool with `gpus`). Set 1 on a pool without room for one extra VM, whose rollout would otherwise stall with the new Machine Pending.
|
||||
## @minimum 0
|
||||
## @maximum 2147483647
|
||||
|
||||
## @param {map[string]NodeGroup} nodeGroups - Worker node groups of the ComputePlane cluster (cluster shape only — the security posture is fixed by the module and cannot be overridden through this knob: the module forwards only the documented per-pool fields to the `kubernetes-nodes` pool chart, so any other `NodeGroup` field, e.g. `talos`/`version`/`images`, is silently dropped — accepted but inert). Same structure as the `kubernetes` application's `nodeGroups`. When left empty, the module itself materialises a default scale-from-zero `md0` pool with `roles: [ingress-nginx]` (the wrapped `kubernetes` chart no longer injects one after the Phase 2 split), so the cluster sits idle until a pending Pod (e.g. the ingress-nginx controller) triggers the cluster-autoscaler. Supplied groups are authoritative (not merged with the default), so at least one of them must carry `roles: [ingress-nginx]` — enforced at render time. Each supplied pool key must be at most 15 characters so its `kubernetes-nodes-computeplane-cluster-<pool>` HelmRelease name fits Helm's 53-character limit — also enforced at render time.
|
||||
nodeGroups: {}
|
||||
+1
-1
@@ -8,7 +8,7 @@ spec:
|
||||
singular: computeplane
|
||||
plural: computeplanes
|
||||
openAPISchema: |-
|
||||
{"title":"Chart Values","type":"object","properties":{"nodeGroups":{"description":"Worker node groups of the ComputePlane cluster (cluster shape only — the security posture is fixed by the module and cannot be overridden through this knob: the module forwards only the documented per-pool fields to the `kubernetes-nodes` pool chart, so any other `NodeGroup` field, e.g. `talos`/`version`/`images`, is silently dropped — accepted but inert). Same structure as the `kubernetes` application's `nodeGroups`. When left empty, the module itself materialises a default scale-from-zero `md0` pool with `roles: [ingress-nginx]` (the wrapped `kubernetes` chart no longer injects one after the Phase 2 split), so the cluster sits idle until a pending Pod (e.g. the ingress-nginx controller) triggers the cluster-autoscaler. Supplied groups are authoritative (not merged with the default), so at least one of them must carry `roles: [ingress-nginx]` — enforced at render time. Each supplied pool key must be at most 15 characters so its `kubernetes-nodes-computeplane-cluster-<pool>` HelmRelease name fits Helm's 53-character limit — also enforced at render time.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskSize","instanceType","maxReplicas","minReplicas"],"properties":{"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"kubelet":{"description":"Kubelet resource reservations for this node group.","type":"object","properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"maxReplicas":{"description":"Maximum number of replicas.","type":"integer","default":10},"minReplicas":{"description":"Minimum number of replicas.","type":"integer","default":0},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Set both `cpu` and `memory` together or neither.","type":"object","properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"roles":{"description":"List of node roles. At least one group must carry `ingress-nginx` — the module pins the ingress-nginx addon on, and its controller only schedules onto nodes with that role (enforced at render time).","type":"array","items":{"type":"string"}},"storageClass":{"description":"StorageClass for worker node persistent disks. When empty, the `kubernetes-nodes` pool chart applies its own default (`replicated`).","type":"string","x-cozystack-options":{"source":"storageclass"}}}}}}}
|
||||
{"title":"Chart Values","type":"object","properties":{"nodeGroups":{"description":"Worker node groups of the ComputePlane cluster (cluster shape only — the security posture is fixed by the module and cannot be overridden through this knob: the module forwards only the documented per-pool fields to the `kubernetes-nodes` pool chart, so any other `NodeGroup` field, e.g. `talos`/`version`/`images`, is silently dropped — accepted but inert). Same structure as the `kubernetes` application's `nodeGroups`. When left empty, the module itself materialises a default scale-from-zero `md0` pool with `roles: [ingress-nginx]` (the wrapped `kubernetes` chart no longer injects one after the Phase 2 split), so the cluster sits idle until a pending Pod (e.g. the ingress-nginx controller) triggers the cluster-autoscaler. Supplied groups are authoritative (not merged with the default), so at least one of them must carry `roles: [ingress-nginx]` — enforced at render time. Each supplied pool key must be at most 15 characters so its `kubernetes-nodes-computeplane-cluster-<pool>` HelmRelease name fits Helm's 53-character limit — also enforced at render time.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskSize","instanceType","maxReplicas","minReplicas"],"properties":{"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"kubelet":{"description":"Kubelet resource reservations for this node group.","type":"object","properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"maxReplicas":{"description":"Maximum number of replicas.","type":"integer","default":10},"maxUnavailable":{"description":"Workers a rollout may remove before their replacement is up. Unset, the `kubernetes-nodes` pool chart uses 0, so each replacement joins before an old worker is drained (1 on a pool with `gpus`). Set 1 on a pool without room for one extra VM, whose rollout would otherwise stall with the new Machine Pending.","type":"integer","maximum":2147483647,"minimum":0},"minReplicas":{"description":"Minimum number of replicas.","type":"integer","default":0},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Set both `cpu` and `memory` together or neither.","type":"object","properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"roles":{"description":"List of node roles. At least one group must carry `ingress-nginx` — the module pins the ingress-nginx addon on, and its controller only schedules onto nodes with that role (enforced at render time).","type":"array","items":{"type":"string"}},"storageClass":{"description":"StorageClass for worker node persistent disks. When empty, the `kubernetes-nodes` pool chart applies its own default (`replicated`).","type":"string","x-cozystack-options":{"source":"storageclass"}}}}}}}
|
||||
release:
|
||||
prefix: ""
|
||||
labels:
|
||||
|
||||
@@ -21,7 +21,7 @@ spec:
|
||||
singular: kubernetesnodes
|
||||
plural: kubernetesnodeses
|
||||
openAPISchema: |-
|
||||
{"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool's objects are named `kubernetes-<cluster>-<pool>`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `iLine truncated
|
||||
{"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool's objects are named `kubernetes-<cluster>-<pool>`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `iLine truncated
|
||||
release:
|
||||
labels:
|
||||
sharding.fluxcd.io/key: tenants
|
||||
|
||||
Reference in new issue
Block a user