mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 22:13:18 +09:00
feat(kubernetes): let a node group set the CPU limit of its worker VMs (#3859)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Adds `nodeGroups.<name>.podCpuLimit` to the kubernetes chart. It renders into `domain.resources.limits.cpu` on the worker VM, which is the CPU ceiling of the virt-launcher `compute` container. The field has no default, so a cluster that does not set it renders exactly what it rendered before. The knob exists because there is currently no way to give a worker VM a ceiling different from its vCPU count. `nodeGroups.<name>.resources.cpu` sizes the guest, not the pod. KubeVirt fills the pod side in: when the namespace carries a ResourceQuota that limits CPU, it sets `limits.cpu` to the guest vCPU count, and a tenant namespace with `resourceQuotas` set always does. So a worker sized at N vCPUs runs under a ceiling of N, and the QEMU emulator thread and the IO threads come out of the same N as the vCPU threads. Declaring the limit on the VM is the only clean way out: KubeVirt stops deriving one as soon as the VMI carries it (`doesVMIRequireAutoResourceLimits` returns early), and the value declared on the VM lands on the container. Removing `cpu` from the tenant's `resourceQuotas` instead would be worse, because the tenant LimitRange then applies its 250m default. The field moves the ceiling and nothing else. It renders no CPU request, so the request stays what KubeVirt computes from the vCPU count and the cluster CPU allocation ratio, and the scheduler weight of a worker does not change with it. That is deliberate: a knob that moved both would make its own effect unreadable. It cannot be combined with `instanceType` sizing, and the chart says so at render time rather than letting the failure land later. KubeVirt's instancetype applier reports `domain.resources.limits.cpu` as a conflict instead of resolving it, so a VM that carries both an instancetype and the limit is refused by the admission webhook, which leaves a node group with no Machine and a Kubernetes CR that explains nothing. Sizing a group by `resources` is exactly what drops the instancetype from the VM, so that is the combination the limit is usable in, and a group still sized by its `instanceType` now fails the render with a message naming the group. This sits beside the chart's existing guard for the neighbouring `resources.cpu` without `resources.memory` mistake, which is the same conflict class one field over. Rendering is byte-identical for a node group that sets nothing, down to the whitespace, and that is load-bearing rather than tidy. The `KubevirtMachineTemplate` around this block is named by a hash of its content, so a stray key or a moved space would rename the template for every node group in every cluster and roll every live worker VM on upgrade. The existing hash pins in `cluster_test.yaml` cover it: they stay green, and they go red under a mutation that renders the limit unconditionally. The e2e suite sets the limit to 3 for its two-vCPU workers. This is an experiment aimed at a measured mechanism, not a claimed fix for cozystack/cozystack#3513. What a per-thread capture on a red run showed: the two vCPU threads of a stuck worker burn 1.6 to 1.8 cores between them under a ceiling of exactly 2.0 while the guest console records no progress at all, and the emulator, IO, vhost and vnc threads together stay under half a percent of the sampling interval. That is the shape of a vCPU spinning on a lock whose holder is the other vCPU, preempted out of the quota the two share, and it predicts that a ceiling with room above the vCPU count lets the preempted one back in. If the node-join failure rate does not move, the prediction was wrong and the mechanism needs a different lever; the chart field is useful either way, since a ceiling equal to the vCPU count is not a sensible default for a nested guest that pays emulation overhead out of it. Tests are helm-unittest in the same package: the limit lands in `domain.resources.limits.cpu`, a fractional quantity survives, no CPU request appears beside it, the ephemeral-storage pair the block already carried is untouched on both sides, nothing renders when the field is absent, and a group that keeps its instancetype fails the render on the message that names the reason. The `kubernetes-nodes` chart gets the same field, its guard and its own tests, because that chart is the other half of a byte-identity contract rather than a separate feature surface. Its `tests/render-parity.sh` renders a pool through both charts and compares the four pool objects byte for byte, on the grounds that the `KubevirtMachineTemplate` is content-hash named. A field only one side knows would not fail that gate, it would pass it and then quietly drop the ceiling when a pool is adopted into a split-out release. The gate gains a case that sets the limit and compares the two renders, and that case goes red if either side stops rendering it. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not a UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> Walked against the diff. One repository is reached: `terraform-provider-cozystack` models every app field by hand, and a new field in `packages/apps/kubernetes/values.schema.json` needs the schema, the model and the expand/flatten pair over there, the same way `logSerialConsole` did in cozystack/terraform-provider-cozystack#26 and `kernelModules` did in cozystack/terraform-provider-cozystack#20. The follow-up is filed as cozystack/terraform-provider-cozystack#32. The website is not reached: its trigger is adding, renaming or removing a package, and the reference page for an existing package is regenerated from its `README.md` by the release bot. The remaining repositories restate nothing this diff touches. The change to `hack/e2e-chainsaw/_lib/run-kubernetes.sh` is a value inside the suite, not a move, a rename or a make target, so `ccp` is untouched. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: https://github.com/cozystack/terraform-provider-cozystack/issues/32 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(kubernetes): a node group can now set `podCpuLimit`, the CPU ceiling of its worker VMs, independently of the vCPU count that sizes the guest. Without it KubeVirt keeps deriving that ceiling from the vCPU count, which leaves the QEMU emulator and IO threads sharing the guest's own allowance. Unset by default, so existing clusters render and behave exactly as before. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional `podCpuLimit` configuration for setting CPU limits on worker VM containers. * Supports integer and fractional Kubernetes CPU quantities. * Preserves CPU requests and ephemeral-storage settings while applying the configured limit. * Worker changes trigger VM template rollouts. * **Bug Fixes** * Added validation for incompatible combinations with instance-type sizing and insufficient resource configuration. * **Documentation** * Updated configuration references and schemas with usage, sizing, scheduling, and rollout guidance. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
18 files changed
+486
-53
No files matched your search
Generated
+2
@@ -286,6 +286,8 @@ type NodeGroup struct {
|
||||
MinReplicas int `json:"minReplicas"`
|
||||
// Per-group override for `nodeHealthCheck.nodeStartupTimeout`. When unset, the cluster-wide `nodeHealthCheck.nodeStartupTimeout` applies.
|
||||
NodeStartupTimeout string `json:"nodeStartupTimeout,omitempty"`
|
||||
// CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the group by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one; the CPU request is left alone at the vCPU-derived value, so this changes the ceiling and not the scheduler weight. Changing it rolls the node group, because the worker VM template is named by a hash of its content. There is a floor: KubeVirt still supplies the request, so a value below the guest vCPU count divided by the cluster CPU allocation ratio leaves the container with a request above its limit, and the worker's Pod is then rejected on admission rather than at render time.
|
||||
PodCpuLimit resource.Quantity `json:"podCpuLimit,omitempty"`
|
||||
// Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored for that node group (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time.
|
||||
Resources Resources `json:"resources,omitempty"`
|
||||
// List of node roles.
|
||||
|
||||
@@ -410,6 +410,7 @@ func (in *NodeGroup) DeepCopyInto(out *NodeGroup) {
|
||||
copy(*out, *in)
|
||||
}
|
||||
out.Kubelet = in.Kubelet
|
||||
out.PodCpuLimit = in.PodCpuLimit.DeepCopy()
|
||||
in.Resources.DeepCopyInto(&out.Resources)
|
||||
if in.Roles != nil {
|
||||
in, out := &in.Roles, &out.Roles
|
||||
|
||||
+2
@@ -42,6 +42,8 @@ type ConfigSpec struct {
|
||||
// Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time.
|
||||
// +kubebuilder:default:={}
|
||||
Resources Resources `json:"resources,omitempty"`
|
||||
// CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one; the CPU request is left alone at the vCPU-derived value, so this changes the ceiling and not the scheduler weight. Changing it rolls the pool, because the worker VM template is named by a hash of its content. There is a floor: KubeVirt still supplies the request, so a value below the guest vCPU count divided by the cluster CPU allocation ratio leaves the container with a request above its limit, and the worker's Pod is then rejected on admission rather than at render time. Keep in sync with the parent kubernetes chart.
|
||||
PodCpuLimit resource.Quantity `json:"podCpuLimit,omitempty"`
|
||||
// List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
|
||||
// +kubebuilder:default:={}
|
||||
Gpus []GPU `json:"gpus,omitempty"`
|
||||
|
||||
@@ -60,6 +60,7 @@ func (in *ConfigSpec) DeepCopyInto(out *ConfigSpec) {
|
||||
copy(*out, *in)
|
||||
}
|
||||
in.Resources.DeepCopyInto(&out.Resources)
|
||||
out.PodCpuLimit = in.PodCpuLimit.DeepCopy()
|
||||
if in.Gpus != nil {
|
||||
in, out := &in.Gpus, &out.Gpus
|
||||
*out = make([]GPU, len(*in))
|
||||
|
||||
@@ -3163,6 +3163,16 @@ ${ouroboros_addon}
|
||||
logSerialConsole: true
|
||||
maxReplicas: 10
|
||||
minReplicas: 2
|
||||
# Headroom over the vCPU count, where KubeVirt would otherwise put the
|
||||
# ceiling. Workers failing to join (cozystack/cozystack#3513) have been
|
||||
# measured burning their vCPU threads flat out at a ceiling equal to
|
||||
# their vCPU count while the guest kernel made no progress, which is what
|
||||
# a vCPU spinning on a lock looks like when the vCPU holding that lock is
|
||||
# the one preempted out of the quota they share. Whether the spare CPU
|
||||
# lets the preempted one back in is what this value is here to answer:
|
||||
# measured mechanism, predicted remedy. If the join failure rate does not
|
||||
# move, the lever is elsewhere and this goes back to the vCPU count.
|
||||
podCpuLimit: 3
|
||||
# Two vCPUs at the memory the workers had before. Sized by resources
|
||||
# rather than by instanceType: u1.large doubles memory along with the
|
||||
# vCPUs, and four 8Gi workers do not fit beside the rest of the suite,
|
||||
|
||||
@@ -29,28 +29,29 @@ the parent cluster `kubernetes-<cluster>` in the same namespace, plus a
|
||||
|
||||
### Pool Sizing
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------- |
|
||||
| `minReplicas` | Minimum number of replicas in this pool. Used as the cluster-autoscaler floor. | `int` | `0` |
|
||||
| `maxReplicas` | Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling. | `int` | `10` |
|
||||
| `instanceType` | Virtual machine instance type. | `string` | `u1.medium` |
|
||||
| `diskSize` | System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs. | `quantity` | `20Gi` |
|
||||
| `roles` | List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller. | `[]string` | `[]` |
|
||||
| `resources` | Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. | `object` | `{}` |
|
||||
| `resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
|
||||
| `gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` |
|
||||
| `kubelet` | Kubelet resource reservations for this pool. | `object` | `{}` |
|
||||
| `kubelet.systemReservedMemory` | Memory reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.kubeReservedMemory` | Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.systemReservedCpu` | CPU reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` |
|
||||
| `kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` |
|
||||
| `logSerialConsole` | Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it. | `bool` | `false` |
|
||||
| `maxUnhealthy` | Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. | `string` | `50%` |
|
||||
| `nodeStartupTimeout` | Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). | `string` | `10m` |
|
||||
| Name | Description | Type | Value |
|
||||
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------- |
|
||||
| `minReplicas` | Minimum number of replicas in this pool. Used as the cluster-autoscaler floor. | `int` | `0` |
|
||||
| `maxReplicas` | Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling. | `int` | `10` |
|
||||
| `instanceType` | Virtual machine instance type. | `string` | `u1.medium` |
|
||||
| `diskSize` | System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs. | `quantity` | `20Gi` |
|
||||
| `roles` | List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller. | `[]string` | `[]` |
|
||||
| `resources` | Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. | `object` | `{}` |
|
||||
| `resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `podCpuLimit` | CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one; the CPU request is left alone at the vCPU-derived value, so this changes the ceiling and not the scheduler weight. Changing it rolls the pool, because the worker VM template is named by a hash of its content. There is a floor: KubeVirt still supplies the request, so a value below the guest vCPU count divided by the cluster CPU allocation ratio leaves the container with a request above its limit, and the worker's Pod is then rejected on admission rather than at render time. Keep in sync with the parent kubernetes chart. | `quantity` | `""` |
|
||||
| `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
|
||||
| `gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` |
|
||||
| `kubelet` | Kubelet resource reservations for this pool. | `object` | `{}` |
|
||||
| `kubelet.systemReservedMemory` | Memory reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.kubeReservedMemory` | Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.systemReservedCpu` | CPU reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` |
|
||||
| `kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` |
|
||||
| `logSerialConsole` | Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it. | `bool` | `false` |
|
||||
| `maxUnhealthy` | Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. | `string` | `50%` |
|
||||
| `nodeStartupTimeout` | Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). | `string` | `10m` |
|
||||
|
||||
|
||||
### Kubernetes Version
|
||||
|
||||
@@ -42,6 +42,7 @@
|
||||
"gpus" .Values.gpus
|
||||
"kubelet" .Values.kubelet
|
||||
"logSerialConsole" .Values.logSerialConsole
|
||||
"podCpuLimit" .Values.podCpuLimit
|
||||
}}
|
||||
{{- define "kubevirtmachinetemplate" -}}
|
||||
spec:
|
||||
@@ -166,7 +167,21 @@ spec:
|
||||
{{- end }}
|
||||
resources:
|
||||
{{/* 2Gi is enough: persistent DataVolumes are not counted as pod ephemeral storage */}}
|
||||
{{- include "cozy-lib.resources.sanitize" (list (dict "ephemeral-storage" "2Gi") $) | nindent 14 }}
|
||||
{{- $podResources := include "cozy-lib.resources.sanitize" (list (dict "ephemeral-storage" "2Gi") $) }}
|
||||
{{- /* The limit alone, and only for a pool that asked for one.
|
||||
KubeVirt derives both sides from the guest vCPU count, and a
|
||||
limit declared here is what stops it deriving that one; a
|
||||
request declared here would override the vCPU-derived one
|
||||
and move the scheduler weight along with the ceiling. The
|
||||
enclosing KubevirtMachineTemplate is content-hash named, so
|
||||
a pool that sets nothing has to render the same bytes it
|
||||
did before this field existed, down to the whitespace. */}}
|
||||
{{- with .group.podCpuLimit }}
|
||||
{{- $limited := $podResources | fromYaml }}
|
||||
{{- $_ := set $limited.limits "cpu" (toString .) }}
|
||||
{{- $podResources = toYaml $limited }}
|
||||
{{- end }}
|
||||
{{- $podResources | nindent 14 }}
|
||||
evictionStrategy: External
|
||||
volumes:
|
||||
- name: system
|
||||
@@ -191,6 +206,15 @@ spec:
|
||||
{{- if and $group.instanceType (or (and $hasCpu (not $hasMem)) (and $hasMem (not $hasCpu))) }}
|
||||
{{- fail (printf "nodeGroup %s: set both resources.cpu and resources.memory, or neither" $groupName) }}
|
||||
{{- end }}
|
||||
{{- /* Same conflict class, other field: podCpuLimit renders
|
||||
domain.resources.limits.cpu, and a VM that carries an instancetype and
|
||||
that key is refused by KubeVirt's instancetype applier, which reports it
|
||||
as a conflict rather than resolving it. The pool has to be sized by
|
||||
resources for the limit to be usable at all, since that is what drops
|
||||
the instancetype from the VM. */}}
|
||||
{{- if and $group.podCpuLimit $group.instanceType (not (and $hasCpu $hasMem)) }}
|
||||
{{- fail (printf "nodeGroup %s: podCpuLimit cannot be combined with instanceType sizing, because KubeVirt rejects a VirtualMachine that references an instancetype and also sets domain.resources.limits.cpu. Set both resources.cpu and resources.memory, which drops the instancetype from the VM, or drop podCpuLimit." $groupName) }}
|
||||
{{- end }}
|
||||
{{/* Resolve instanceType once per group — reused for kubelet reservations and capacity annotations. */}}
|
||||
{{- $instanceType := dict }}
|
||||
{{- if and $group.instanceType (not (and $hasCpu $hasMem)) }}
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
suite: worker pool pod CPU limit
|
||||
|
||||
# Mirror of the parent kubernetes chart's suite, on the pool-shaped values of
|
||||
# this chart. `podCpuLimit` is the only knob that sets
|
||||
# `domain.resources.limits.cpu` on a worker VM. KubeVirt otherwise derives that
|
||||
# limit from the guest vCPU count whenever the namespace carries a ResourceQuota
|
||||
# that limits CPU, and it stops deriving as soon as the VMI declares the limit
|
||||
# itself. The request stays whatever KubeVirt computes from the vCPU count and
|
||||
# the cluster CPU allocation ratio: this field moves the ceiling alone.
|
||||
|
||||
templates:
|
||||
- templates/nodegroup.yaml
|
||||
release:
|
||||
name: kubernetes-nodes-myk8s-md0
|
||||
namespace: tenant-test
|
||||
set:
|
||||
cluster: myk8s
|
||||
_cluster:
|
||||
cluster-domain: cozy.local
|
||||
version: "v1.35"
|
||||
minReplicas: 0
|
||||
maxReplicas: 3
|
||||
instanceType: ""
|
||||
diskSize: 20Gi
|
||||
storageClass: replicated
|
||||
roles:
|
||||
- ingress-nginx
|
||||
# Sizing is set per case rather than here, because helm-unittest merges a
|
||||
# case's `set` into these values and a case that needs an unsized pool cannot
|
||||
# take cpu/memory back out again.
|
||||
tests:
|
||||
- it: renders the CPU limit the pool asks for onto the worker VM
|
||||
set:
|
||||
resources:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
podCpuLimit: 3
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: KubevirtMachineTemplate
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.cpu
|
||||
value: "3"
|
||||
# One dimension only: a rendered CPU request would override KubeVirt's
|
||||
# vCPU-derived one and move the scheduler weight along with the ceiling.
|
||||
- notExists:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.requests.cpu
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.ephemeral-storage
|
||||
value: "2Gi"
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.requests.ephemeral-storage
|
||||
value: "53687091"
|
||||
|
||||
- it: accepts a fractional quantity
|
||||
set:
|
||||
resources:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
podCpuLimit: 1500m
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: KubevirtMachineTemplate
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.cpu
|
||||
value: "1500m"
|
||||
|
||||
- it: leaves the limit to KubeVirt when the pool sets none
|
||||
set:
|
||||
resources:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: KubevirtMachineTemplate
|
||||
asserts:
|
||||
# Absent, not empty: the KubevirtMachineTemplate is content-hash named, so
|
||||
# a `cpu:` key rendered for a pool that asked for nothing would rename the
|
||||
# template and roll every live worker VM of the pool on upgrade. The
|
||||
# render-parity gate compares those bytes against the parent chart.
|
||||
- notExists:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.cpu
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.ephemeral-storage
|
||||
value: "2Gi"
|
||||
|
||||
# KubeVirt's instancetype applier reports domain.resources.limits.cpu as a
|
||||
# conflict instead of resolving it, so a VM carrying both is refused at
|
||||
# admission with no Machine and no explanation on the CR. Sizing a pool by
|
||||
# resources is what drops the instancetype from the VM, which is why the limit
|
||||
# is only usable there.
|
||||
- it: refuses the limit on a pool still sized by its instancetype
|
||||
set:
|
||||
instanceType: u1.medium
|
||||
podCpuLimit: 3
|
||||
asserts:
|
||||
# Matched on the message, not on "some render error": a pool carrying half
|
||||
# a size trips the neighbouring resources.cpu / resources.memory guard
|
||||
# first, so a bare failedTemplate here would pass on the wrong error.
|
||||
- failedTemplate:
|
||||
errorPattern: 'nodeGroup md0: podCpuLimit cannot be combined with instanceType sizing'
|
||||
@@ -164,8 +164,24 @@ kubelet: {}
|
||||
logSerialConsole: true
|
||||
EOF
|
||||
|
||||
# --- Case: explicit pod CPU limit ---
|
||||
cat >"$WORK/case-podcpulimit.yaml" <<'EOF'
|
||||
minReplicas: 0
|
||||
maxReplicas: 3
|
||||
instanceType: ""
|
||||
diskSize: 20Gi
|
||||
storageClass: replicated
|
||||
roles: []
|
||||
resources:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
gpus: []
|
||||
kubelet: {}
|
||||
podCpuLimit: 3
|
||||
EOF
|
||||
|
||||
RC=0
|
||||
for c in resources gpu kubelet serialconsole; do
|
||||
for c in resources gpu kubelet serialconsole podcpulimit; do
|
||||
write_values "$WORK/case-${c}.yaml"
|
||||
diff_kinds "$c" || RC=1
|
||||
done
|
||||
|
||||
@@ -88,6 +88,19 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"podCpuLimit": {
|
||||
"description": "CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one; the CPU request is left alone at the vCPU-derived value, so this changes the ceiling and not the scheduler weight. Changing it rolls the pool, because the worker VM template is named by a hash of its content. There is a floor: KubeVirt still supplies the request, so a value below the guest vCPU count divided by the cluster CPU allocation ratio leaves the container with a request above its limit, and the worker's Pod is then rejected on admission rather than at render time. Keep in sync with the parent kubernetes chart.",
|
||||
"pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$",
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "integer"
|
||||
},
|
||||
{
|
||||
"type": "string"
|
||||
}
|
||||
],
|
||||
"x-kubernetes-int-or-string": true
|
||||
},
|
||||
"gpus": {
|
||||
"description": "List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).",
|
||||
"type": "array",
|
||||
|
||||
@@ -52,6 +52,8 @@ roles: []
|
||||
## @param {Resources} [resources] - Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time.
|
||||
resources: {}
|
||||
|
||||
## @param {quantity} [podCpuLimit] - CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one; the CPU request is left alone at the vCPU-derived value, so this changes the ceiling and not the scheduler weight. Changing it rolls the pool, because the worker VM template is named by a hash of its content. There is a floor: KubeVirt still supplies the request, so a value below the guest vCPU count divided by the cluster CPU allocation ratio leaves the container with a request above its limit, and the worker's Pod is then rejected on admission rather than at render time. Keep in sync with the parent kubernetes chart.
|
||||
|
||||
## @param {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
|
||||
gpus: []
|
||||
|
||||
|
||||
@@ -108,32 +108,33 @@ See the reference for components utilized in this service:
|
||||
|
||||
### Application-specific Parameters
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ----------- |
|
||||
| `nodeGroups` | Worker nodes configuration map. When left empty, a default node group `md0` is emitted with `minReplicas: 0` and `roles: [ingress-nginx]` — the MachineDeployment renders but provisions no workers until an unschedulable Pod triggers the cluster-autoscaler (or an operator scales the group manually). Enabling `addons.ingressNginx.enabled: true` on a CR with default `nodeGroups: {}` therefore requires either supplying a nodeGroup with `roles: [ingress-nginx]` and `minReplicas >= 1`, or waiting for the autoscaler to bring up the default md0 in response to the ingress-nginx controller Pods becoming Pending. Provide your own groups to take full control — they are not merged with the default, so you may name and omit groups freely. | `map[string]object` | `{}` |
|
||||
| `nodeGroups[name].minReplicas` | Minimum number of replicas. | `int` | `0` |
|
||||
| `nodeGroups[name].maxReplicas` | Maximum number of replicas. | `int` | `10` |
|
||||
| `nodeGroups[name].instanceType` | Virtual machine instance type. | `string` | `u1.medium` |
|
||||
| `nodeGroups[name].diskSize` | System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs. Pre-Talos installs used a separate disk-kubelet PVC for kubelet/containerd state; on Talos this is consolidated onto the single system disk imaged from the factory artifact. | `quantity` | `20Gi` |
|
||||
| `nodeGroups[name].storageClass` | StorageClass for worker node persistent disks. When empty, falls back to the application-level storageClass. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile, not set on the DataVolume here — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so the fallback targets the replicated/DRBD application storageClass rather than a possibly non-DRBD cluster default. NOTE: deliberately not marked immutable — the field is optional and undefaulted, so a strict `self == oldSelf` rule would block any future attempt to set it on an existing node group. | `string` | `""` |
|
||||
| `nodeGroups[name].roles` | List of node roles. | `[]string` | `[]` |
|
||||
| `nodeGroups[name].resources` | Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored for that node group (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. | `object` | `{}` |
|
||||
| `nodeGroups[name].resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `nodeGroups[name].resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `nodeGroups[name].gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
|
||||
| `nodeGroups[name].gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` |
|
||||
| `nodeGroups[name].kubelet` | Kubelet resource reservations for this node group. | `object` | `{}` |
|
||||
| `nodeGroups[name].kubelet.systemReservedMemory` | Memory reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `nodeGroups[name].kubelet.kubeReservedMemory` | Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `nodeGroups[name].kubelet.systemReservedCpu` | CPU reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `nodeGroups[name].kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `nodeGroups[name].kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` |
|
||||
| `nodeGroups[name].kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` |
|
||||
| `nodeGroups[name].logSerialConsole` | Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the node group, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the node group down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it. | `bool` | `false` |
|
||||
| `nodeGroups[name].maxUnhealthy` | Per-group override for `nodeHealthCheck.maxUnhealthy`. When unset, the cluster-wide `nodeHealthCheck.maxUnhealthy` applies. Accepts a bare integer ("0", "1", ...) or an integer percentage ("0%", "50%"). | `string` | `""` |
|
||||
| `nodeGroups[name].nodeStartupTimeout` | Per-group override for `nodeHealthCheck.nodeStartupTimeout`. When unset, the cluster-wide `nodeHealthCheck.nodeStartupTimeout` applies. | `string` | `""` |
|
||||
| `version` | Kubernetes major.minor version to deploy | `string` | `v1.35` |
|
||||
| `host` | External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty. | `string` | `""` |
|
||||
| Name | Description | Type | Value |
|
||||
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ----------- |
|
||||
| `nodeGroups` | Worker nodes configuration map. When left empty, a default node group `md0` is emitted with `minReplicas: 0` and `roles: [ingress-nginx]` — the MachineDeployment renders but provisions no workers until an unschedulable Pod triggers the cluster-autoscaler (or an operator scales the group manually). Enabling `addons.ingressNginx.enabled: true` on a CR with default `nodeGroups: {}` therefore requires either supplying a nodeGroup with `roles: [ingress-nginx]` and `minReplicas >= 1`, or waiting for the autoscaler to bring up the default md0 in response to the ingress-nginx controller Pods becoming Pending. Provide your own groups to take full control — they are not merged with the default, so you may name and omit groups freely. | `map[string]object` | `{}` |
|
||||
| `nodeGroups[name].minReplicas` | Minimum number of replicas. | `int` | `0` |
|
||||
| `nodeGroups[name].maxReplicas` | Maximum number of replicas. | `int` | `10` |
|
||||
| `nodeGroups[name].instanceType` | Virtual machine instance type. | `string` | `u1.medium` |
|
||||
| `nodeGroups[name].diskSize` | System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs. Pre-Talos installs used a separate disk-kubelet PVC for kubelet/containerd state; on Talos this is consolidated onto the single system disk imaged from the factory artifact. | `quantity` | `20Gi` |
|
||||
| `nodeGroups[name].storageClass` | StorageClass for worker node persistent disks. When empty, falls back to the application-level storageClass. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile, not set on the DataVolume here — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so the fallback targets the replicated/DRBD application storageClass rather than a possibly non-DRBD cluster default. NOTE: deliberately not marked immutable — the field is optional and undefaulted, so a strict `self == oldSelf` rule would block any future attempt to set it on an existing node group. | `string` | `""` |
|
||||
| `nodeGroups[name].roles` | List of node roles. | `[]string` | `[]` |
|
||||
| `nodeGroups[name].resources` | Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored for that node group (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. | `object` | `{}` |
|
||||
| `nodeGroups[name].resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `nodeGroups[name].resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `nodeGroups[name].podCpuLimit` | CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the group by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one; the CPU request is left alone at the vCPU-derived value, so this changes the ceiling and not the scheduler weight. Changing it rolls the node group, because the worker VM template is named by a hash of its content. There is a floor: KubeVirt still supplies the request, so a value below the guest vCPU count divided by the cluster CPU allocation ratio leaves the container with a request above its limit, and the worker's Pod is then rejected on admission rather than at render time. | `quantity` | `""` |
|
||||
| `nodeGroups[name].gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
|
||||
| `nodeGroups[name].gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` |
|
||||
| `nodeGroups[name].kubelet` | Kubelet resource reservations for this node group. | `object` | `{}` |
|
||||
| `nodeGroups[name].kubelet.systemReservedMemory` | Memory reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `nodeGroups[name].kubelet.kubeReservedMemory` | Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `nodeGroups[name].kubelet.systemReservedCpu` | CPU reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `nodeGroups[name].kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `nodeGroups[name].kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` |
|
||||
| `nodeGroups[name].kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` |
|
||||
| `nodeGroups[name].logSerialConsole` | Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the node group, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the node group down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it. | `bool` | `false` |
|
||||
| `nodeGroups[name].maxUnhealthy` | Per-group override for `nodeHealthCheck.maxUnhealthy`. When unset, the cluster-wide `nodeHealthCheck.maxUnhealthy` applies. Accepts a bare integer ("0", "1", ...) or an integer percentage ("0%", "50%"). | `string` | `""` |
|
||||
| `nodeGroups[name].nodeStartupTimeout` | Per-group override for `nodeHealthCheck.nodeStartupTimeout`. When unset, the cluster-wide `nodeHealthCheck.nodeStartupTimeout` applies. | `string` | `""` |
|
||||
| `version` | Kubernetes major.minor version to deploy | `string` | `v1.35` |
|
||||
| `host` | External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty. | `string` | `""` |
|
||||
|
||||
|
||||
### Cluster Addons
|
||||
|
||||
@@ -187,7 +187,21 @@ spec:
|
||||
{{- end }}
|
||||
resources:
|
||||
{{/* 2Gi is enough: persistent DataVolumes are not counted as pod ephemeral storage */}}
|
||||
{{- include "cozy-lib.resources.sanitize" (list (dict "ephemeral-storage" "2Gi") $) | nindent 14 }}
|
||||
{{- $podResources := include "cozy-lib.resources.sanitize" (list (dict "ephemeral-storage" "2Gi") $) }}
|
||||
{{- /* The limit alone, and only for a group that asked for one.
|
||||
KubeVirt derives both sides from the guest vCPU count, and a
|
||||
limit declared here is what stops it deriving that one; a
|
||||
request declared here would override the vCPU-derived one
|
||||
and move the scheduler weight along with the ceiling. The
|
||||
enclosing KubevirtMachineTemplate is content-hash named, so
|
||||
a group that sets nothing has to render the same bytes it
|
||||
did before this field existed, down to the whitespace. */}}
|
||||
{{- with .group.podCpuLimit }}
|
||||
{{- $limited := $podResources | fromYaml }}
|
||||
{{- $_ := set $limited.limits "cpu" (toString .) }}
|
||||
{{- $podResources = toYaml $limited }}
|
||||
{{- end }}
|
||||
{{- $podResources | nindent 14 }}
|
||||
evictionStrategy: External
|
||||
volumes:
|
||||
- name: system
|
||||
@@ -533,6 +547,15 @@ metadata:
|
||||
{{- if and $group.instanceType (or (and $hasCpu (not $hasMem)) (and $hasMem (not $hasCpu))) }}
|
||||
{{- fail (printf "nodeGroup %s: set both resources.cpu and resources.memory, or neither" $groupName) }}
|
||||
{{- end }}
|
||||
{{- /* Same conflict class, other field: podCpuLimit renders
|
||||
domain.resources.limits.cpu, and a VM that carries an instancetype and
|
||||
that key is refused by KubeVirt's instancetype applier, which reports it
|
||||
as a conflict rather than resolving it. The group has to be sized by
|
||||
resources for the limit to be usable at all, since that is what drops
|
||||
the instancetype from the VM. */}}
|
||||
{{- if and $group.podCpuLimit $group.instanceType (not (and $hasCpu $hasMem)) }}
|
||||
{{- fail (printf "nodeGroup %s: podCpuLimit cannot be combined with instanceType sizing, because KubeVirt rejects a VirtualMachine that references an instancetype and also sets domain.resources.limits.cpu. Set both resources.cpu and resources.memory, which drops the instancetype from the VM, or drop podCpuLimit." $groupName) }}
|
||||
{{- end }}
|
||||
{{/* Resolve instanceType once per group — reused for kubelet reservations and capacity annotations. */}}
|
||||
{{- $instanceType := dict }}
|
||||
{{- if and $group.instanceType (not (and $hasCpu $hasMem)) }}
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
suite: worker pod CPU limit
|
||||
|
||||
# `podCpuLimit` is the only knob that sets `domain.resources.limits.cpu` on a
|
||||
# worker VM. KubeVirt otherwise derives that limit from the guest vCPU count
|
||||
# whenever the namespace carries a ResourceQuota that limits CPU, and it stops
|
||||
# deriving as soon as the VMI declares the limit itself, so the value rendered
|
||||
# here is what the compute container gets. The request stays whatever KubeVirt
|
||||
# computes from the vCPU count and the cluster CPU allocation ratio: this field
|
||||
# moves the ceiling alone.
|
||||
|
||||
templates:
|
||||
- templates/cluster.yaml
|
||||
|
||||
kubernetesProvider:
|
||||
scheme:
|
||||
"instancetype.kubevirt.io/v1beta1/VirtualMachineClusterInstancetype":
|
||||
gvr:
|
||||
group: instancetype.kubevirt.io
|
||||
version: v1beta1
|
||||
resource: virtualmachineclusterinstancetypes
|
||||
namespaced: false
|
||||
"cluster.x-k8s.io/v1beta1/MachineSet":
|
||||
gvr:
|
||||
group: cluster.x-k8s.io
|
||||
version: v1beta1
|
||||
resource: machinesets
|
||||
namespaced: true
|
||||
"infrastructure.cluster.x-k8s.io/v1alpha1/KubevirtMachineTemplate":
|
||||
gvr:
|
||||
group: infrastructure.cluster.x-k8s.io
|
||||
version: v1alpha1
|
||||
resource: kubevirtmachinetemplates
|
||||
namespaced: true
|
||||
objects:
|
||||
- kind: VirtualMachineClusterInstancetype
|
||||
apiVersion: instancetype.kubevirt.io/v1beta1
|
||||
metadata:
|
||||
name: u1.medium
|
||||
- kind: MachineSet
|
||||
apiVersion: cluster.x-k8s.io/v1beta1
|
||||
metadata:
|
||||
name: ignored-ms
|
||||
namespace: tenant-test
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
infrastructureRef:
|
||||
kind: Other
|
||||
name: ignored
|
||||
- kind: KubevirtMachineTemplate
|
||||
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
|
||||
metadata:
|
||||
name: ignored-kmt
|
||||
namespace: tenant-test
|
||||
|
||||
tests:
|
||||
- it: renders the CPU limit a node group asks for onto the worker VM
|
||||
release:
|
||||
name: test-k8s
|
||||
namespace: tenant-test
|
||||
set:
|
||||
_namespace:
|
||||
etcd: etcd
|
||||
ingress: nginx
|
||||
host: example.com
|
||||
_cluster:
|
||||
cluster-domain: cozy.local
|
||||
nodeGroups:
|
||||
worker0:
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: "u1.medium"
|
||||
diskSize: 20Gi
|
||||
roles:
|
||||
- worker
|
||||
resources:
|
||||
cpu: 2
|
||||
memory: 4Gi
|
||||
podCpuLimit: 3
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.cpu
|
||||
value: "3"
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: KubevirtMachineTemplate
|
||||
# The knob is deliberately one-dimensional. A rendered CPU request would
|
||||
# override KubeVirt's vCPU-derived one and move the scheduler weight
|
||||
# along with the ceiling, which is a second variable this field does not
|
||||
# exist to change.
|
||||
- notExists:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.requests.cpu
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: KubevirtMachineTemplate
|
||||
# The ephemeral-storage pair the block carried before this field existed
|
||||
# survives on both sides, request still divided by the cluster
|
||||
# ephemeral-storage allocation ratio.
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.ephemeral-storage
|
||||
value: "2Gi"
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: KubevirtMachineTemplate
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.requests.ephemeral-storage
|
||||
value: "53687091"
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: KubevirtMachineTemplate
|
||||
|
||||
- it: accepts a fractional quantity
|
||||
release:
|
||||
name: test-k8s
|
||||
namespace: tenant-test
|
||||
set:
|
||||
_namespace:
|
||||
etcd: etcd
|
||||
ingress: nginx
|
||||
host: example.com
|
||||
_cluster:
|
||||
cluster-domain: cozy.local
|
||||
nodeGroups:
|
||||
worker0:
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: "u1.medium"
|
||||
diskSize: 20Gi
|
||||
roles:
|
||||
- worker
|
||||
resources:
|
||||
cpu: 1
|
||||
memory: 2Gi
|
||||
podCpuLimit: 1500m
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.cpu
|
||||
value: "1500m"
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: KubevirtMachineTemplate
|
||||
|
||||
# KubeVirt's instancetype applier reports domain.resources.limits.cpu as a
|
||||
# conflict instead of resolving it, so a VM carrying both is refused at
|
||||
# admission with no Machine and no explanation on the Kubernetes CR. Sizing a
|
||||
# group by resources is what drops the instancetype from the VM, which is why
|
||||
# the limit is only usable there. Failing at render time is the same trade the
|
||||
# neighbouring resources.cpu / resources.memory guard makes.
|
||||
- it: refuses the limit on a group still sized by its instancetype
|
||||
release:
|
||||
name: test-k8s
|
||||
namespace: tenant-test
|
||||
set:
|
||||
_namespace:
|
||||
etcd: etcd
|
||||
ingress: nginx
|
||||
host: example.com
|
||||
_cluster:
|
||||
cluster-domain: cozy.local
|
||||
nodeGroups:
|
||||
worker0:
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: "u1.medium"
|
||||
diskSize: 20Gi
|
||||
roles:
|
||||
- worker
|
||||
podCpuLimit: 3
|
||||
asserts:
|
||||
# Matched on the message, not on "some render error": a group carrying
|
||||
# half a size trips the neighbouring resources.cpu / resources.memory
|
||||
# guard first, so a bare failedTemplate here would pass on the wrong
|
||||
# error.
|
||||
- failedTemplate:
|
||||
errorPattern: 'nodeGroup worker0: podCpuLimit cannot be combined with instanceType sizing'
|
||||
|
||||
- it: leaves the limit to KubeVirt when the node group sets none
|
||||
release:
|
||||
name: test-k8s
|
||||
namespace: tenant-test
|
||||
set:
|
||||
_namespace:
|
||||
etcd: etcd
|
||||
ingress: nginx
|
||||
host: example.com
|
||||
_cluster:
|
||||
cluster-domain: cozy.local
|
||||
nodeGroups:
|
||||
worker0:
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: "u1.medium"
|
||||
diskSize: 20Gi
|
||||
roles:
|
||||
- worker
|
||||
resources:
|
||||
cpu: 2
|
||||
memory: 4Gi
|
||||
asserts:
|
||||
# Absent, not empty. The enclosing KubevirtMachineTemplate is named by a
|
||||
# hash of its content, so a `cpu:` key rendered for a group that asked
|
||||
# for nothing would rename the template and roll every live worker VM of
|
||||
# every cluster on upgrade.
|
||||
- notExists:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.cpu
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: KubevirtMachineTemplate
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.ephemeral-storage
|
||||
value: "2Gi"
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: KubevirtMachineTemplate
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.requests.ephemeral-storage
|
||||
value: "53687091"
|
||||
documentSelector:
|
||||
path: kind
|
||||
value: KubevirtMachineTemplate
|
||||
@@ -125,6 +125,19 @@
|
||||
"description": "Per-group override for `nodeHealthCheck.nodeStartupTimeout`. When unset, the cluster-wide `nodeHealthCheck.nodeStartupTimeout` applies.",
|
||||
"type": "string"
|
||||
},
|
||||
"podCpuLimit": {
|
||||
"description": "CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the group by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one; the CPU request is left alone at the vCPU-derived value, so this changes the ceiling and not the scheduler weight. Changing it rolls the node group, because the worker VM template is named by a hash of its content. There is a floor: KubeVirt still supplies the request, so a value below the guest vCPU count divided by the cluster CPU allocation ratio leaves the container with a request above its limit, and the worker's Pod is then rejected on admission rather than at render time.",
|
||||
"pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$",
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "integer"
|
||||
},
|
||||
{
|
||||
"type": "string"
|
||||
}
|
||||
],
|
||||
"x-kubernetes-int-or-string": true
|
||||
},
|
||||
"resources": {
|
||||
"description": "Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored for that node group (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time.",
|
||||
"type": "object",
|
||||
|
||||
@@ -86,6 +86,7 @@ storageClass: replicated
|
||||
## @x-cozystack-options {source: storageclass}
|
||||
## @field {[]string} roles - List of node roles.
|
||||
## @field {Resources} [resources] - Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored for that node group (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time.
|
||||
## @field {quantity} [podCpuLimit] - CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the group by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one; the CPU request is left alone at the vCPU-derived value, so this changes the ceiling and not the scheduler weight. Changing it rolls the node group, because the worker VM template is named by a hash of its content. There is a floor: KubeVirt still supplies the request, so a value below the guest vCPU count divided by the cluster CPU allocation ratio leaves the container with a request above its limit, and the worker's Pod is then rejected on admission rather than at render time.
|
||||
## @field {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
|
||||
## @field {Kubelet} [kubelet] - Kubelet resource reservations for this node group.
|
||||
## @field {bool} [logSerialConsole]=false - Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the node group, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the node group down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.
|
||||
|
||||
@@ -20,7 +20,7 @@ spec:
|
||||
singular: kubernetesnodes
|
||||
plural: kubernetesnodeses
|
||||
openAPISchema: |-
|
||||
{"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool name (the part after `<cluster>-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.","type":"string","default":""},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class.","type":"string","default":"replicated","x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"kubelet":{"description":"Kubelet resource reservations for this pool.","type":"object","default":{},"properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"logSerialConsole":{"description":"Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility intoLine truncated
|
||||
{"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool name (the part after `<cluster>-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.","type":"string","default":""},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class.","type":"string","default":"replicated","x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"podCpuLimit":{"description":"CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one; the CPU request is left alone at the vCPU-derived value, so this changes the ceiling and not the scheduler weight. Changing it rolls the pool, because the worker VM template is named by a hash of its content. There is a floor: KubeVirt still supplies the request, so a value below the guest vCPU count divided by the cluster CPU allocation ratio leaves the container with a request above its limit, and the worker's Pod is then rejected on admission rather than at render time. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"Line truncated
|
||||
release:
|
||||
labels:
|
||||
sharding.fluxcd.io/key: tenants
|
||||
|
||||
+1
-1
@@ -32,7 +32,7 @@ spec:
|
||||
singular: kubernetes
|
||||
plural: kuberneteses
|
||||
openAPISchema: |-
|
||||
{"title":"Chart Values","type":"object","properties":{"storageClass":{"description":"Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"nodeGroups":{"description":"Worker nodes configuration map. When left empty, a default node group `md0` is emitted with `minReplicas: 0` and `roles: [ingress-nginx]` — the MachineDeployment renders but provisions no workers until an unschedulable Pod triggers the cluster-autoscaler (or an operator scales the group manually). Enabling `addons.ingressNginx.enabled: true` on a CR with default `nodeGroups: {}` therefore requires either supplying a nodeGroup with `roles: [ingress-nginx]` and `minReplicas >= 1`, or waiting for the autoscaler to bring up the default md0 in response to the ingress-nginx controller Pods becoming Pending. Provide your own groups to take full control — they are not merged with the default, so you may name and omit groups freely.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskSize","instanceType","maxReplicas","minReplicas"],"properties":{"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs. Pre-Talos installs used a separate disk-kubelet PVC for kubelet/containerd state; on Talos this is consolidated onto the single system disk imaged from the factory artifact.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"kubelet":{"description":"Kubelet resource reservations for this node group.","type":"object","properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"logSerialConsole":{"description":"Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the node group, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the node group down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.","type":"boolean","default":false},"maxReplicas":{"description":"Maximum number of replicas.","type":"integer","default":10},"maxUnhealthy":{"description":"Per-group override for `nodeHealthCheck.maxUnhealthy`. When unset, the cluster-wide `nodeHealthCheck.maxUnhealthy` applies. Accepts a bare integer (\"0\", \"1\", ...) or an integer percentage (\"0%\", \"50%\").","type":"string"},"minReplicas":{"description":"Minimum number of replicas.","type":"integer","default":0},"nodeStartupTimeout":{"description":"Per-group override for `nodeHealthCheck.nodeStartupTimeout`. When unset, the cluster-wide `nodeHealthCheck.nodeStartupTimeout` applies.","type":"string"},"resources":{"description":"Explicit CPU and memory for each worker node, as an Line truncated
|
||||
{"title":"Chart Values","type":"object","properties":{"storageClass":{"description":"Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"nodeGroups":{"description":"Worker nodes configuration map. When left empty, a default node group `md0` is emitted with `minReplicas: 0` and `roles: [ingress-nginx]` — the MachineDeployment renders but provisions no workers until an unschedulable Pod triggers the cluster-autoscaler (or an operator scales the group manually). Enabling `addons.ingressNginx.enabled: true` on a CR with default `nodeGroups: {}` therefore requires either supplying a nodeGroup with `roles: [ingress-nginx]` and `minReplicas >= 1`, or waiting for the autoscaler to bring up the default md0 in response to the ingress-nginx controller Pods becoming Pending. Provide your own groups to take full control — they are not merged with the default, so you may name and omit groups freely.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskSize","instanceType","maxReplicas","minReplicas"],"properties":{"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs. Pre-Talos installs used a separate disk-kubelet PVC for kubelet/containerd state; on Talos this is consolidated onto the single system disk imaged from the factory artifact.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"kubelet":{"description":"Kubelet resource reservations for this node group.","type":"object","properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"logSerialConsole":{"description":"Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the node group, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the node group down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.","type":"boolean","default":false},"maxReplicas":{"description":"Maximum number of replicas.","type":"integer","default":10},"maxUnhealthy":{"description":"Per-group override for `nodeHealthCheck.maxUnhealthy`. When unset, the cluster-wide `nodeHealthCheck.maxUnhealthy` applies. Accepts a bare integer (\"0\", \"1\", ...) or an integer percentage (\"0%\", \"50%\").","type":"string"},"minReplicas":{"description":"Minimum number of replicas.","type":"integer","default":0},"nodeStartupTimeout":{"description":"Per-group override for `nodeHealthCheck.nodeStartupTimeout`. When unset, the cluster-wide `nodeHealthCheck.nodeStartupTimeout` applies.","type":"string"},"podCpuLimit":{"description":"CPU limit for the worker VM's virt-launcher `compuLine truncated
|
||||
release:
|
||||
prefix: kubernetes-
|
||||
labels:
|
||||
|
||||
Reference in new issue
Block a user