feat(kubernetes-nodes): add worker node pool chart

Introduce the kubernetes-nodes application chart: one HelmRelease per
worker node pool, attached to a parent Kubernetes cluster by name
convention (kubernetes-<cluster>). Renders the KubevirtMachineTemplate,
MachineDeployment, MachineHealthCheck, worker WorkloadMonitor and the
talos-reconcile Job with its RBAC, extracted from the monolithic
kubernetes chart. Pool objects are byte-identical to the parent's render
for the same group, so a later adoption never churns live worker VMs.

This is the additive half of the Phase 2 split (community#8); the parent
kubernetes chart is left unchanged.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
This commit is contained in:
Ivan Okhotnikov authored and IvanHunters committed 2026-07-21 22:35:10 +03:00
1 parent aef9e201c7
commit fd14126bea
17 files changed
+1556

No files matched your search

@@ -0,0 +1,5 @@
.helmignore
/logos
/Makefile
/hack
/images/*/*
@@ -0,0 +1,7 @@
apiVersion: v2
name: kubernetes-nodes
description: Worker node pool for a managed Kubernetes cluster
icon: /logos/kubernetes.svg
type: application
version: 0.0.0 # Placeholder, the actual version will be automatically set during the build process
appVersion: 1.32.6
+19
View File
@@ -0,0 +1,19 @@
include ../../../hack/package.mk
.PHONY: test generate update
test:
helm unittest .
generate:
cozyvalues-gen -m 'kubernetesnodes' -v values.yaml -s values.schema.json -r README.md -g ../../../api/apps/v1alpha1/kubernetesnodes/types.go
../../../hack/update-crd.sh
# Keep the worker-image inputs (Talos<->Kubernetes support matrix and the
# kubectl.tag used by the talos-reconcile Job) in lockstep with the parent
# kubernetes chart. They are copied, not symlinked, so `helm package` bundles
# them; run this after the parent chart's `make update`.
update:
cp ../kubernetes/files/versions.yaml files/versions.yaml
cp ../kubernetes/images/kubectl.tag images/kubectl.tag
$(MAKE) generate
+79
View File
@@ -0,0 +1,79 @@
# Kubernetes Worker Node Pool
`kubernetes-nodes` manages a single worker node pool for a managed Kubernetes
cluster. It is the worker half of the Kubernetes-app split (see
[cozystack/community#8](https://github.com/cozystack/community/pull/8)): the
control plane lives in the `kubernetes` chart, and each pool is a separate
`kubernetes-nodes` release attached to a parent cluster by name.
A `KubernetesNodes` CR named `<cluster>-<pool>` (e.g. `myk8s-md0`) provisions a
`MachineDeployment` + `KubevirtMachineTemplate` + `MachineHealthCheck` wired to
the parent cluster `kubernetes-<cluster>` in the same namespace, plus a
`talos-reconcile` Job that produces the pool's `TalosConfigTemplate`.
## Parameters
### Linkage
| Name | Description | Type | Value |
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----- |
| `cluster` | Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. | `string` | `""` |
### Common Parameters
| Name | Description | Type | Value |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | ------------ |
| `storageClass` | StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. | `string` | `replicated` |
### Pool Sizing
| Name | Description | Type | Value |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------- |
| `minReplicas` | Minimum number of replicas in this pool. Used as the cluster-autoscaler floor. | `int` | `0` |
| `maxReplicas` | Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling. | `int` | `10` |
| `instanceType` | Virtual machine instance type. | `string` | `u1.medium` |
| `diskSize` | System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs. | `quantity` | `20Gi` |
| `roles` | List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller. | `[]string` | `[]` |
| `resources` | Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. | `object` | `{}` |
| `resources.cpu` | CPU available. | `quantity` | `""` |
| `resources.memory` | Memory (RAM) available. | `quantity` | `""` |
| `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
| `gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` |
| `kubelet` | Kubelet resource reservations for this pool. | `object` | `{}` |
| `kubelet.systemReservedMemory` | Memory reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
| `kubelet.kubeReservedMemory` | Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
| `kubelet.systemReservedCpu` | CPU reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
| `kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
| `kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` |
| `kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` |
| `maxUnhealthy` | Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. | `string` | `50%` |
| `nodeStartupTimeout` | Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). | `string` | `10m` |
### Kubernetes Version
| Name | Description | Type | Value |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- |
| `version` | Kubernetes major.minor version the pool joins. Must match the parent cluster's version and satisfy the Talos<->Kubernetes support matrix against `talos.version`. | `string` | `v1.35` |
### Talos Worker Image
| Name | Description | Type | Value |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------ |
| `talos` | Talos worker image configuration. Keep in sync with the parent cluster's `talos`. | `object` | `{}` |
| `talos.version` | Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`. | `string` | `v1.13.6` |
| `talos.schematicID` | Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. | `string` | `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` |
| `talos.imageFactoryURL` | Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. | `string` | `https://factory.talos.dev` |
| `talos.installerRepository` | OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. | `string` | `factory.talos.dev/installer` |
### Images
| Name | Description | Type | Value |
| ---------------- | ---------------------------------------------------------------------------------------- | -------- | ----- |
| `images` | Optional image overrides for air-gapped or rate-limited registries. | `object` | `{}` |
| `images.kubectl` | Image used by the talos-reconcile Job (kubectl). Empty falls back to images/kubectl.tag. | `string` | `""` |
+1
View File
@@ -0,0 +1 @@
../../../library/cozy-lib
@@ -0,0 +1,5 @@
"v1.35": "v1.35.6"
"v1.34": "v1.34.9"
"v1.33": "v1.33.13"
"v1.32": "v1.32.13"
"v1.31": "v1.31.14"
@@ -0,0 +1 @@
docker.io/alpine/k8s:1.33.4@sha256:b0523f0a244ddc4c8e055aa335c040d3d78b3ead5528f4544395f7f9f69c7b68
@@ -0,0 +1,10 @@
<svg width="144" height="144" viewBox="0 0 144 144" fill="none" xmlns="http://www.w3.org/2000/svg">
<rect width="144" height="144" rx="24" fill="url(#paint0_linear_681_2845)"/>
<path d="M71.9968 19C70.3039 19.0002 68.9312 20.5322 68.9314 22.4221C68.9314 22.4511 68.9373 22.4788 68.9379 22.5076C68.9354 22.7644 68.9231 23.0737 68.9314 23.2973C68.9717 24.3873 69.2082 25.2216 69.3506 26.2258C69.6084 28.3752 69.8245 30.1569 69.6912 31.813C69.5615 32.4375 69.1037 33.0086 68.6956 33.4056L68.6235 34.7086C66.7839 34.8617 64.9319 35.1421 63.0821 35.5641C55.1226 37.3798 48.2695 41.4991 43.052 47.0609C42.7134 46.8288 42.1211 46.4019 41.945 46.2712C41.3977 46.3454 40.8445 46.5151 40.1241 46.0935C38.7522 45.1657 37.5028 43.8851 35.991 42.3424C35.2982 41.6044 34.7966 40.9018 33.9735 40.1904C33.7866 40.0289 33.5014 39.8104 33.2923 39.6442C32.6489 39.1288 31.89 38.86 31.157 38.8348C30.2147 38.8024 29.3075 39.1725 28.7138 39.9206C27.6584 41.2506 27.9963 43.2833 29.4671 44.4614C29.482 44.4734 29.4979 44.4827 29.5129 44.4943C29.715 44.6589 29.9625 44.8698 30.1483 45.0076C31.0217 45.6555 31.8195 45.9872 32.6897 46.5015C34.5231 47.6391 36.043 48.5823 37.2486 49.7196C37.7194 50.2237 37.8016 51.1122 37.8643 51.4964L38.8468 52.3782C33.5872 60.3308 31.153 70.1539 32.5915 80.1627L31.3077 80.5378C30.9693 80.9768 30.4912 81.6676 29.9911 81.8738C28.4138 82.3729 26.6387 82.5562 24.4956 82.7819C23.4894 82.866 22.6213 82.8158 21.5546 83.0188C21.3198 83.0635 20.9927 83.1491 20.7358 83.2097C20.7269 83.2116 20.7186 83.2142 20.7096 83.2162C20.6956 83.2195 20.6772 83.2263 20.6638 83.2294C18.857 83.668 17.6963 85.3365 18.0699 86.9805C18.4437 88.6248 20.2086 89.6248 22.0262 89.2312C22.0393 89.2282 22.0584 89.2277 22.072 89.2246C22.0926 89.2199 22.1106 89.2099 22.131 89.2049C22.3844 89.149 22.7019 89.0868 22.9236 89.0272C23.9723 88.7451 24.7318 88.3306 25.6746 87.9677C27.7029 87.2368 29.3828 86.6262 31.0195 86.3883C31.703 86.3345 32.4232 86.812 32.7814 87.0134L34.1177 86.7831C37.1926 96.3613 43.6366 104.103 51.7963 108.961L51.2396 110.303C51.4403 110.824 51.6616 111.53 51.5121 112.045C50.9171 113.595 49.898 115.231 48.7374 117.055C48.1755 117.898 47.6004 118.552 47.0934 119.516C46.972 119.747 46.8175 120.102 46.7004 120.346C45.9125 122.039 46.4904 123.99 48.0038 124.722C49.5268 125.459 51.4171 124.682 52.2352 122.985C52.2364 122.982 52.2406 122.98 52.2417 122.978C52.2426 122.976 52.2409 122.973 52.2417 122.971C52.3582 122.731 52.5233 122.415 52.6216 122.188C53.056 121.189 53.2005 120.332 53.5059 119.365C54.317 117.318 54.7626 115.17 55.8791 113.832C56.1849 113.466 56.6833 113.325 57.2001 113.186L57.8944 111.922C65.008 114.665 72.9705 115.402 80.9245 113.587C82.7391 113.173 84.4908 112.637 86.1843 111.994C86.3794 112.342 86.742 113.011 86.8393 113.179C87.3644 113.351 87.9377 113.439 88.4047 114.133C89.2401 115.567 89.8114 117.263 90.5073 119.312C90.8128 120.279 90.9638 121.136 91.3981 122.136C91.4971 122.363 91.6614 122.684 91.778 122.925C92.5944 124.628 94.4907 125.407 96.0159 124.669C97.5292 123.937 98.1077 121.986 97.3194 120.293C97.2023 120.049 97.0412 119.695 96.9198 119.464C96.4127 118.499 95.8377 117.852 95.2758 117.009C94.1152 115.185 93.1526 113.67 92.5575 112.12C92.3087 111.32 92.5995 110.823 92.7933 110.303C92.6772 110.17 92.4288 109.414 92.2824 109.059C100.762 104.029 107.017 95.9985 109.955 86.7239C110.351 86.7865 111.041 86.9091 111.265 86.9542C111.726 86.6487 112.149 86.2501 112.981 86.3159C114.617 86.5537 116.297 87.1645 118.326 87.8953C119.268 88.2581 120.028 88.6793 121.077 88.9614C121.298 89.021 121.616 89.0766 121.869 89.1325C121.889 89.1375 121.908 89.1475 121.928 89.1522C121.942 89.1553 121.961 89.1558 121.974 89.1588C123.792 89.552 125.557 88.5526 125.93 86.9081C126.303 85.2641 125.143 83.5952 123.336 83.157C123.074 83.097 122.701 82.995 122.446 82.9465C121.379 82.7435 120.511 82.7935 119.505 82.7095C117.361 82.4839 115.586 82.3004 114.009 81.8014C113.366 81.5507 112.908 80.7819 112.686 80.4655L111.448 80.1035C112.09 75.438 111.917 70.5825 110.806 65.7243C109.685 60.8208 107.704 56.3361 105.062 52.3848C105.379 52.0948 105.979 51.5612 106.149 51.4043C106.199 50.8517 106.156 50.2722 106.725 49.6603C107.931 48.523 109.451 47.5799 111.284 46.4423C112.154 45.9279 112.959 45.5964 113.832 44.9484C114.03 44.8019 114.299 44.5699 114.507 44.4022C115.977 43.2237 116.316 41.1911 115.26 39.8614C114.204 38.5317 112.159 38.4065 110.688 39.585C110.479 39.7516 110.195 39.9688 110.007 40.1312C109.184 40.8426 108.676 41.5452 107.983 42.2832C106.471 43.8259 105.222 45.113 103.85 46.0409C103.255 46.3885 102.385 46.2682 101.99 46.2449L100.824 47.0806C94.1753 40.0763 85.1235 35.5982 75.3766 34.7283C75.3494 34.3179 75.3137 33.5761 75.3046 33.3529C74.9056 32.9693 74.4235 32.6418 74.3024 31.813C74.1691 30.1569 74.3917 28.3752 74.6496 26.2258C74.7919 25.2216 75.0284 24.3873 75.0688 23.2973C75.078 23.0495 75.0632 22.69 75.0622 22.4221C75.062 20.5322 73.6898 18.9998 71.9968 19ZM68.1585 42.8886L67.248 59.0447L67.1825 59.0776C67.1214 60.5229 65.9375 61.677 64.4839 61.677C63.8884 61.677 63.3388 61.4849 62.8922 61.1571L62.866 61.1703L49.6807 51.7794C53.7331 47.7759 58.9164 44.8172 64.89 43.4546C65.9812 43.2056 67.0719 43.0209 68.15Line truncated
<defs>
<linearGradient id="paint0_linear_681_2845" x1="10" y1="15.5" x2="144" y2="131.5" gradientUnits="userSpaceOnUse">
<stop stop-color="#4D87FF"/>
<stop offset="1" stop-color="#0547D0"/>
</linearGradient>
</defs>
</svg>

After

Width:  |  Height:  |  Size: 7.0 KiB

@@ -0,0 +1,12 @@
Worker node pool "{{ include "kubernetes-nodes.groupName" . }}" for cluster "{{ .Values.cluster }}".
This pool is a separate release from the control plane. Two operational notes:
1. Deletion order. Deleting the parent Kubernetes cluster while this pool still
exists leaves the pool release trying to recreate CAPI objects against a
cluster that is being torn down. Delete the KubernetesNodes pools first, then
the parent Kubernetes cluster.
2. Version skew. Keep this pool's `version` and `talos.*` in sync with the parent
Kubernetes cluster. A pool on a Kubernetes/Talos combination the parent does
not run can produce workers the control plane will not admit.
@@ -0,0 +1,84 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "kubernetes.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "kubernetes.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "kubernetes.labels" -}}
helm.sh/chart: {{ include "kubernetes.chart" . }}
{{ include "kubernetes.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "kubernetes.selectorLabels" -}}
app.kubernetes.io/name: {{ include "kubernetes.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
DNS domain used INSIDE the tenant cluster (kubelet --cluster-domain,
apiserver --service-cluster-ip-range FQDNs, CoreDNS authoritative zone).
Pinned to Kamaji's default `networkProfile.clusterDomain`. Kept identical to
the parent kubernetes chart so the worker machineconfig this chart applies
matches the control plane it joins.
Distinct from .Values._cluster["cluster-domain"], which is the MANAGEMENT
cluster domain (e.g. cozy.local) where the Kamaji control plane lives.
*/}}
{{- define "kubernetes.tenantClusterDomain" -}}
cluster.local
{{- end }}
{{/*
Reconstruct the parent CAPI cluster name from the linkage value.
The pool attaches to the parent Kubernetes CR named .Values.cluster, whose
HelmRelease (and therefore CAPI Cluster / KamajiControlPlane / KubevirtCluster
and every worker object it owns) is `kubernetes-<cluster>`. This chart does
NOT use its own .Release.Name for CAPI wiring — the pool lives in a separate
HelmRelease from the control plane, so every reference that the monolithic
chart made through $.Release.Name is reconstructed here as kubernetes-<cluster>
instead. Linkage is by name convention (mirrors vm-instance -> vm-disk), not
ownerReference or lookup-gated render.
*/}}
{{- define "kubernetes-nodes.clusterName" -}}
{{- if not .Values.cluster -}}
{{- fail "kubernetes-nodes: .Values.cluster is required — set it to the parent Kubernetes CR name so the pool attaches to cluster kubernetes-<cluster>" -}}
{{- end -}}
{{- printf "kubernetes-%s" .Values.cluster -}}
{{- end -}}
{{/*
The node-group name for this pool, derived from the release name.
A KubernetesNodes CR is named <cluster>-<pool> and gets the release prefix
`kubernetes-nodes-`, so the release name is `kubernetes-nodes-<cluster>-<pool>`.
The group name is the <pool> suffix. Enforcing the `<cluster>-` segment keeps
every rendered object named `kubernetes-<cluster>-<pool>` — byte-identical to
what the monolithic chart rendered for the same group — and prevents two
clusters in one namespace from colliding on a pool named e.g. `md0`.
*/}}
{{- define "kubernetes-nodes.groupName" -}}
{{- $prefix := printf "kubernetes-nodes-%s-" .Values.cluster -}}
{{- if not (hasPrefix $prefix .Release.Name) -}}
{{- fail (printf "kubernetes-nodes: release name %q must start with %q — name the KubernetesNodes CR <cluster>-<pool> (cluster=%q)" .Release.Name $prefix .Values.cluster) -}}
{{- end -}}
{{- trimPrefix $prefix .Release.Name -}}
{{- end -}}
@@ -0,0 +1,12 @@
{{/*
Convert a CPU quantity string (e.g. "100m", "0.5", "1", "4") to millicores.
Returns a number string: "100", "500", "1000", "4000".
*/}}
{{- define "kubernetes.cpuToMillicores" -}}
{{- $str := . | toString -}}
{{- if hasSuffix "m" $str -}}
{{- trimSuffix "m" $str -}}
{{- else -}}
{{- mulf ($str | float64) 1000.0 | int | toString -}}
{{- end -}}
{{- end -}}
@@ -0,0 +1,7 @@
{{- define "kubernetes.versionMap" }}
{{- $versionMap := .Files.Get "files/versions.yaml" | fromYaml }}
{{- if not (hasKey $versionMap .Values.version) }}
{{- printf `Kubernetes version %s is not supported, allowed versions are %s` $.Values.version (keys $versionMap) | fail }}
{{- end }}
{{- index $versionMap .Values.version }}
{{- end }}
@@ -0,0 +1,25 @@
{{- $clusterName := include "kubernetes-nodes.clusterName" . }}
{{- $groupName := include "kubernetes-nodes.groupName" . }}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ .Release.Name }}-dashboard-resources
rules:
- apiGroups:
- cozystack.io
resources:
- workloadmonitors
resourceNames:
- {{ $clusterName }}-{{ $groupName }}
verbs: ["get", "list", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: {{ .Release.Name }}-dashboard-resources
subjects:
{{ include "cozy-lib.rbac.subjectsForTenantAndAccessLevel" (list "use" .Release.Namespace) }}
roleRef:
kind: Role
name: {{ .Release.Name }}-dashboard-resources
apiGroup: rbac.authorization.k8s.io
@@ -0,0 +1,543 @@
{{- /*
Worker node pool for a managed Kubernetes cluster.
This chart is the second half of the Kubernetes-app split (cozystack/community#8):
the control plane (Cluster / KamajiControlPlane / KubevirtCluster / PKI) lives in
the `kubernetes` chart, and each worker pool is a separate `kubernetes-nodes`
release. The pool objects (KubevirtMachineTemplate, MachineDeployment,
MachineHealthCheck, worker WorkloadMonitor) and the talos-reconcile Job that
produces the pool's TalosConfigTemplate are rendered here, wired to the parent
cluster `kubernetes-<cluster>` by name convention (see _helpers.tpl). Every
reference the monolithic chart made through $.Release.Name is reconstructed here
as the parent clusterName so the objects are byte-identical to what the parent
rendered for the same group — the KubevirtMachineTemplate is content-hash-named,
so any drift would roll every live worker VM.
Talos -> Kubernetes support matrix. Each Talos minor release supports a bounded
window of Kubernetes minor versions; running a kubelet outside that window
produces a silently broken Talos+kubelet combination. Fail the render instead.
Source: https://docs.siderolabs.com/talos/v1.13/getting-started/support-matrix
*/}}
{{- $talosK8sSupportMatrix := dict
"v1.13" (list "v1.31" "v1.32" "v1.33" "v1.34" "v1.35" "v1.36")
}}
{{- $talosMinor := regexFind "^v[0-9]+\\.[0-9]+" .Values.talos.version }}
{{- with index $talosK8sSupportMatrix $talosMinor }}
{{- if not (has $.Values.version .) }}
{{- fail (printf "Kubernetes %s is not supported by Talos %s. Supported versions: %v. Bump 'version' or set talos.version explicitly." $.Values.version $.Values.talos.version .) }}
{{- end }}
{{- end }}
{{- $clusterName := include "kubernetes-nodes.clusterName" . }}
{{- $groupName := include "kubernetes-nodes.groupName" . }}
{{- $kubevirtmachinetemplateNames := list }}
{{- $group := dict
"minReplicas" .Values.minReplicas
"maxReplicas" .Values.maxReplicas
"instanceType" .Values.instanceType
"diskSize" .Values.diskSize
"storageClass" .Values.storageClass
"roles" .Values.roles
"resources" .Values.resources
"gpus" .Values.gpus
"kubelet" .Values.kubelet
}}
{{- define "kubevirtmachinetemplate" -}}
spec:
{{- /* Talos has no SSH; use 'none' so CAPK does not block Machine.Ready
on an SSH probe that will never succeed. The Machine reaches Ready
once kubelet registers with the tenant apiserver via standard CSR. */}}
virtualMachineBootstrapCheck:
checkStrategy: none
virtualMachineTemplate:
metadata:
namespace: {{ $.Release.Namespace }}
labels:
{{- range .group.roles }}
node-role.kubernetes.io/{{ . }}: ""
{{- end }}
spec:
{{- /* Explicit resources take precedence over instanceType. When a node
group is sized by resources (both cpu and memory set), the
instancetype is omitted from the VM: KubeVirt rejects a
VirtualMachine that references an instancetype and also overrides
either domain.cpu or domain.memory. */}}
{{- $sizedByResources := and .group.resources .group.resources.cpu .group.resources.memory }}
{{- if and .group.instanceType (not $sizedByResources) }}
instancetype:
kind: VirtualMachineClusterInstancetype
name: {{ .group.instanceType }}
{{- end }}
runStrategy: Always
dataVolumeTemplates:
- metadata:
name: disk-system
annotations:
cdi.kubevirt.io/storage.usePopulator: "false"
spec:
{{- /* Use the openstack/configdrive Talos image: KubeVirt CAPK
injects bootstrap data via cloudInitConfigDrive (OpenStack
metadata service shape), and the openstack platform image
is the one that reads it. The nocloud-amd64 image silently
ignores configdrive and never applies the machineconfig. */}}
source:
http:
url: {{ $.Values.talos.imageFactoryURL | trimSuffix "/" }}/image/{{ $.Values.talos.schematicID }}/{{ $.Values.talos.version }}/openstack-amd64.raw.xz
{{- if hasKey .group "ephemeralStorage" }}
{{- fail (printf "nodeGroup %q: ephemeralStorage is no longer supported and should have been automatically migrated to diskSize by platform migration 41. If you see this error after upgrading, the migration did not run — check the cozystack-migration-hook Job logs in cozy-system." .groupName) }}
{{- end }}
storage:
resources:
requests:
storage: {{ .group.diskSize | default "20Gi" | quote }}
{{- /*
Worker VMs live-migrate, so their disks need ReadWriteMany. The
RWX access mode is supplied by the chosen StorageClass's CDI
StorageProfile — it is not set on the DataVolume here. linstor-csi
rejects RWX volumes that are not on a DRBD-backed StorageClass, so
an empty nodeGroup storageClass falls back to the application-level
storageClass (a replicated/DRBD class) instead of the cluster
default, which may be a non-DRBD class that the CSI driver would
reject. Both empty still omits the field so the cluster default
applies.
*/}}
{{- with .group.storageClass | default $.Values.storageClass }}
storageClassName: {{ . | quote }}
{{- end }}
template:
metadata:
annotations:
kubevirt.io/allow-pod-bridge-network-live-migration: "true"
labels:
apps.cozystack.io/application.group: apps.cozystack.io
apps.cozystack.io/application.kind: Kubernetes
apps.cozystack.io/application.name: {{ .clusterName | trimPrefix "kubernetes-" | quote }}
{{- range .group.roles }}
node-role.kubernetes.io/{{ . }}: ""
{{- end }}
cluster.x-k8s.io/deployment-name: {{ .clusterName }}-{{ .groupName }}
spec:
{{- if .Values._cluster.scheduling }}
{{- $rawConstraints := get .Values._cluster.scheduling "globalAppTopologySpreadConstraints" }}
{{- if $rawConstraints }}
{{- $rawConstraints | fromYaml | toYaml | nindent 10 }}
labelSelector:
matchLabels:
cluster.x-k8s.io/cluster-name: {{ .clusterName }}
{{- end }}
{{- end }}
domain:
{{- if and .group.resources .group.resources.cpu }}
cpu:
threads: 1
cores: {{ .group.resources.cpu }}
sockets: 1
{{- end }}
devices:
{{- if .group.gpus }}
gpus:
{{- range $i, $gpu := .group.gpus }}
- name: gpu{{ add $i 1 }}
deviceName: {{ $gpu.name }}
{{- end }}
{{- end }}
disks:
- name: system
disk:
bus: virtio
pciAddress: 0000:07:00.0
blockSize:
custom:
logical: 512
physical: 4096
interfaces:
- name: default
bridge: {}
{{- if and .group.resources .group.resources.memory }}
memory:
guest: {{ .group.resources.memory }}
{{- end }}
resources:
{{/* 2Gi is enough: persistent DataVolumes are not counted as pod ephemeral storage */}}
{{- include "cozy-lib.resources.sanitize" (list (dict "ephemeral-storage" "2Gi") $) | nindent 14 }}
evictionStrategy: External
volumes:
- name: system
dataVolume:
name: disk-system
networks:
- name: default
pod: {}
{{- end }}
{{- /* instanceType and explicit resources are alternative sizing sources, and
explicit resources win. KubeVirt rejects a VirtualMachine that references
an instancetype while also overriding either domain.cpu or domain.memory,
so when a group is sized by resources (both cpu and memory set) the
instancetype is omitted from the VM (see the VM spec above) and the lookup
below is skipped — a stale instancetype name cannot fail an otherwise
valid, resource-sized group. Explicit resources are all-or-nothing: a
partial spec (only one of cpu/memory) would emit one domain override
alongside the instancetype, which KubeVirt rejects, so it fails here with
a clear render-time error instead of a late admission rejection. */}}
{{- $hasCpu := and $group.resources $group.resources.cpu }}
{{- $hasMem := and $group.resources $group.resources.memory }}
{{- if and $group.instanceType (or (and $hasCpu (not $hasMem)) (and $hasMem (not $hasCpu))) }}
{{- fail (printf "nodeGroup %s: set both resources.cpu and resources.memory, or neither" $groupName) }}
{{- end }}
{{/* Resolve instanceType once per group — reused for kubelet reservations and capacity annotations. */}}
{{- $instanceType := dict }}
{{- if and $group.instanceType (not (and $hasCpu $hasMem)) }}
{{- $instanceType = (lookup "instancetype.kubevirt.io/v1beta1" "VirtualMachineClusterInstancetype" "" $group.instanceType) }}
{{- if not $instanceType }}
{{- fail (printf "nodeGroup %s: specified instanceType %q not found in cluster" $groupName $group.instanceType) }}
{{- end }}
{{- end }}
{{/* Compute kubelet reservations: explicit resources.memory/cpu wins over instanceType spec.
Auto-computed system/kube reserved memory = 5% of effective memory, clamped to [256Mi, 1Gi].
Auto-computed system/kube reserved CPU = 5% of effective CPU, clamped to [50m, 500m]. */}}
{{- $effectiveMemory := "" }}
{{- if and $group.resources $group.resources.memory }}
{{- $effectiveMemory = $group.resources.memory | toString }}
{{- else if and $instanceType $instanceType.spec $instanceType.spec.memory $instanceType.spec.memory.guest }}
{{- $effectiveMemory = $instanceType.spec.memory.guest | toString }}
{{- end }}
{{- $effectiveCpu := "" }}
{{- if and $group.resources $group.resources.cpu }}
{{- $effectiveCpu = $group.resources.cpu | toString }}
{{- else if and $instanceType $instanceType.spec $instanceType.spec.cpu $instanceType.spec.cpu.guest }}
{{- $effectiveCpu = $instanceType.spec.cpu.guest | toString }}
{{- end }}
{{- $autoReservedMi := 256 }}
{{- if $effectiveMemory }}
{{- $effectiveMemMi := divf (include "cozy-lib.resources.toFloat" $effectiveMemory | float64) 1048576.0 | int }}
{{- $fivePercentMi := mulf ($effectiveMemMi | float64) 0.05 | int }}
{{- $autoReservedMi = min (max $fivePercentMi 256) 1024 }}
{{- end }}
{{- $autoReservedMillicores := 50 }}
{{- if $effectiveCpu }}
{{- $effectiveCpuMillicores := include "kubernetes.cpuToMillicores" $effectiveCpu | int }}
{{- $fivePercentMillis := mulf ($effectiveCpuMillicores | float64) 0.05 | int }}
{{- $autoReservedMillicores = min (max $fivePercentMillis 50) 500 }}
{{- end }}
{{- $kubeletOverride := $group.kubelet | default dict }}
{{- $systemReservedMemory := $kubeletOverride.systemReservedMemory | default (printf "%dMi" $autoReservedMi) }}
{{- $kubeReservedMemory := $kubeletOverride.kubeReservedMemory | default (printf "%dMi" $autoReservedMi) }}
{{- $systemReservedCpu := $kubeletOverride.systemReservedCpu | default (printf "%dm" $autoReservedMillicores) }}
{{- $kubeReservedCpu := $kubeletOverride.kubeReservedCpu | default (printf "%dm" $autoReservedMillicores) }}
{{- $evictionHardMemory := $kubeletOverride.evictionHardMemory | default "7%" }}
{{- $evictionSoftMemory := $kubeletOverride.evictionSoftMemory | default "10%" }}
{{/* Validate reservation field formats (user-supplied values only; computed defaults are valid by construction). */}}
{{- $validations := list
(dict "key" "systemReservedMemory" "value" $kubeletOverride.systemReservedMemory "pattern" `^[0-9]+(\.[0-9]+)?([eE][+-]?[0-9]+)?(Ki|Mi|Gi|Ti|Pi|Ei|k|M|G|T|P|E)?$` "hint" "a valid Kubernetes memory quantity (e.g. 128Mi, 1Gi, 1e9, 500e6)")
(dict "key" "kubeReservedMemory" "value" $kubeletOverride.kubeReservedMemory "pattern" `^[0-9]+(\.[0-9]+)?([eE][+-]?[0-9]+)?(Ki|Mi|Gi|Ti|Pi|Ei|k|M|G|T|P|E)?$` "hint" "a valid Kubernetes memory quantity (e.g. 128Mi, 1Gi, 1e9, 500e6)")
(dict "key" "systemReservedCpu" "value" $kubeletOverride.systemReservedCpu "pattern" `^([0-9]+m|[0-9]+(\.[0-9]+)?)$` "hint" "a valid Kubernetes CPU quantity (integer millicores like 100m, or decimal cores like 0.5, 1)")
(dict "key" "kubeReservedCpu" "value" $kubeletOverride.kubeReservedCpu "pattern" `^([0-9]+m|[0-9]+(\.[0-9]+)?)$` "hint" "a valid Kubernetes CPU quantity (integer millicores like 100m, or decimal cores like 0.5, 1)")
}}
{{- range $validations }}
{{- if and .value (not (regexMatch .pattern (.value | toString))) }}
{{- fail (printf "nodeGroup %s: invalid %s value %q — must be %s" $groupName .key (.value | toString) .hint) }}
{{- end }}
{{- end }}
{{/* Validate that user-supplied reservation values are positive (non-zero) */}}
{{- range $_, $entry := list (dict "key" "systemReservedMemory" "value" $kubeletOverride.systemReservedMemory) (dict "key" "kubeReservedMemory" "value" $kubeletOverride.kubeReservedMemory) }}
{{- if $entry.value }}
{{- $bytes := include "cozy-lib.resources.toFloat" ($entry.value | toString) | float64 }}
{{- if le $bytes 0.0 }}
{{- fail (printf "nodeGroup %s: %s must be a positive value, got %s" $groupName $entry.key ($entry.value | toString)) }}
{{- end }}
{{- end }}
{{- end }}
{{- range $_, $entry := list (dict "key" "systemReservedCpu" "value" $kubeletOverride.systemReservedCpu) (dict "key" "kubeReservedCpu" "value" $kubeletOverride.kubeReservedCpu) }}
{{- if $entry.value }}
{{- $cpuMillis := include "kubernetes.cpuToMillicores" $entry.value | float64 }}
{{- if le $cpuMillis 0.0 }}
{{- fail (printf "nodeGroup %s: %s must be a positive value, got %s" $groupName $entry.key ($entry.value | toString)) }}
{{- end }}
{{- end }}
{{- end }}
{{/* Validate absolute eviction threshold formats before converting */}}
{{- if not (hasSuffix "%" $evictionHardMemory) }}
{{- if not (regexMatch `^[0-9]+(\.[0-9]+)?([eE][+-]?[0-9]+)?(Ki|Mi|Gi|Ti|Pi|Ei|k|M|G|T|P|E)?$` $evictionHardMemory) }}
{{- fail (printf "nodeGroup %s: invalid evictionHardMemory value %q — must be a percentage (e.g. 7%%) or a valid Kubernetes memory quantity (e.g. 500Mi, 1Gi, 1e9)" $groupName $evictionHardMemory) }}
{{- end }}
{{- end }}
{{- if not (hasSuffix "%" $evictionSoftMemory) }}
{{- if not (regexMatch `^[0-9]+(\.[0-9]+)?([eE][+-]?[0-9]+)?(Ki|Mi|Gi|Ti|Pi|Ei|k|M|G|T|P|E)?$` $evictionSoftMemory) }}
{{- fail (printf "nodeGroup %s: invalid evictionSoftMemory value %q — must be a percentage (e.g. 10%%) or a valid Kubernetes memory quantity (e.g. 500Mi, 1Gi, 1e9)" $groupName $evictionSoftMemory) }}
{{- end }}
{{- end }}
{{- if and (hasSuffix "%" $evictionHardMemory) (hasSuffix "%" $evictionSoftMemory) }}
{{- $hardPctStr := trimSuffix "%" $evictionHardMemory }}
{{- $softPctStr := trimSuffix "%" $evictionSoftMemory }}
{{- if not (regexMatch `^[0-9]+(\.[0-9]+)?$` $hardPctStr) }}
{{- fail (printf "nodeGroup %s: evictionHardMemory percentage value %q is not a valid number" $groupName $hardPctStr) }}
{{- end }}
{{- if not (regexMatch `^[0-9]+(\.[0-9]+)?$` $softPctStr) }}
{{- fail (printf "nodeGroup %s: evictionSoftMemory percentage value %q is not a valid number" $groupName $softPctStr) }}
{{- end }}
{{- $hardPct := $hardPctStr | float64 }}
{{- $softPct := $softPctStr | float64 }}
{{- if le $hardPct 0.0 }}
{{- fail (printf "nodeGroup %s: evictionHardMemory percentage must be greater than 0 and at most 100, got %s%%" $groupName $hardPctStr) }}
{{- end }}
{{- if gt $hardPct 100.0 }}
{{- fail (printf "nodeGroup %s: evictionHardMemory percentage must be greater than 0 and at most 100, got %s%%" $groupName $hardPctStr) }}
{{- end }}
{{- if le $softPct 0.0 }}
{{- fail (printf "nodeGroup %s: evictionSoftMemory percentage must be greater than 0 and at most 100, got %s%%" $groupName $softPctStr) }}
{{- end }}
{{- if gt $softPct 100.0 }}
{{- fail (printf "nodeGroup %s: evictionSoftMemory percentage must be greater than 0 and at most 100, got %s%%" $groupName $softPctStr) }}
{{- end }}
{{- if ge $hardPct $softPct }}
{{- fail (printf "nodeGroup %s: evictionHardMemory (%s) must be strictly less than evictionSoftMemory (%s)" $groupName $evictionHardMemory $evictionSoftMemory) }}
{{- end }}
{{- else if and (not (hasSuffix "%" $evictionHardMemory)) (not (hasSuffix "%" $evictionSoftMemory)) }}
{{- $hardBytes := include "cozy-lib.resources.toFloat" $evictionHardMemory | float64 }}
{{- $softBytes := include "cozy-lib.resources.toFloat" $evictionSoftMemory | float64 }}
{{- if ge $hardBytes $softBytes }}
{{- fail (printf "nodeGroup %s: evictionHardMemory (%s) must be strictly less than evictionSoftMemory (%s)" $groupName $evictionHardMemory $evictionSoftMemory) }}
{{- end }}
{{- else }}
{{- fail (printf "nodeGroup %s: evictionHardMemory (%s) and evictionSoftMemory (%s) must use the same unit type (both percentage or both absolute)" $groupName $evictionHardMemory $evictionSoftMemory) }}
{{- end }}
{{/* Resolve eviction hard threshold to bytes for arithmetic.
Percentage is relative to effective node memory; absolute values are used as-is. */}}
{{- $evictionHardBytes := 0.0 }}
{{- if $effectiveMemory }}
{{- if hasSuffix "%" $evictionHardMemory }}
{{- $evictionPct := trimSuffix "%" $evictionHardMemory | float64 }}
{{- $evictionHardBytes = mulf (include "cozy-lib.resources.toFloat" $effectiveMemory | float64) (divf $evictionPct 100.0) }}
{{- else }}
{{- $evictionHardBytes = include "cozy-lib.resources.toFloat" $evictionHardMemory | float64 }}
{{- end }}
{{- end }}
{{/* Validate that total reserved memory (system + kube + eviction) does not exceed effective node memory */}}
{{- if $effectiveMemory }}
{{- $sysBytes := include "cozy-lib.resources.toFloat" $systemReservedMemory | float64 }}
{{- $kubeBytes := include "cozy-lib.resources.toFloat" $kubeReservedMemory | float64 }}
{{- $totalReservedBytes := addf (addf $sysBytes $kubeBytes) $evictionHardBytes }}
{{- $effectiveMemBytes := include "cozy-lib.resources.toFloat" $effectiveMemory | float64 }}
{{- if ge $totalReservedBytes $effectiveMemBytes }}
{{- fail (printf "nodeGroup %s: total reserved memory (systemReservedMemory=%s + kubeReservedMemory=%s + evictionHardMemory=%s) must be less than effective node memory (%s)" $groupName $systemReservedMemory $kubeReservedMemory $evictionHardMemory $effectiveMemory) }}
{{- end }}
{{- end }}
{{/* Validate that total reserved CPU does not exceed effective node CPU */}}
{{- if $effectiveCpu }}
{{- $sysCpuMillis := include "kubernetes.cpuToMillicores" $systemReservedCpu | float64 }}
{{- $kubeCpuMillis := include "kubernetes.cpuToMillicores" $kubeReservedCpu | float64 }}
{{- $totalCpuMillis := addf $sysCpuMillis $kubeCpuMillis }}
{{- $effectiveCpuMillicoresV := include "kubernetes.cpuToMillicores" $effectiveCpu | int }}
{{- $effectiveCpuMillis := float64 $effectiveCpuMillicoresV }}
{{- if ge $totalCpuMillis $effectiveCpuMillis }}
{{- fail (printf "nodeGroup %s: total reserved CPU (systemReservedCpu=%s + kubeReservedCpu=%s) must be less than effective node CPU (%s cores)" $groupName $systemReservedCpu $kubeReservedCpu $effectiveCpu) }}
{{- end }}
{{- end }}
{{/* Guard: explicit kubelet overrides require effective memory for validation */}}
{{- if not $effectiveMemory }}
{{- $hasOverride := or $kubeletOverride.systemReservedMemory $kubeletOverride.kubeReservedMemory $kubeletOverride.evictionHardMemory $kubeletOverride.evictionSoftMemory }}
{{- if $hasOverride }}
{{- fail (printf "nodeGroup %s: kubelet memory reservations require effective memory to validate — set resources.memory or instanceType" $groupName) }}
{{- end }}
{{- end }}
{{/* Guard: explicit kubelet CPU overrides require effective CPU for validation */}}
{{- if not $effectiveCpu }}
{{- $hasCpuOverride := or $kubeletOverride.systemReservedCpu $kubeletOverride.kubeReservedCpu }}
{{- if $hasCpuOverride }}
{{- fail (printf "nodeGroup %s: cannot validate kubelet CPU reservations without effective CPU — set resources.cpu or a valid instanceType" $groupName) }}
{{- end }}
{{- end }}
---
{{- /* TalosConfigTemplate is not rendered here — the chart cannot produce a
valid Talos worker machineconfig at Helm template time because three of
the inputs (apiserver Service ClusterIP, Talos CA, Kamaji-issued tenant
CA) are created asynchronously. The main-phase Job in
templates/talos-reconcile-job.yaml waits for those inputs and produces
the TalosConfigTemplate with a KamajiControlPlane ownerReference for
natural GC. The MachineDeployment below references the TCT name; CAPI's
Machine controller blocks on its absence and unblocks the moment the Job
applies it. */}}
---
{{- $context := deepCopy $ }}
{{- $_ := set $context "group" $group }}
{{- $_ := set $context "groupName" $groupName }}
{{- $_ := set $context "clusterName" $clusterName }}
{{- $kubevirtmachinetemplate := include "kubevirtmachinetemplate" $context }}
{{- $kubevirtmachinetemplateHash := $kubevirtmachinetemplate | sha256sum | trunc 6 }}
{{- $kubevirtmachinetemplateName := printf "%s-%s-%s" $clusterName $groupName $kubevirtmachinetemplateHash }}
{{- $kubevirtmachinetemplateNames = append $kubevirtmachinetemplateNames $kubevirtmachinetemplateName }}
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtMachineTemplate
metadata:
name: {{ $clusterName }}-{{ $groupName }}-{{ $kubevirtmachinetemplateHash }}
namespace: {{ $.Release.Namespace }}
spec:
template:
{{- $kubevirtmachinetemplate | nindent 4 }}
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
name: {{ $clusterName }}-{{ $groupName }}
namespace: {{ $.Release.Namespace }}
annotations:
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "{{ $group.minReplicas }}"
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "{{ $group.maxReplicas }}"
{{- if $effectiveMemory }}
{{- $sysBytes := include "cozy-lib.resources.toFloat" $systemReservedMemory | float64 }}
{{- $kubeBytes := include "cozy-lib.resources.toFloat" $kubeReservedMemory | float64 }}
{{- $totalReservedBytes := addf (addf $sysBytes $kubeBytes) $evictionHardBytes }}
{{- $allocatableBytes := subf (include "cozy-lib.resources.toFloat" $effectiveMemory | float64) $totalReservedBytes }}
{{- $allocatableMi := divf $allocatableBytes 1048576.0 | floor }}
capacity.cluster-autoscaler.kubernetes.io/memory: "{{ $allocatableMi }}Mi"
{{- end }}
{{- if $effectiveCpu }}
{{- $sysCpuMillisA := include "kubernetes.cpuToMillicores" $systemReservedCpu | float64 }}
{{- $kubeCpuMillisA := include "kubernetes.cpuToMillicores" $kubeReservedCpu | float64 }}
{{- $effectiveCpuMillicoresA := include "kubernetes.cpuToMillicores" $effectiveCpu | int }}
{{- $effectiveCpuMillisA := float64 $effectiveCpuMillicoresA }}
{{- $allocatableCpuMillis := subf $effectiveCpuMillisA (addf $sysCpuMillisA $kubeCpuMillisA) }}
{{- $allocatableCpuMillisInt := $allocatableCpuMillis | ceil | int }}
capacity.cluster-autoscaler.kubernetes.io/cpu: "{{ $allocatableCpuMillisInt }}m"
{{- end }}
spec:
clusterName: {{ $clusterName }}
# `spec.replicas` is intentionally omitted so the field is owned by
# the cluster-autoscaler alone. Two independent CAPI mechanisms —
# not Server-Side-Apply field managers — carry the guarantee:
# 1. Create: the MachineDeployment defaulting webhook seeds
# `spec.replicas` from the `cluster-api-autoscaler-node-group-
# min-size` annotation (see
# https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling).
# 2. Update: `calculateMachineDeploymentReplicas` preserves the
# existing live value (`oldMD.Spec.Replicas`) whenever the
# incoming spec has `replicas: nil`, clamping into
# `[min, max]` if it drifts outside — so a cluster the
# autoscaler holds above min-size is not drained by a
# `helm upgrade` that would otherwise re-render this field.
# Flux helm-controller applies this chart through the Helm SDK
# three-way merge (not SSA), so field-manager ownership is not the
# mechanism at play. The chart guarantee reduces to "leave the
# field out of the rendered manifest and let CAPI's update-time
# defaulting keep the live value stable". Emitting a chart-managed
# `replicas: N` here (as this chart did before
# cozystack/cozystack#3231) drains workers on every `helm upgrade`
# action until the autoscaler notices unschedulable Pods and
# scales back up — a real disruption window on every routine
# platform bump.
strategy:
rollingUpdate:
maxSurge: {{ $group.maxReplicas }}
maxUnavailable: 1
type: RollingUpdate
selector:
matchLabels:
cluster.x-k8s.io/cluster-name: {{ $clusterName }}
cluster.x-k8s.io/deployment-name: {{ $clusterName }}-{{ $groupName }}
template:
metadata:
labels:
cluster.x-k8s.io/cluster-name: {{ $clusterName }}
cluster.x-k8s.io/deployment-name: {{ $clusterName }}-{{ $groupName }}
{{- range $group.roles }}
node-role.kubernetes.io/{{ . }}: ""
{{- end }}
spec:
bootstrap:
configRef:
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: TalosConfigTemplate
name: {{ $clusterName }}-{{ $groupName }}
namespace: {{ $.Release.Namespace }}
clusterName: {{ $clusterName }}
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtMachineTemplate
name: {{ $clusterName }}-{{ $groupName }}-{{ $kubevirtmachinetemplateHash }}
namespace: {{ $.Release.Namespace }}
version: {{ include "kubernetes.versionMap" $}}
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineHealthCheck
metadata:
name: {{ $clusterName }}-{{ $groupName }}
namespace: {{ $.Release.Namespace }}
spec:
clusterName: {{ $clusterName }}
{{- /* MHC spec.maxUnhealthy is an IntOrString. CAPI admission accepts a
bare integer (0, 1, ...) or a percentage string ("0%", "50%"), but
rejects a quoted plain integer ("0"). Quote only when the operator
supplied a percentage; otherwise render the value as a bare int so
the admission webhook is happy with the chart-default "0". */}}
{{- $maxUnhealthy := toString $.Values.maxUnhealthy }}
{{- $muSource := "maxUnhealthy" }}
{{- if hasSuffix "%" $maxUnhealthy }}
{{- if not (regexMatch "^[0-9]+%$" $maxUnhealthy) }}
{{- fail (printf "%s must be a bare integer (e.g. 0, 1) or an integer percentage (e.g. 50%%), got %q" $muSource $maxUnhealthy) }}
{{- end }}
maxUnhealthy: {{ $maxUnhealthy | quote }}
{{- else if regexMatch "^[0-9]+$" $maxUnhealthy }}
maxUnhealthy: {{ $maxUnhealthy | int }}
{{- else }}
{{- fail (printf "%s must be a bare integer (e.g. 0, 1) or an integer percentage (e.g. 50%%), got %q" $muSource $maxUnhealthy) }}
{{- end }}
nodeStartupTimeout: {{ $.Values.nodeStartupTimeout }}
selector:
matchLabels:
cluster.x-k8s.io/cluster-name: {{ $clusterName }}
cluster.x-k8s.io/deployment-name: {{ $clusterName }}-{{ $groupName }}
unhealthyConditions:
- type: Ready
status: Unknown
timeout: 30s
- type: Ready
status: "False"
timeout: 300s
---
apiVersion: cozystack.io/v1alpha1
kind: WorkloadMonitor
metadata:
name: {{ $clusterName }}-{{ $groupName }}
namespace: {{ $.Release.Namespace }}
spec:
minReplicas: {{ $group.minReplicas }}
kind: kubernetes
type: worker
selector:
cluster.x-k8s.io/cluster-name: {{ $clusterName }}
cluster.x-k8s.io/deployment-name: {{ $clusterName }}-{{ $groupName }}
cluster.x-k8s.io/role: worker
version: {{ $.Chart.Version }}
---
{{- /*
We must preserve all previous KubevirtMachineTemplates for THIS pool until a
MachineSet stops referencing them. Scoped to this pool's own hash-named KMTs
(prefix kubernetes-<cluster>-<group>-) so sibling kubernetes-nodes releases in
the same namespace never re-emit and fight over each other's templates.
*/ -}}
{{- $mss := (lookup "cluster.x-k8s.io/v1beta1" "MachineSet" $.Release.Namespace "").items }}
{{- $kmtPrefix := printf "%s-%s-" $clusterName $groupName }}
{{- $oldKubevirtmachinetemplates := dict }}
{{- range $kmt := (lookup "infrastructure.cluster.x-k8s.io/v1alpha1" "KubevirtMachineTemplate" $.Release.Namespace "").items }}
{{- range $or := $kmt.metadata.ownerReferences }}
{{- if and (eq $or.kind "Cluster") (eq $or.name $clusterName) (hasPrefix $kmtPrefix $kmt.metadata.name) }}
{{- range $ms := $mss }}
{{- if and (eq $ms.spec.template.spec.infrastructureRef.kind "KubevirtMachineTemplate") (eq $ms.spec.template.spec.infrastructureRef.name $kmt.metadata.name) }}
{{- if not (has $kmt.metadata.name $kubevirtmachinetemplateNames) }}
{{- $oldKubevirtmachinetemplates = merge $oldKubevirtmachinetemplates (dict $kmt.metadata.name $kmt) }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- range $oldKubevirtmachinetemplates }}
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtMachineTemplate
metadata:
name: {{ .metadata.name }}
namespace: {{ .metadata.namespace }}
spec:
{{- .spec | toYaml | nindent 2 }}
{{- end }}
@@ -0,0 +1,423 @@
{{- /*
Main-phase Job (NOT a Helm hook) that produces this pool's TalosConfigTemplate
and patches the parent KamajiControlPlane.spec.network.certSANs with the live
apiserver Service ClusterIP.
The Talos worker machineconfig needs values that only exist AFTER the parent
control-plane release has applied: the apiserver Service ClusterIP (emitted by
Kamaji), the Talos-CA Secret (issued by cert-manager from the parent's Talos
PKI), and the Kubernetes-CA Secret (generated by Kamaji). A Helm `lookup` for
any of these races on every fresh install, so this Job blocks until all four
inputs are observably present, then `kubectl apply`s the TalosConfigTemplate
(idempotent) with an ownerReference to the parent KamajiControlPlane so it is
garbage-collected on the parent's uninstall.
Cross-release ordering. The Job reads objects owned by the PARENT `kubernetes`
release (Service, Secrets, KamajiControlPlane named kubernetes-<cluster>). No
hard per-tenant dependsOn is possible across the two HelmReleases, so ordering
is handled at runtime by this Job's ~10-minute input-wait loop plus the
release.cozystack.io/helm-install-disable-wait annotation on the pool's
ApplicationDefinition.
Churn-free across upgrades. The Job is immutable, so it carries a content-hash
name suffix; an identical render yields the same name and Helm no-ops it. The
applied TalosConfigTemplate keeps a stable name (kubernetes-<cluster>-<group>,
referenced by MachineDeployment.bootstrap.configRef), so an unchanged render
re-applies a byte-identical TCT — a CAPI no-op that never rolls workers.
*/}}
{{- $clusterName := include "kubernetes-nodes.clusterName" . }}
{{- $groupName := include "kubernetes-nodes.groupName" . }}
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ .Release.Name }}-talos-reconcile
namespace: {{ .Release.Namespace }}
labels:
{{- include "kubernetes.labels" . | nindent 4 }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ .Release.Name }}-talos-reconcile
namespace: {{ .Release.Namespace }}
labels:
{{- include "kubernetes.labels" . | nindent 4 }}
rules:
- apiGroups: [""]
resources: ["services"]
resourceNames: ["{{ $clusterName }}"]
verbs: ["get"]
- apiGroups: [""]
resources: ["secrets"]
resourceNames:
- "{{ $clusterName }}-talos-secrets"
- "{{ $clusterName }}-talos-ca"
- "{{ $clusterName }}-ca"
verbs: ["get"]
- apiGroups: ["controlplane.cluster.x-k8s.io"]
resources: ["kamajicontrolplanes"]
resourceNames: ["{{ $clusterName }}"]
verbs: ["get", "patch"]
- apiGroups: ["bootstrap.cluster.x-k8s.io"]
resources: ["talosconfigtemplates"]
verbs: ["get", "create", "patch", "update"]
---
{{- /* The tenant-root chart installs a default-deny egress policy on every
pod in the tenant namespace; without an explicit allow, this Job
cannot reach the management-cluster apiserver to read the Service
ClusterIP / Secrets / KamajiControlPlane it has to reconcile. Cilium
policies are additive, so this rule layers on top of the tenant
default-deny without weakening it for anything else. Endpoint
selector is scoped to pods carrying the reconcile label only. */}}
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: {{ .Release.Name }}-talos-reconcile
namespace: {{ .Release.Namespace }}
labels:
{{- include "kubernetes.labels" . | nindent 4 }}
spec:
endpointSelector:
matchLabels:
cozystack.io/talos-reconcile: "{{ .Release.Name }}"
egress:
- toEntities:
- kube-apiserver
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
"k8s:k8s-app": kube-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
- port: "53"
protocol: TCP
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ .Release.Name }}-talos-reconcile
namespace: {{ .Release.Namespace }}
labels:
{{- include "kubernetes.labels" . | nindent 4 }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ .Release.Name }}-talos-reconcile
subjects:
- kind: ServiceAccount
name: {{ .Release.Name }}-talos-reconcile
namespace: {{ .Release.Namespace }}
---
{{- /* Cluster-scoped read on the management CoreDNS Service so the Job
can look up its ClusterIP at runtime and inject it into the worker
machineconfig as a host-level nameserver. resourceNames pins the
grant to the single well-known Service name kube-dns. */}}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: {{ .Release.Namespace }}-{{ .Release.Name }}-talos-reconcile-coredns
labels:
{{- include "kubernetes.labels" . | nindent 4 }}
rules:
- apiGroups: [""]
resources: ["services"]
resourceNames: ["kube-dns"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: {{ .Release.Namespace }}-{{ .Release.Name }}-talos-reconcile-coredns
labels:
{{- include "kubernetes.labels" . | nindent 4 }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: {{ .Release.Namespace }}-{{ .Release.Name }}-talos-reconcile-coredns
subjects:
- kind: ServiceAccount
name: {{ .Release.Name }}-talos-reconcile
namespace: {{ .Release.Namespace }}
{{- /* Job spec rendered through a named template so the Job name can carry
a content hash of the whole spec. $.Release is the pool release (used for
the SA it runs as and its network-policy label); .clusterName targets the
parent control-plane objects (Service/Secrets/KCP/TCT). */}}
{{- define "kubernetes-nodes.talos-reconcile-job-spec" -}}
spec:
backoffLimit: 30
ttlSecondsAfterFinished: 600
template:
metadata:
labels:
cozystack.io/talos-reconcile: {{ $.Release.Name | quote }}
spec:
restartPolicy: OnFailure
serviceAccountName: {{ $.Release.Name }}-talos-reconcile
containers:
- name: kubectl
image: "{{ .kubectlImage }}"
env:
- name: RELEASE
value: {{ .clusterName | quote }}
- name: NS
value: {{ $.Release.Namespace | quote }}
- name: GROUP_NAME
value: {{ .groupName | quote }}
- name: TALOS_TOKEN
valueFrom:
secretKeyRef:
name: {{ .clusterName }}-talos-secrets
key: token
- name: CLUSTER_ID
valueFrom:
secretKeyRef:
name: {{ .clusterName }}-talos-secrets
key: clusterId
- name: CLUSTER_SECRET
valueFrom:
secretKeyRef:
name: {{ .clusterName }}-talos-secrets
key: clusterSecret
- name: BOOTSTRAP_TOKEN
valueFrom:
secretKeyRef:
name: {{ .clusterName }}-talos-secrets
key: bootstrapToken
command:
- sh
- -ec
- |
echo "[$(date -u +%H:%M:%S)] talos-reconcile/${GROUP_NAME}: waiting for Service ClusterIP, Talos CA, K8s CA, management CoreDNS"
for i in $(seq 1 120); do
SVC_IP=$(kubectl -n "$NS" get svc "$RELEASE" -o jsonpath='{.spec.clusterIP}' 2>/dev/null || true)
TALOS_CA_B64=$(kubectl -n "$NS" get secret "${RELEASE}-talos-ca" -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)
K8S_CA_B64=$(kubectl -n "$NS" get secret "${RELEASE}-ca" -o jsonpath='{.data.ca\.crt}' 2>/dev/null || true)
COREDNS_IP=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}' 2>/dev/null || true)
if [ -n "$SVC_IP" ] && [ -n "$TALOS_CA_B64" ] && [ -n "$K8S_CA_B64" ] && [ -n "$COREDNS_IP" ]; then
break
fi
sleep 5
done
if [ -z "$SVC_IP" ] || [ -z "$TALOS_CA_B64" ] || [ -z "$K8S_CA_B64" ] || [ -z "$COREDNS_IP" ]; then
echo "[$(date -u +%H:%M:%S)] timed out waiting for lookup inputs (svc='$SVC_IP' talos-ca=$([ -n "$TALOS_CA_B64" ] && echo ok) k8s-ca=$([ -n "$K8S_CA_B64" ] && echo ok) coredns='$COREDNS_IP')"
exit 1
fi
echo "[$(date -u +%H:%M:%S)] inputs ready (svc=${SVC_IP} coredns=${COREDNS_IP})"
# Patch KamajiControlPlane.spec.network.certSANs so the apiserver
# TLS cert covers the live Service ClusterIP that Talos workers
# dial. JSON-merge-patch is array-replace semantics, which is what
# we want here — Kamaji owns the full SAN list.
kubectl -n "$NS" patch kamajicontrolplane "$RELEASE" --type=merge --patch "$(
cat <<JSON
{"spec":{"network":{"certSANs":["${RELEASE}.${NS}.svc","${RELEASE}.${NS}.svc.{{ .dnsDomain }}","${SVC_IP}"]}}}
JSON
)"
KCP_UID=$(kubectl -n "$NS" get kamajicontrolplane "$RELEASE" -o jsonpath='{.metadata.uid}')
# Apply the TalosConfigTemplate. ownerReference back to the parent
# KamajiControlPlane makes the standard Kubernetes garbage collector
# clean it up when the parent cluster is deleted.
cat <<EOF | kubectl apply -f -
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: TalosConfigTemplate
metadata:
name: ${RELEASE}-${GROUP_NAME}
namespace: ${NS}
ownerReferences:
- apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
name: ${RELEASE}
uid: ${KCP_UID}
controller: true
blockOwnerDeletion: true
spec:
template:
spec:
generateType: none
talosVersion: "{{ .talosVersion }}"
data: |
version: v1alpha1
persist: true
machine:
type: worker
token: ${TALOS_TOKEN}
network:
nameservers:
- ${COREDNS_IP}
searchDomains:
- svc.{{ .mgmtClusterDomain }}
- {{ .mgmtClusterDomain }}
{{- if ne .mgmtClusterDomain "cluster.local" }}
- svc.cluster.local
- cluster.local
{{- end }}
extraHostEntries:
- ip: ${SVC_IP}
aliases:
- ${RELEASE}.${NS}.svc
- ${RELEASE}.${NS}.svc.{{ .dnsDomain }}
ca:
crt: ${TALOS_CA_B64}
{{- if .group.gpus }}
nodeLabels:
gpu: "on"
{{- end }}
kubelet:
image: ghcr.io/siderolabs/kubelet:{{ .kubeletVersion }}
extraConfig:
systemReserved:
cpu: "{{ .systemReservedCpu }}"
memory: "{{ .systemReservedMemory }}"
kubeReserved:
cpu: "{{ .kubeReservedCpu }}"
memory: "{{ .kubeReservedMemory }}"
evictionHard:
memory.available: "{{ .evictionHardMemory }}"
nodefs.available: "10%"
imagefs.available: "15%"
nodefs.inodesFree: "5%"
evictionSoft:
memory.available: "{{ .evictionSoftMemory }}"
nodefs.available: "15%"
imagefs.available: "20%"
evictionSoftGracePeriod:
memory.available: "1m30s"
nodefs.available: "1m30s"
imagefs.available: "1m30s"
evictionMinimumReclaim:
memory.available: "256Mi"
install:
disk: /dev/vda
image: {{ $.Values.talos.installerRepository | trimSuffix "/" }}/{{ $.Values.talos.schematicID }}:{{ .talosVersion }}
wipe: false
features:
rbac: true
kubePrism:
enabled: false
cluster:
id: ${CLUSTER_ID}
secret: ${CLUSTER_SECRET}
controlPlane:
endpoint: https://${RELEASE}.${NS}.svc:6443
clusterName: ${RELEASE}
network:
dnsDomain: {{ .dnsDomain }}
podSubnets:
- {{ .podCIDR }}
serviceSubnets:
- {{ .serviceCIDR }}
token: ${BOOTSTRAP_TOKEN}
ca:
crt: ${K8S_CA_B64}
discovery:
enabled: true
registries:
kubernetes:
disabled: true
service:
disabled: true
EOF
echo "[$(date -u +%H:%M:%S)] TalosConfigTemplate ${RELEASE}-${GROUP_NAME} applied"
resources:
requests:
cpu: 10m
memory: 64Mi
limits:
cpu: 100m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65532
seccompProfile:
type: RuntimeDefault
{{- end }}
{{- $kubeletVersion := include "kubernetes.versionMap" $ | trim }}
{{- $talosVersion := $.Values.talos.version }}
{{- $podCIDR := "10.243.0.0/16" }}
{{- $serviceCIDR := "10.95.0.0/16" }}
{{- $dnsDomain := include "kubernetes.tenantClusterDomain" $ }}
{{- $mgmtClusterDomain := (index (.Values._cluster | default dict) "cluster-domain") | default "cozy.local" }}
{{- $group := dict
"instanceType" .Values.instanceType
"resources" .Values.resources
"gpus" .Values.gpus
"kubelet" .Values.kubelet
}}
{{- /* Mirror the kubelet-reservation computation from nodegroup.yaml so the
worker machineconfig this Job applies carries the same numbers the
MachineDeployment validator was happy with. */}}
{{- $instanceType := dict }}
{{- if $group.instanceType }}
{{- $instanceType = lookup "instancetype.kubevirt.io/v1beta1" "VirtualMachineClusterInstancetype" "" $group.instanceType }}
{{- $instanceType = $instanceType | default dict }}
{{- end }}
{{- $effectiveMemory := "" }}
{{- if and $group.resources $group.resources.memory }}
{{- $effectiveMemory = $group.resources.memory | toString }}
{{- else if and $instanceType $instanceType.spec $instanceType.spec.memory $instanceType.spec.memory.guest }}
{{- $effectiveMemory = $instanceType.spec.memory.guest | toString }}
{{- end }}
{{- $effectiveCpu := "" }}
{{- if and $group.resources $group.resources.cpu }}
{{- $effectiveCpu = $group.resources.cpu | toString }}
{{- else if and $instanceType $instanceType.spec $instanceType.spec.cpu $instanceType.spec.cpu.guest }}
{{- $effectiveCpu = $instanceType.spec.cpu.guest | toString }}
{{- end }}
{{- $autoReservedMi := 256 }}
{{- if $effectiveMemory }}
{{- $effectiveMemMi := divf (include "cozy-lib.resources.toFloat" $effectiveMemory | float64) 1048576.0 | int }}
{{- $fivePercentMi := mulf ($effectiveMemMi | float64) 0.05 | int }}
{{- $autoReservedMi = min (max $fivePercentMi 256) 1024 }}
{{- end }}
{{- $autoReservedMillicores := 50 }}
{{- if $effectiveCpu }}
{{- $effectiveCpuMillicores := include "kubernetes.cpuToMillicores" $effectiveCpu | int }}
{{- $fivePercentMillis := mulf ($effectiveCpuMillicores | float64) 0.05 | int }}
{{- $autoReservedMillicores = min (max $fivePercentMillis 50) 500 }}
{{- end }}
{{- $kubeletOverride := $group.kubelet | default dict }}
{{- $systemReservedMemory := $kubeletOverride.systemReservedMemory | default (printf "%dMi" $autoReservedMi) }}
{{- $kubeReservedMemory := $kubeletOverride.kubeReservedMemory | default (printf "%dMi" $autoReservedMi) }}
{{- $systemReservedCpu := $kubeletOverride.systemReservedCpu | default (printf "%dm" $autoReservedMillicores) }}
{{- $kubeReservedCpu := $kubeletOverride.kubeReservedCpu | default (printf "%dm" $autoReservedMillicores) }}
{{- $evictionHardMemory := $kubeletOverride.evictionHardMemory | default "7%" }}
{{- $evictionSoftMemory := $kubeletOverride.evictionSoftMemory | default "10%" }}
{{- $jobContext := deepCopy $ }}
{{- $_ := set $jobContext "group" $group }}
{{- $_ := set $jobContext "groupName" $groupName }}
{{- $_ := set $jobContext "clusterName" $clusterName }}
{{- $_ := set $jobContext "kubeletVersion" $kubeletVersion }}
{{- $_ := set $jobContext "talosVersion" $talosVersion }}
{{- $_ := set $jobContext "podCIDR" $podCIDR }}
{{- $_ := set $jobContext "serviceCIDR" $serviceCIDR }}
{{- $_ := set $jobContext "dnsDomain" $dnsDomain }}
{{- $_ := set $jobContext "mgmtClusterDomain" $mgmtClusterDomain }}
{{- $_ := set $jobContext "systemReservedCpu" $systemReservedCpu }}
{{- $_ := set $jobContext "systemReservedMemory" $systemReservedMemory }}
{{- $_ := set $jobContext "kubeReservedCpu" $kubeReservedCpu }}
{{- $_ := set $jobContext "kubeReservedMemory" $kubeReservedMemory }}
{{- $_ := set $jobContext "evictionHardMemory" $evictionHardMemory }}
{{- $_ := set $jobContext "evictionSoftMemory" $evictionSoftMemory }}
{{- $_ := set $jobContext "kubectlImage" (default ($.Files.Get "images/kubectl.tag" | trim) $.Values.images.kubectl) }}
{{- $jobSpec := include "kubernetes-nodes.talos-reconcile-job-spec" $jobContext }}
{{- $jobHash := $jobSpec | sha256sum | trunc 6 }}
{{- $jobName := printf "%s-talos-reconcile-%s" $.Release.Name $groupName | trunc 56 | trimSuffix "-" }}
---
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $jobName }}-{{ $jobHash }}
namespace: {{ $.Release.Namespace }}
labels:
{{- include "kubernetes.labels" $ | nindent 4 }}
{{ $jobSpec }}
@@ -0,0 +1,215 @@
{
"title": "Chart Values",
"type": "object",
"properties": {
"cluster": {
"description": "Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-\u003ccluster\u003e`; the KubernetesNodes CR must be named `\u003ccluster\u003e-\u003cpool\u003e`.",
"type": "string",
"default": "",
"x-cozystack-options": {
"source": "kubernetes"
}
},
"storageClass": {
"description": "StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class.",
"type": "string",
"default": "replicated",
"x-cozystack-options": {
"source": "storageclass"
}
},
"minReplicas": {
"description": "Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.",
"type": "integer",
"default": 0
},
"maxReplicas": {
"description": "Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.",
"type": "integer",
"default": 10
},
"instanceType": {
"description": "Virtual machine instance type.",
"type": "string",
"default": "u1.medium",
"x-cozystack-options": {
"source": "instancetype"
}
},
"diskSize": {
"description": "System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs.",
"default": "20Gi",
"pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$",
"anyOf": [
{
"type": "integer"
},
{
"type": "string"
}
],
"x-kubernetes-int-or-string": true
},
"roles": {
"description": "List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/\u003cr\u003e`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.",
"type": "array",
"default": [],
"items": {
"type": "string"
}
},
"resources": {
"description": "Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time.",
"type": "object",
"default": {},
"properties": {
"cpu": {
"description": "CPU available.",
"pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$",
"anyOf": [
{
"type": "integer"
},
{
"type": "string"
}
],
"x-kubernetes-int-or-string": true
},
"memory": {
"description": "Memory (RAM) available.",
"pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$",
"anyOf": [
{
"type": "integer"
},
{
"type": "string"
}
],
"x-kubernetes-int-or-string": true
}
}
},
"gpus": {
"description": "List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).",
"type": "array",
"default": [],
"items": {
"type": "object",
"required": [
"name"
],
"properties": {
"name": {
"description": "Name of GPU, such as \"nvidia.com/AD102GL_L40S\".",
"type": "string",
"x-cozystack-options": {
"source": "gpu"
}
}
}
}
},
"kubelet": {
"description": "Kubelet resource reservations for this pool.",
"type": "object",
"default": {},
"properties": {
"evictionHardMemory": {
"description": "Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).",
"type": "string",
"default": "7%"
},
"evictionSoftMemory": {
"description": "Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).",
"type": "string",
"default": "10%"
},
"kubeReservedCpu": {
"description": "CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.",
"type": "string"
},
"kubeReservedMemory": {
"description": "Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.",
"type": "string"
},
"systemReservedCpu": {
"description": "CPU reserved for host OS. Auto-computed from instanceType if empty.",
"type": "string"
},
"systemReservedMemory": {
"description": "Memory reserved for host OS. Auto-computed from instanceType if empty.",
"type": "string"
}
}
},
"maxUnhealthy": {
"description": "Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer (\"0\", \"1\", ...) or a percentage (\"0%\", \"50%\"); bare numeric strings are rejected, so the safer default is a percentage. Drop to \"0%\" once the pool is stable.",
"type": "string",
"default": "50%"
},
"nodeStartupTimeout": {
"description": "Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator).",
"type": "string",
"default": "10m"
},
"version": {
"description": "Kubernetes major.minor version the pool joins. Must match the parent cluster's version and satisfy the Talos\u003c-\u003eKubernetes support matrix against `talos.version`.",
"type": "string",
"default": "v1.35",
"enum": [
"v1.35",
"v1.34",
"v1.33",
"v1.32",
"v1.31"
]
},
"talos": {
"description": "Talos worker image configuration. Keep in sync with the parent cluster's `talos`.",
"type": "object",
"default": {},
"required": [
"imageFactoryURL",
"installerRepository",
"schematicID",
"version"
],
"properties": {
"imageFactoryURL": {
"description": "Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash.",
"type": "string",
"default": "https://factory.talos.dev"
},
"installerRepository": {
"description": "OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `\u003cinstallerRepository\u003e/\u003cschematicID\u003e:\u003cversion\u003e`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.",
"type": "string",
"default": "factory.talos.dev/installer"
},
"schematicID": {
"description": "Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here.",
"type": "string",
"default": "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
},
"version": {
"description": "Talos release used for worker OS image and installer. Must satisfy the chart's Talos\u003c-\u003eKubernetes support matrix against the chosen `version`.",
"type": "string",
"default": "v1.13.6"
}
}
},
"images": {
"description": "Optional image overrides for air-gapped or rate-limited registries.",
"type": "object",
"default": {},
"properties": {
"kubectl": {
"description": "Image used by the talos-reconcile Job (kubectl). Empty falls back to images/kubectl.tag.",
"type": "string",
"default": ""
}
}
}
}
}
+108
View File
@@ -0,0 +1,108 @@
##
## @section Linkage
##
## @param {string} cluster - Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`.
## @x-cozystack-options {source: kubernetes}
cluster: ""
##
## @section Common Parameters
##
## @param {string} storageClass - StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class.
## @x-cozystack-options {source: storageclass}
storageClass: replicated
##
## @section Pool Sizing
##
## @typedef {struct} Resources - Explicit CPU and memory configuration for a node.
## @field {quantity} [cpu] - CPU available.
## @field {quantity} [memory] - Memory (RAM) available.
## @typedef {struct} GPU - GPU configuration.
## @field {string} name - Name of GPU, such as "nvidia.com/AD102GL_L40S".
## @x-cozystack-options {source: gpu}
## @typedef {struct} Kubelet - Kubelet resource reservations for worker nodes.
## @field {string} [systemReservedMemory] - Memory reserved for host OS. Auto-computed from instanceType if empty.
## @field {string} [kubeReservedMemory] - Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.
## @field {string} [systemReservedCpu] - CPU reserved for host OS. Auto-computed from instanceType if empty.
## @field {string} [kubeReservedCpu] - CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.
## @field {string} [evictionHardMemory]="7%" - Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).
## @field {string} [evictionSoftMemory]="10%" - Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).
## @param {int} minReplicas=0 - Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.
minReplicas: 0
## @param {int} maxReplicas=10 - Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.
maxReplicas: 10
## @param {string} instanceType="u1.medium" - Virtual machine instance type.
## @x-cozystack-options {source: instancetype}
instanceType: u1.medium
## @param {quantity} diskSize="20Gi" - System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs.
diskSize: 20Gi
## @param {[]string} roles - List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.
roles: []
## @param {Resources} [resources] - Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time.
resources: {}
## @param {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
gpus: []
## @param {Kubelet} [kubelet] - Kubelet resource reservations for this pool.
kubelet: {}
## @param {string} maxUnhealthy="50%" - Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable.
maxUnhealthy: "50%"
## @param {string} nodeStartupTimeout="10m" - Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator).
nodeStartupTimeout: "10m"
##
## @section Kubernetes Version
##
## @enum {string} Version
## @value v1.35
## @value v1.34
## @value v1.33
## @value v1.32
## @value v1.31
## @param {Version} version - Kubernetes major.minor version the pool joins. Must match the parent cluster's version and satisfy the Talos<->Kubernetes support matrix against `talos.version`.
version: "v1.35"
##
## @section Talos Worker Image
##
## @typedef {struct} Talos - Talos worker image configuration.
## @field {string} version - Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`.
## @field {string} schematicID - Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here.
## @field {string} imageFactoryURL - Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash.
## @field {string} installerRepository - OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.
## @param {Talos} talos - Talos worker image configuration. Keep in sync with the parent cluster's `talos`.
talos:
version: "v1.13.6"
schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
imageFactoryURL: "https://factory.talos.dev"
installerRepository: "factory.talos.dev/installer"
##
## @section Images
##
## @typedef {struct} Images - Optional image overrides for chart-internal helpers.
## @field {string} [kubectl] - Image used by the talos-reconcile Job (kubectl). Empty falls back to images/kubectl.tag.
## @param {Images} images - Optional image overrides for air-gapped or rate-limited registries.
images:
kubectl: ""