mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
fix(kubernetes)!: remove the talos block and other leftovers of the worker pool split (#5191)
## What this PR does Cleanup after the Kubernetes app was split into `Kubernetes` (control plane and addons) and `KubernetesNodes` (one release per worker pool): what the split left behind in values, docs, templates and tests. ### Kubernetes The `Kubernetes` application has no `talos` field any more. Worker pools are `KubernetesNodes` resources and take their Talos release, image, installer and registry mirrors from their own `talos` and `osImage`, so the cluster's copies did nothing. The last one, `talos.version`, fed a duplicate of the Talos to Kubernetes support-matrix check. The pool chart already runs that check against the Talos release each pool actually boots, including one picked through an `osImage` override; the cluster's copy compared the Kubernetes version against a release no node runs. A stored spec that still carries `talos.*` keeps rendering, because the values schema does not close the object and the API does not validate `spec` against it, and a unit test pins that. The pool's check now has render coverage. Its own suite pins the pairings it must accept, and a bats test renders the refusals through `talos.version`, `osImage.factory.version` and `osImage.builtin.version` with schema validation skipped, since the schema offers no Kubernetes version the matrix refuses. I fixed the README and field descriptions that described behavior the chart no longer has, or never had: a dedicated etcd cluster, a `blockSize.matchVolume` setting that does not exist, a default `md0` pool, the host default, the substrate mismatch behavior, and rollout phase names. The README's Breaking Changes section becomes a pointer to the per-release upgrade notes on cozystack.io. Unused helpers and stale comments are removed; the chart renders the same bytes. ### KubernetesNodes Five field descriptions and the `talos` block told users to keep them in sync with parent fields that do not exist. NOTES warned about colliding with a parent-managed `md0` pool, although the parent renders no pools and `md0` is the normal name of a pool. The installer image was credited to an upgrade Job that never runs one. The values descriptions, README intro and NOTES now say what the chart renders today, including which fields have no effect on `substrate: proxmox`: `storageClass`, `diskSize`, `instanceType`, `talos.imageFactoryURL` (beyond its format check) and `talos.imageFactoryCA`. The render error for a pool whose MachineDeployment belongs to another release told users to remove the pool from the parent's `nodeGroups`, which cannot clear the conflict. It now names a remedy that works. Split-era history is gone from Helm and test comments, along with two `storageClass` fallbacks that can only return the value they fall back from, a constant variable, and two `.helmignore` rules that match nothing. The tenant pod and service CIDRs are literals in both charts, and nothing checked that they agree. A new bats test renders both charts and fails when they differ. No rendered manifest of the pool chart changes. I rendered it for the snapshot values, GPU, kubelet-override, factory/CA, storageClass variants and three Proxmox variants on main and on every commit, and the output is byte-identical. The snapshot and TalosConfigTemplate hash tests pass unchanged, so no worker rolls. YAML comments inside the hashed machine config and machine template are left as they are for that reason. ### The rule `docs/agents/overview.md` gains a rule so this does not come back with the next change: app READMEs and field descriptions describe current behavior, a change users must act on goes to the website's upgrade page, and a description does not ask the reader to keep it in sync with another chart. ```release-note [kubernetes] The Kubernetes application no longer has a `talos` field. Talos settings for workers, and the Talos to Kubernetes compatibility check, live on each KubernetesNodes pool. Existing clusters that still store `talos` values keep working; the values are ignored. KubernetesNodes field descriptions, README and install notes now describe the worker pool as it works today, including which fields have no effect on Proxmox pools. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Configuration** * Worker OS image, installer, and registry-mirror settings are configured per worker pool; these settings are no longer available at the cluster level. * Kubernetes and Talos version compatibility is checked for each worker pool. A worker Kubernetes version cannot be ahead of its parent cluster. * Worker pools support KubeVirt and Proxmox, with pool-specific sizing and disk settings. * Rendering is refused when a pool uses a different substrate from its cluster or conflicts with an existing worker pool owned by another release. * The default cluster hostname now includes the `kubernetes-` prefix. * Existing cluster configurations that still include Talos settings receive a warning; those settings are no longer used. * **Documentation** * Updated guidance for cluster and worker-pool settings, storage, upgrades, and version compatibility. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
53 files changed
+686
-713
No files matched your search
Generated
+3
-24
@@ -25,10 +25,10 @@ type ConfigSpec struct {
|
||||
// Kubernetes major.minor version to deploy
|
||||
// +kubebuilder:default:="v1.35"
|
||||
Version Version `json:"version"`
|
||||
// External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty.
|
||||
// External hostname for Kubernetes cluster. Defaults to `kubernetes-<cluster-name>.<tenant-host>` if empty.
|
||||
// +kubebuilder:default:=""
|
||||
Host string `json:"host"`
|
||||
// Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.
|
||||
// Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one.
|
||||
// +kubebuilder:default:="kubevirt"
|
||||
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="substrate is immutable"
|
||||
Substrate string `json:"substrate"`
|
||||
@@ -41,9 +41,6 @@ type ConfigSpec struct {
|
||||
// Kubernetes control-plane configuration.
|
||||
// +kubebuilder:default:={}
|
||||
ControlPlane ControlPlane `json:"controlPlane"`
|
||||
// Talos worker image configuration.
|
||||
// +kubebuilder:default:={}
|
||||
Talos Talos `json:"talos"`
|
||||
// OIDC authentication and per-user RBAC for the tenant kube-apiserver. See https://cozystack.io/docs/v1.6/kubernetes/oidc-authentication/ for the operator guide.
|
||||
// +kubebuilder:default:={}
|
||||
Oidc OIDC `json:"oidc"`
|
||||
@@ -53,7 +50,7 @@ type APIServer struct {
|
||||
// Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it.
|
||||
// +kubebuilder:default:={}
|
||||
AdmissionControllers []AdmissionController `json:"admissionControllers,omitempty"`
|
||||
// Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior).
|
||||
// Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default.
|
||||
// +kubebuilder:default:={}
|
||||
ExtraArgs []string `json:"extraArgs,omitempty"`
|
||||
// Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default.
|
||||
@@ -349,24 +346,6 @@ type Scheduler struct {
|
||||
ResourcesPreset ResourcesPreset `json:"resourcesPreset"`
|
||||
}
|
||||
|
||||
type Talos struct {
|
||||
// Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash.
|
||||
// +kubebuilder:default:="https://factory.talos.dev"
|
||||
ImageFactoryURL string `json:"imageFactoryURL"`
|
||||
// OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.
|
||||
// +kubebuilder:default:="factory.talos.dev/installer"
|
||||
InstallerRepository string `json:"installerRepository"`
|
||||
// Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap.
|
||||
// +kubebuilder:default:={}
|
||||
RegistryMirrors k8sRuntime.RawExtension `json:"registryMirrors"`
|
||||
// Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here.
|
||||
// +kubebuilder:default:="ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
|
||||
SchematicID string `json:"schematicID"`
|
||||
// Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`.
|
||||
// +kubebuilder:default:="v1.13.6"
|
||||
Version string `json:"version"`
|
||||
}
|
||||
|
||||
type VeleroAddon struct {
|
||||
// Enable Velero.
|
||||
// +kubebuilder:default:=false
|
||||
|
||||
@@ -154,7 +154,6 @@ func (in *ConfigSpec) DeepCopyInto(out *ConfigSpec) {
|
||||
in.Proxmox.DeepCopyInto(&out.Proxmox)
|
||||
in.Addons.DeepCopyInto(&out.Addons)
|
||||
in.ControlPlane.DeepCopyInto(&out.ControlPlane)
|
||||
in.Talos.DeepCopyInto(&out.Talos)
|
||||
in.Oidc.DeepCopyInto(&out.Oidc)
|
||||
}
|
||||
|
||||
@@ -570,22 +569,6 @@ func (in *Scheduler) DeepCopy() *Scheduler {
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *Talos) DeepCopyInto(out *Talos) {
|
||||
*out = *in
|
||||
in.RegistryMirrors.DeepCopyInto(&out.RegistryMirrors)
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Talos.
|
||||
func (in *Talos) DeepCopy() *Talos {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(Talos)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *VeleroAddon) DeepCopyInto(out *VeleroAddon) {
|
||||
*out = *in
|
||||
|
||||
+23
-23
@@ -18,11 +18,11 @@ type Config struct {
|
||||
}
|
||||
|
||||
type ConfigSpec struct {
|
||||
// Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool name (the part after `<cluster>-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.
|
||||
// Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool's objects are named `kubernetes-<cluster>-<pool>`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over.
|
||||
// +kubebuilder:default:=""
|
||||
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="cluster is immutable"
|
||||
Cluster string `json:"cluster"`
|
||||
// StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.
|
||||
// StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.
|
||||
// +kubebuilder:default:="replicated"
|
||||
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="storageClass is immutable"
|
||||
StorageClass string `json:"storageClass"`
|
||||
@@ -32,10 +32,10 @@ type ConfigSpec struct {
|
||||
// Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.
|
||||
// +kubebuilder:default:=10
|
||||
MaxReplicas int `json:"maxReplicas"`
|
||||
// Virtual machine instance type.
|
||||
// KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them.
|
||||
// +kubebuilder:default:="u1.medium"
|
||||
InstanceType string `json:"instanceType"`
|
||||
// System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.
|
||||
// System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size.
|
||||
// +kubebuilder:default:="20Gi"
|
||||
DiskSize resource.Quantity `json:"diskSize"`
|
||||
// List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.
|
||||
@@ -44,16 +44,16 @@ type ConfigSpec struct {
|
||||
// Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `"010"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `"010"` reserves what `"10"` does.
|
||||
// +kubebuilder:default:={}
|
||||
Resources Resources `json:"resources,omitempty"`
|
||||
// CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.
|
||||
// CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content.
|
||||
PodCpuLimit resource.Quantity `json:"podCpuLimit,omitempty"`
|
||||
// CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.
|
||||
// CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content.
|
||||
PodCpuRequest resource.Quantity `json:"podCpuRequest,omitempty"`
|
||||
// List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
|
||||
// +kubebuilder:default:={}
|
||||
Gpus []GPU `json:"gpus,omitempty"`
|
||||
// Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.
|
||||
// Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.
|
||||
KernelModules []KernelModule `json:"kernelModules,omitempty"`
|
||||
// Kubelet resource reservations for this pool.
|
||||
// Kubelet resource reservations for this pool. An empty reservation is computed from the worker's size: `resources` when set, otherwise the `instanceType`.
|
||||
// +kubebuilder:default:={}
|
||||
Kubelet Kubelet `json:"kubelet,omitempty"`
|
||||
// Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.
|
||||
@@ -62,25 +62,25 @@ type ConfigSpec struct {
|
||||
// Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At "50%" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two.
|
||||
// +kubebuilder:default:="50%"
|
||||
MaxUnhealthy string `json:"maxUnhealthy"`
|
||||
// Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.
|
||||
// Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (the Talos image import from the Image Factory, or a storage class slow to provision or clone the worker disk). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.
|
||||
// +kubebuilder:default:="20m"
|
||||
NodeStartupTimeout string `json:"nodeStartupTimeout"`
|
||||
// How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.
|
||||
// How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips.
|
||||
// +kubebuilder:default:="5m"
|
||||
ReadyUnknownTimeout string `json:"readyUnknownTimeout"`
|
||||
// How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.
|
||||
// How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips.
|
||||
// +kubebuilder:default:="10m"
|
||||
ReadyFalseTimeout string `json:"readyFalseTimeout"`
|
||||
// How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted. Keep in sync with the parent kubernetes chart.
|
||||
// How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted.
|
||||
// +kubebuilder:default:="30m"
|
||||
MaxNodeProvisionTime string `json:"maxNodeProvisionTime"`
|
||||
// Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`.
|
||||
// Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool. The render fails when it is ahead. Its pairing with the Talos release the pool resolves (`talos.version`, or the `osImage` override) is checked against the chart's Talos<->Kubernetes support matrix: for a Talos minor the matrix lists, a Kubernetes minor outside that minor's list is refused; a Talos minor the matrix does not list is not checked.
|
||||
// +kubebuilder:default:="v1.35"
|
||||
Version Version `json:"version"`
|
||||
// Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool.
|
||||
// +kubebuilder:default:={}
|
||||
OsImage WorkerImage `json:"osImage,omitempty"`
|
||||
// Talos worker image configuration. Keep in sync with the parent cluster's `talos`.
|
||||
// Talos worker image configuration for this pool.
|
||||
// +kubebuilder:default:={}
|
||||
Talos Talos `json:"talos"`
|
||||
// Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is refused: the chart reads the live MachineDeployment's template kind and fails the render rather than rolling every worker to the other hypervisor and deleting the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down.
|
||||
@@ -111,13 +111,13 @@ type Kubelet struct {
|
||||
// Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).
|
||||
// +kubebuilder:default:="10%"
|
||||
EvictionSoftMemory string `json:"evictionSoftMemory,omitempty"`
|
||||
// CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.
|
||||
// CPU reserved for kubelet and container runtime. When empty, 5% of the worker's CPU, clamped to 50m..500m.
|
||||
KubeReservedCpu string `json:"kubeReservedCpu,omitempty"`
|
||||
// Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.
|
||||
// Memory reserved for kubelet and container runtime. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi.
|
||||
KubeReservedMemory string `json:"kubeReservedMemory,omitempty"`
|
||||
// CPU reserved for host OS. Auto-computed from instanceType if empty.
|
||||
// CPU reserved for host OS. When empty, 5% of the worker's CPU, clamped to 50m..500m.
|
||||
SystemReservedCpu string `json:"systemReservedCpu,omitempty"`
|
||||
// Memory reserved for host OS. Auto-computed from instanceType if empty.
|
||||
// Memory reserved for host OS. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi.
|
||||
SystemReservedMemory string `json:"systemReservedMemory,omitempty"`
|
||||
}
|
||||
|
||||
@@ -160,13 +160,13 @@ type Resources struct {
|
||||
}
|
||||
|
||||
type Talos struct {
|
||||
// Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there.
|
||||
// Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. On `substrate: proxmox` it has no effect on the workers either: the ConfigMap is still rendered, but nothing references it.
|
||||
// +kubebuilder:default:=""
|
||||
ImageFactoryCA string `json:"imageFactoryCA,omitempty"`
|
||||
// Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal.
|
||||
// Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. On `substrate: proxmox` the workers clone a VM template that already carries the OS, so this field has no effect there beyond that format check.
|
||||
// +kubebuilder:default:="https://factory.talos.dev"
|
||||
ImageFactoryURL string `json:"imageFactoryURL"`
|
||||
// OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.
|
||||
// OCI repository prefix for the Talos installer image, written into the worker machine config as `machine.install.image`. Resolved as `<installerRepository>/<schematicID>:<version>` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.
|
||||
// +kubebuilder:default:="factory.talos.dev/installer"
|
||||
InstallerRepository string `json:"installerRepository"`
|
||||
// Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.
|
||||
@@ -175,7 +175,7 @@ type Talos struct {
|
||||
// Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID.
|
||||
// +kubebuilder:default:="ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
|
||||
SchematicID string `json:"schematicID"`
|
||||
// Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.
|
||||
// Default Talos release for this pool's worker OS image and for the installer image written into the worker machine config. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix lists, for each Talos minor the chart knows, the Kubernetes minors it supports and refuses a pool `version` outside that list; a Talos minor it does not list passes unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.
|
||||
// +kubebuilder:default:="v1.13.6"
|
||||
Version string `json:"version"`
|
||||
}
|
||||
@@ -195,7 +195,7 @@ type WorkerImageBuiltin struct {
|
||||
}
|
||||
|
||||
type WorkerImageFactory struct {
|
||||
// Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.
|
||||
// Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. This redirects only the worker boot-disk import; the installer image in the worker machine config is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.
|
||||
ImageFactoryURL string `json:"imageFactoryURL,omitempty"`
|
||||
// Talos schematic ID. Defaults to `talos.schematicID`.
|
||||
SchematicID string `json:"schematicID,omitempty"`
|
||||
|
||||
@@ -158,6 +158,8 @@ When a document could go in either `docs/` or the website, the test is whether a
|
||||
|
||||
Documenting a feature's current limitations is useful and belongs with the feature. Documenting the *development process* around it does not: no "What is NOT in Phase N", no "out of scope for this PR", no "this PR" phrasing, no action plans or review negotiation. A merged document describes what the code does now, and states a limitation as a limitation rather than as a roadmap position. Scope, phasing and follow-ups belong in the pull request description or an issue.
|
||||
|
||||
An app's README and the field descriptions in its `values.yaml` describe what the app does now, for the person using it: no history ("moves from", "previously", "used to", rollout phase names) and no account of how another chart behaved before. A change users must act on is announced by a `!` in the pull request title and described on the website's per-version upgrade page (`content/en/docs/next/operations/cluster/upgrade-to-vX.Y.md`), so an app README carries no history or breaking-changes section. A description says what the field does, what it accepts and what it refuses, not what it replaced, and does not tell the reader to keep it in sync with another chart; when two charts have to agree, a test says so. A change that touches a README or a description fixes what is wrong in that place while it is there.
|
||||
|
||||
A document that names an unmerged pull request or an unreleased dependency needs a tracking issue, because nothing re-checks that claim later and it silently becomes false once the thing ships.
|
||||
|
||||
### Domain references for specific tasks
|
||||
|
||||
@@ -60,7 +60,7 @@ git diff --name-only $PREV..$NEW -- 'packages/*/charts/*/Chart.yaml' 'packages/*
|
||||
done
|
||||
|
||||
# B. Out-of-Chart.yaml pins (Talos, tenant K8s, vendored images)
|
||||
for path in packages/core/talos/images/talos/profiles/installer.yaml packages/core/installer/values.yaml packages/apps/kubernetes/values.yaml; do
|
||||
for path in packages/core/talos/images/talos/profiles/installer.yaml packages/core/installer/values.yaml packages/apps/kubernetes/values.yaml packages/apps/kubernetes-nodes/values.yaml; do
|
||||
git diff $PREV..$NEW -- "$path" | head -40
|
||||
done
|
||||
git diff --name-only $PREV..$NEW -- 'images/*/Dockerfile'
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
#!/usr/bin/env bats
|
||||
# -----------------------------------------------------------------------------
|
||||
# The tenant cluster's pod and service CIDRs are written by two charts:
|
||||
#
|
||||
# * packages/apps/kubernetes renders them into the CAPI Cluster's
|
||||
# spec.clusterNetwork, which the control plane is built from;
|
||||
# * packages/apps/kubernetes-nodes renders them into every worker's Talos
|
||||
# machine config (cluster.network.podSubnets / serviceSubnets), inside the
|
||||
# TalosConfigTemplate the talos-reconcile Job applies.
|
||||
#
|
||||
# Each chart carries its own literal, and neither render fails when the two
|
||||
# disagree: the workers boot, join, and route pod and Service traffic over
|
||||
# ranges the control plane does not use. This guard renders both charts and
|
||||
# compares the lists, so a range moved on one side only fails here.
|
||||
#
|
||||
# The worker side is read the way the Job produces it: the heredoc is cut out
|
||||
# of the Job's command, run through sh, and the TalosConfigTemplate it prints is
|
||||
# parsed, then the machine config inside it. A text grep of the Job would also
|
||||
# match a comment.
|
||||
#
|
||||
# Requires: helm, yq (mikefarah v4+). Runs under both `bats` and the in-repo
|
||||
# cozytest.sh runner, so TMP is provisioned inline per test and cleaned without
|
||||
# an EXIT trap.
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
load test_helper
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
|
||||
|
||||
# $1: output directory. Writes pods and services, one CIDR per line.
|
||||
cluster_cidrs() {
|
||||
helm template kubernetes-myk8s "$REPO_ROOT/packages/apps/kubernetes" \
|
||||
--namespace tenant-test \
|
||||
-f "$REPO_ROOT/packages/apps/kubernetes/tests/values-ci.yaml" \
|
||||
> "$1/cluster.yaml" 2> "$1/cluster.err" \
|
||||
|| { cat "$1/cluster.err" >&2; return 1; }
|
||||
yq 'select(.kind == "Cluster" and .apiVersion == "cluster.x-k8s.io/v1beta1") | .spec.clusterNetwork.pods.cidrBlocks[]' \
|
||||
"$1/cluster.yaml" > "$1/cluster.pods"
|
||||
yq 'select(.kind == "Cluster" and .apiVersion == "cluster.x-k8s.io/v1beta1") | .spec.clusterNetwork.services.cidrBlocks[]' \
|
||||
"$1/cluster.yaml" > "$1/cluster.services"
|
||||
}
|
||||
|
||||
# $1: output directory. Writes pool.pods and pool.services, one CIDR per line.
|
||||
pool_cidrs() {
|
||||
helm template kubernetes-nodes-myk8s-md0 "$REPO_ROOT/packages/apps/kubernetes-nodes" \
|
||||
--namespace tenant-test \
|
||||
--set cluster=myk8s \
|
||||
--set _cluster.cluster-domain=cozy.local \
|
||||
--set instanceType="" \
|
||||
--set resources.cpu=2 --set resources.memory=4Gi \
|
||||
--show-only templates/talos-reconcile-job.yaml \
|
||||
> "$1/pool.yaml" 2> "$1/pool.err" \
|
||||
|| { cat "$1/pool.err" >&2; return 1; }
|
||||
yq 'select(.kind == "Job") | .spec.template.spec.containers[0].command[2]' "$1/pool.yaml" > "$1/cmd.sh"
|
||||
awk '
|
||||
/(^|="?\$\()cat <<EOF/ { print "cat <<EOF"; inblock=1; next }
|
||||
inblock && /^EOF$/ { print "EOF"; inblock=0; next }
|
||||
inblock { print }
|
||||
' "$1/cmd.sh" > "$1/heredoc.sh"
|
||||
sh "$1/heredoc.sh" > "$1/tct.yaml" 2> "$1/heredoc.err" \
|
||||
|| { cat "$1/heredoc.err" >&2; return 1; }
|
||||
yq 'select(.kind == "TalosConfigTemplate") | .spec.template.spec.data' "$1/tct.yaml" > "$1/machineconfig.yaml"
|
||||
yq '.cluster.network.podSubnets[]' "$1/machineconfig.yaml" > "$1/pool.pods"
|
||||
yq '.cluster.network.serviceSubnets[]' "$1/machineconfig.yaml" > "$1/pool.services"
|
||||
}
|
||||
|
||||
# $1: directory, $2: pods or services, $3: what to call it in a message.
|
||||
assert_agree() {
|
||||
cluster=$(cat "$1/cluster.$2")
|
||||
pool=$(cat "$1/pool.$2")
|
||||
# An empty read means a field moved or a render came out short; comparing
|
||||
# "" with "" would pass on nothing.
|
||||
if [ -z "$cluster" ] || [ -z "$pool" ]; then
|
||||
echo "FAIL: could not read the $3 CIDRs from both renders."
|
||||
echo " kubernetes Cluster.spec.clusterNetwork: '${cluster:-<missing>}'"
|
||||
echo " kubernetes-nodes machine config cluster.network: '${pool:-<missing>}'"
|
||||
return 1
|
||||
fi
|
||||
if [ "$cluster" != "$pool" ]; then
|
||||
echo "FAIL: the tenant $3 CIDRs differ between the two charts."
|
||||
echo " packages/apps/kubernetes (CAPI Cluster): $cluster"
|
||||
echo " packages/apps/kubernetes-nodes (worker machine config): $pool"
|
||||
echo "Workers would route $3 traffic over a range the control plane does not"
|
||||
echo "use. Change both charts together. Moving the worker side also renames"
|
||||
echo "the content-hashed TalosConfigTemplate and rolls every worker."
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
@test "kubernetes and kubernetes-nodes render the same pod CIDRs" {
|
||||
TMP=$(mktemp -d)
|
||||
cluster_cidrs "$TMP"
|
||||
pool_cidrs "$TMP"
|
||||
assert_agree "$TMP" pods pod
|
||||
rm -rf "$TMP"
|
||||
}
|
||||
|
||||
@test "kubernetes and kubernetes-nodes render the same service CIDRs" {
|
||||
TMP=$(mktemp -d)
|
||||
cluster_cidrs "$TMP"
|
||||
pool_cidrs "$TMP"
|
||||
assert_agree "$TMP" services service
|
||||
rm -rf "$TMP"
|
||||
}
|
||||
@@ -78,19 +78,16 @@ CATALOG_VALUES="$REPO_ROOT/packages/system/kubernetes-worker-image/values.yaml"
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Second family of default-drift guards on the same chart: the Talos <-> Kubernetes
|
||||
# support matrix. The render guard keyed only on `talos.version`, so a pool could
|
||||
# reach a different Talos release through `osImage.builtin.version` /
|
||||
# `osImage.factory.version` and boot a kubelet outside its support window with no
|
||||
# render failure. The table now lives once in templates/_helpers.tpl and both
|
||||
# paths consult it.
|
||||
# support matrix. A pool reaches a Talos release through `talos.version` or through
|
||||
# `osImage.builtin.version` / `osImage.factory.version`, and the table in
|
||||
# templates/_helpers.tpl has to be consulted for whichever one the pool boots.
|
||||
#
|
||||
# The symmetry is structural, not observable from a render: the matrix ships one
|
||||
# row (`v1.13`) whose Kubernetes list is a superset of `values.schema.json`'s
|
||||
# `version` enum, so no schema-valid pairing can make either call site fail today.
|
||||
# A helm-unittest case would have to put an out-of-enum `version` in, and
|
||||
# helm-unittest validates the schema, so it cannot. These two guards cover what a
|
||||
# render cannot: that both call sites exist, and that the table still means
|
||||
# something for the defaults the chart ships.
|
||||
# The matrix ships one row (`v1.13`) whose Kubernetes list is a superset of
|
||||
# `values.schema.json`'s `version` enum, so no schema-valid pairing makes either
|
||||
# call site fail today. hack/kubernetes-nodes-talos-matrix.bats renders the
|
||||
# refusals with the schema skipped; these two guards cover what it does not: that
|
||||
# both call sites stay wired, and that the table still means something for the
|
||||
# defaults the chart ships.
|
||||
|
||||
NODES_TEMPLATE="$REPO_ROOT/packages/apps/kubernetes-nodes/templates/nodegroup.yaml"
|
||||
NODES_SCHEMA="$REPO_ROOT/packages/apps/kubernetes-nodes/values.schema.json"
|
||||
@@ -148,30 +145,3 @@ NODES_SCHEMA="$REPO_ROOT/packages/apps/kubernetes-nodes/values.schema.json"
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Third guard, and the premise the first one rests on. The catalog is keyed off
|
||||
# the kubernetes-nodes chart's `talos` defaults, but the parent kubernetes chart
|
||||
# carries its own copy of the same block since the Phase 2 split, and its own
|
||||
# values.yaml says to keep the two in sync. Nothing enforced that: the pool
|
||||
# chart's `make update` copies files/versions.yaml and images/kubectl.tag from
|
||||
# the parent and not the talos block, so a one-sided bump leaves the parent's
|
||||
# support-matrix guard reasoning about one Talos release while every worker boots
|
||||
# another — and the catalog check above would go on passing, because it only ever
|
||||
# reads one of the two files.
|
||||
|
||||
PARENT_VALUES="$REPO_ROOT/packages/apps/kubernetes/values.yaml"
|
||||
|
||||
@test "the parent kubernetes chart and kubernetes-nodes agree on their Talos defaults" {
|
||||
for key in version schematicID imageFactoryURL installerRepository registryMirrors; do
|
||||
parent=$(yq ".talos.${key}" "$PARENT_VALUES")
|
||||
pool=$(yq ".talos.${key}" "$APP_VALUES")
|
||||
if [ "$parent" != "$pool" ]; then
|
||||
echo "talos.${key} differs between the two charts:" >&2
|
||||
echo " packages/apps/kubernetes/values.yaml: $parent" >&2
|
||||
echo " packages/apps/kubernetes-nodes/values.yaml: $pool" >&2
|
||||
echo "both are declared 'keep in sync' and a worker pool follows the second while the parent's support-matrix guard reads the first" >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
}
|
||||
@@ -18,7 +18,7 @@ cozy_e2e_storage_class() {
|
||||
esac
|
||||
}
|
||||
|
||||
# talos_spec_block: emit the tenant Kubernetes CR `spec.talos` block with the Talos
|
||||
# talos_spec_block: emit the tenant KubernetesNodes CR `spec.talos` block with the Talos
|
||||
# OS image cache (imageFactoryURL) when the in-sandbox cache is up. Prints nothing
|
||||
# when it falls back to the public default, so the chart default applies.
|
||||
# Indented for insertion directly under `spec:` in the heredoc below.
|
||||
@@ -5702,8 +5702,9 @@ YAML
|
||||
|
||||
# Point worker DataVolume imports at the in-sandbox Talos OS image cache when it
|
||||
# is up, falling back to the public factory otherwise. Worker image pulls use
|
||||
# the chart's public-registry default. Emitted right under spec: as
|
||||
# `talos: { imageFactoryURL: ... }`, or nothing when the default applies.
|
||||
# the chart's public-registry default. Emitted right under spec: of the
|
||||
# KubernetesNodes pool below as `talos: { imageFactoryURL: ... }`, or nothing
|
||||
# when the default applies. The Kubernetes CR takes no worker image field.
|
||||
local talos_block
|
||||
talos_block=$(talos_spec_block)
|
||||
|
||||
@@ -5714,7 +5715,6 @@ metadata:
|
||||
name: "${test_name}"
|
||||
namespace: tenant-test
|
||||
spec:
|
||||
${talos_block}
|
||||
${oidc_block}
|
||||
addons:
|
||||
certManager:
|
||||
|
||||
@@ -49,10 +49,10 @@
|
||||
# Best-effort: never fail the suite on the band-aid. Remove once tenant workers
|
||||
# no longer bulk-pull the OS image from the public internet in CI.
|
||||
local sid ver
|
||||
sid=$(yq '.talos.schematicID' packages/apps/kubernetes/values.yaml 2>/dev/null)
|
||||
ver=$(yq '.talos.version' packages/apps/kubernetes/values.yaml 2>/dev/null)
|
||||
sid=$(yq '.talos.schematicID' packages/apps/kubernetes-nodes/values.yaml 2>/dev/null)
|
||||
ver=$(yq '.talos.version' packages/apps/kubernetes-nodes/values.yaml 2>/dev/null)
|
||||
if [ -z "$sid" ] || [ "$sid" = "null" ] || [ -z "$ver" ] || [ "$ver" = "null" ]; then
|
||||
echo "WARNING: could not read talos.schematicID/version from values.yaml — skipping mirror (fallback to public factory)"
|
||||
echo "WARNING: could not read talos.schematicID/version from packages/apps/kubernetes-nodes/values.yaml — skipping mirror (fallback to public factory)"
|
||||
return 0
|
||||
fi
|
||||
# The CiliumClusterwideNetworkPolicy in the manifest is intentionally NOT applied
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
# This Deployment fetches the image ONCE at e2e-install time with an aggressive
|
||||
# curl retry loop (decoupling the one flaky external fetch from the many
|
||||
# per-worker imports), then serves it locally over plain HTTP with range support.
|
||||
# Tenant `Kubernetes` CRs point `spec.talos.imageFactoryURL` at this Service
|
||||
# Tenant `KubernetesNodes` CRs point `spec.talos.imageFactoryURL` at this Service
|
||||
# (see hack/e2e-chainsaw/_lib/run-kubernetes.sh) so every worker imports from the
|
||||
# warm local copy instead of the public internet. Tenant namespaces run under a
|
||||
# default-deny Cilium egress installed by the tenant chart, whose allow rules cover
|
||||
@@ -28,7 +28,7 @@
|
||||
# mirror can only help, never make CI worse.
|
||||
#
|
||||
# The __SCHEMATIC_ID__ / __TALOS_VERSION__ placeholders are substituted at apply
|
||||
# time by hack/e2e-install-cozystack.bats from packages/apps/kubernetes/values.yaml.
|
||||
# time by hack/e2e-install-cozystack.bats from packages/apps/kubernetes-nodes/values.yaml.
|
||||
#
|
||||
# CI band-aid, not a product artifact. The durable fix is the overridable
|
||||
# `talos.imageFactoryURL` chart value this manifest exercises; a self-hosted
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env bats
|
||||
# Unit test: the refusals of the Talos to Kubernetes support-matrix guard in
|
||||
# packages/apps/kubernetes-nodes (kubernetes-nodes.assertTalosSupportsKubernetes,
|
||||
# called from templates/nodegroup.yaml for talos.version and again for the
|
||||
# release osImage resolves to).
|
||||
#
|
||||
# On the shipped chart a refusal is unreachable: the values schema offers only
|
||||
# Kubernetes versions the v1.13 row lists, and files/versions.yaml, which another
|
||||
# template fails on first, holds the same set. The guard is there for the day a
|
||||
# Talos row stops listing a version the chart still offers. So this renders a
|
||||
# copy of the chart with --skip-schema-validation and one extra versions.yaml row
|
||||
# for a version the matrix does not list. The pairings the guard lets through
|
||||
# are in the chart's own tests/talos_k8s_support_matrix_test.yaml.
|
||||
|
||||
load test_helper
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
|
||||
CHART="$REPO_ROOT/packages/apps/kubernetes-nodes"
|
||||
|
||||
# render_pool <args...> renders the pool chart copy with the given --set flags and
|
||||
# prints helm's output, errors included. The copy lives in the test's own tmpdir.
|
||||
render_pool() {
|
||||
copy="$BATS_TEST_TMPDIR/chart"
|
||||
if [ ! -d "$copy" ]; then
|
||||
cp -RL "$CHART" "$copy"
|
||||
printf '"v1.30": "v1.30.14"\n' >> "$copy/files/versions.yaml"
|
||||
fi
|
||||
helm template kubernetes-nodes-myk8s-md0 "$copy" --namespace tenant-test \
|
||||
--skip-schema-validation \
|
||||
--set cluster=myk8s \
|
||||
--set _cluster.cluster-domain=cozy.local \
|
||||
--set minReplicas=1 --set maxReplicas=3 \
|
||||
--set instanceType= --set resources.cpu=2 --set resources.memory=4Gi \
|
||||
--set diskSize=20Gi --set storageClass=replicated \
|
||||
"$@" 2>&1
|
||||
}
|
||||
|
||||
@test "an in-window pairing renders on the same copy, so the refusals below are the guard's" {
|
||||
out=$(render_pool --set version=v1.31) || { echo "$out" >&2; return 1; }
|
||||
printf '%s\n' "$out" | grep -q '^kind: KubevirtMachineTemplate$'
|
||||
}
|
||||
|
||||
@test "a Kubernetes version outside the row of talos.version is refused" {
|
||||
if out=$(render_pool --set version=v1.30); then
|
||||
echo "render succeeded with Kubernetes v1.30 on the default Talos" >&2
|
||||
return 1
|
||||
fi
|
||||
printf '%s\n' "$out" | grep -qF "Kubernetes v1.30 is not supported by Talos v1.13.6. Supported versions: [v1.31 v1.32 v1.33 v1.34 v1.35 v1.36]. Bump 'version' or set talos.version explicitly." \
|
||||
|| { echo "$out" >&2; return 1; }
|
||||
}
|
||||
|
||||
@test "an older patch release of a listed Talos minor is checked against the same row" {
|
||||
if out=$(render_pool --set version=v1.30 --set talos.version=v1.13.0); then
|
||||
echo "render succeeded with Kubernetes v1.30 on Talos v1.13.0: the guard is not keyed on the minor" >&2
|
||||
return 1
|
||||
fi
|
||||
printf '%s\n' "$out" | grep -qF 'Kubernetes v1.30 is not supported by Talos v1.13.0.' \
|
||||
|| { echo "$out" >&2; return 1; }
|
||||
}
|
||||
|
||||
# talos.version names a minor the matrix does not list, so the first call lets the
|
||||
# pool through and only the call fed the osImage release can refuse it.
|
||||
@test "a Kubernetes version outside the row of osImage.factory.version is refused" {
|
||||
if out=$(render_pool --set version=v1.30 --set talos.version=v1.12.0 --set osImage.factory.version=v1.13.0); then
|
||||
echo "render succeeded with Kubernetes v1.30 on Talos v1.13.0 reached through osImage.factory" >&2
|
||||
return 1
|
||||
fi
|
||||
printf '%s\n' "$out" | grep -qF 'Kubernetes v1.30 is not supported by Talos v1.13.0.' \
|
||||
&& printf '%s\n' "$out" | grep -qF 'nodeGroup md0 reaches this Talos release through its osImage override' \
|
||||
|| { echo "$out" >&2; return 1; }
|
||||
}
|
||||
|
||||
@test "a Kubernetes version outside the row of osImage.builtin.version is refused" {
|
||||
if out=$(render_pool --set version=v1.30 --set talos.version=v1.12.0 --set osImage.builtin.version=v1.13.0); then
|
||||
echo "render succeeded with Kubernetes v1.30 on Talos v1.13.0 reached through osImage.builtin" >&2
|
||||
return 1
|
||||
fi
|
||||
printf '%s\n' "$out" | grep -qF 'Kubernetes v1.30 is not supported by Talos v1.13.0.' \
|
||||
&& printf '%s\n' "$out" | grep -qF 'nodeGroup md0 reaches this Talos release through its osImage override' \
|
||||
|| { echo "$out" >&2; return 1; }
|
||||
}
|
||||
@@ -2,7 +2,7 @@
|
||||
# -----------------------------------------------------------------------------
|
||||
# Unit tests for talos_spec_block in hack/e2e-chainsaw/_lib/run-kubernetes.sh
|
||||
#
|
||||
# It is the composition point for the tenant Kubernetes CR's `spec.talos`: the
|
||||
# It is the composition point for the tenant KubernetesNodes CR's `spec.talos`: the
|
||||
# Talos OS image cache contributes imageFactoryURL when available. The result is
|
||||
# spliced into an unquoted heredoc directly under `spec:`, so an indentation slip
|
||||
# does not fail this function -- it produces a CR the API rejects, and every
|
||||
@@ -32,7 +32,7 @@ load test_helper
|
||||
# Splice a talos_spec_block result under `spec:` of a minimal CR and print the
|
||||
# document, so yq can be asked what the tenant apiserver would see.
|
||||
spec_doc() {
|
||||
printf 'apiVersion: apps.cozystack.io/v1alpha1\nkind: Kubernetes\nmetadata:\n name: t\nspec:\n%s\n host: ""\n' "$1"
|
||||
printf 'apiVersion: apps.cozystack.io/v1alpha1\nkind: KubernetesNodes\nmetadata:\n name: t\nspec:\n%s\n cluster: t\n' "$1"
|
||||
}
|
||||
|
||||
@test "no Talos image cache renders nothing, so the chart default applies" {
|
||||
@@ -59,6 +59,6 @@ spec_doc() {
|
||||
# talos_spec_block writes imageFactoryURL by hand. If the chart ever renames
|
||||
# it, the CR would be rejected only at e2e time, and this file is the cheapest
|
||||
# place to notice.
|
||||
yq -e '.talos | has("imageFactoryURL")' packages/apps/kubernetes/values.yaml >/dev/null \
|
||||
|| { echo "packages/apps/kubernetes/values.yaml has no talos.imageFactoryURL; talos_spec_block would emit a key the chart rejects" >&2; exit 1; }
|
||||
yq -e '.talos | has("imageFactoryURL")' packages/apps/kubernetes-nodes/values.yaml >/dev/null \
|
||||
|| { echo "packages/apps/kubernetes-nodes/values.yaml has no talos.imageFactoryURL; talos_spec_block would emit a key the pool chart ignores" >&2; exit 1; }
|
||||
}
|
||||
@@ -1,5 +1,3 @@
|
||||
.helmignore
|
||||
/logos
|
||||
/Makefile
|
||||
/hack
|
||||
/images/*/*
|
||||
@@ -9,16 +9,15 @@ generate:
|
||||
cozyvalues-gen -m 'kubernetesnodes' -v values.yaml -s values.schema.json -r README.md -g ../../../api/apps/v1alpha1/kubernetesnodes/types.go
|
||||
../../../hack/update-crd.sh
|
||||
|
||||
# Keep the worker-image inputs (Talos<->Kubernetes support matrix and the
|
||||
# kubectl.tag used by the talos-reconcile Job) in lockstep with the parent
|
||||
# kubernetes chart. They are copied, not symlinked, so `helm package` bundles
|
||||
# them; run this after the parent chart's `make update`.
|
||||
# Copy the Kubernetes version map (files/versions.yaml, minor -> patch release)
|
||||
# and the kubectl.tag used by the talos-reconcile Job from the kubernetes
|
||||
# chart, which owns both. They are copied, not symlinked, so `helm package`
|
||||
# bundles them; run this after either changes there.
|
||||
#
|
||||
# _talosconfigtemplate.tpl is not copied from anywhere: the parent chart no
|
||||
# longer carries worker node pools, so this chart owns the worker machine
|
||||
# config outright. It stays here because the TalosConfigTemplate it renders is
|
||||
# content-hash named and both consumers in this chart - nodegroup.yaml, which
|
||||
# writes bootstrap.configRef, and talos-reconcile-job.yaml, which creates the
|
||||
# _talosconfigtemplate.tpl is not copied from anywhere: this chart owns the
|
||||
# worker machine config. The TalosConfigTemplate it renders is content-hash
|
||||
# named and both consumers in this chart - nodegroup.yaml, which writes
|
||||
# bootstrap.configRef, and talos-reconcile-job.yaml, which creates the
|
||||
# object - have to hash the same bytes.
|
||||
update:
|
||||
cp ../kubernetes/files/versions.yaml files/versions.yaml
|
||||
|
||||
@@ -1,92 +1,93 @@
|
||||
# Kubernetes Worker Node Pool
|
||||
|
||||
`kubernetes-nodes` manages a single worker node pool for a managed Kubernetes
|
||||
cluster. It is the worker half of the Kubernetes-app split (see
|
||||
[cozystack/community#8](https://github.com/cozystack/community/pull/8)): the
|
||||
control plane lives in the `kubernetes` chart, and each pool is a separate
|
||||
`kubernetes-nodes` release attached to a parent cluster by name.
|
||||
`kubernetes-nodes` manages one worker node pool of a managed Kubernetes cluster. The control plane and the cluster addons come from the `kubernetes` application; every pool is a separate `KubernetesNodes` resource that attaches to its cluster by name.
|
||||
|
||||
A `KubernetesNodes` CR named `<cluster>-<pool>` (e.g. `myk8s-md0`) provisions a
|
||||
`MachineDeployment` + `KubevirtMachineTemplate` + `MachineHealthCheck` wired to
|
||||
the parent cluster `kubernetes-<cluster>` in the same namespace, plus a
|
||||
`talos-reconcile` Job that produces the pool's `TalosConfigTemplate`.
|
||||
A `KubernetesNodes` resource named `<cluster>-<pool>` (for example `myk8s-md0`) sets `cluster: <cluster>` and renders, in the namespace of the `kubernetes-<cluster>` cluster:
|
||||
|
||||
- a `MachineDeployment` that the cluster autoscaler scales between `minReplicas` and `maxReplicas`;
|
||||
- the machine template its workers are created from: a `KubevirtMachineTemplate` on `substrate: kubevirt` (the default), or a `ProxmoxMachineTemplate` on `substrate: proxmox`;
|
||||
- a `MachineHealthCheck` that replaces workers which do not become Ready or stop being Ready;
|
||||
- a `WorkloadMonitor` for the pool's workers;
|
||||
- a `talos-reconcile` Job that creates the pool's `TalosConfigTemplate`, the Talos machine config every worker boots with.
|
||||
|
||||
The pool's Kubernetes minor (`version`) may lag the cluster's during an upgrade but may not lead it: upgrade the `Kubernetes` resource first, then each pool. Delete the pools of a cluster before the cluster itself.
|
||||
|
||||
## Parameters
|
||||
|
||||
### Linkage
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----- |
|
||||
| `cluster` | Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool name (the part after `<cluster>-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict. | `string` | `""` |
|
||||
| Name | Description | Type | Value |
|
||||
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | ----- |
|
||||
| `cluster` | Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool's objects are named `kubernetes-<cluster>-<pool>`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over. | `string` | `""` |
|
||||
|
||||
|
||||
### Common Parameters
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------ |
|
||||
| `storageClass` | StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. | `string` | `replicated` |
|
||||
| Name | Description | Type | Value |
|
||||
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------ |
|
||||
| `storageClass` | StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`. | `string` | `replicated` |
|
||||
|
||||
|
||||
### Pool Sizing
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------- |
|
||||
| `minReplicas` | Minimum number of replicas in this pool. Used as the cluster-autoscaler floor. | `int` | `0` |
|
||||
| `maxReplicas` | Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling. | `int` | `10` |
|
||||
| `instanceType` | Virtual machine instance type. | `string` | `u1.medium` |
|
||||
| `diskSize` | System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. | `quantity` | `20Gi` |
|
||||
| `roles` | List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller. | `[]string` | `[]` |
|
||||
| `resources` | Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `"010"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `"010"` reserves what `"10"` does. | `object` | `{}` |
|
||||
| `resources.cpu` | Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores. | `quantity` | `""` |
|
||||
| `resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `podCpuLimit` | CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. | `quantity` | `""` |
|
||||
| `podCpuRequest` | CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. | `quantity` | `""` |
|
||||
| `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
|
||||
| `gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` |
|
||||
| `kernelModules` | Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. | `[]object` | `[]` |
|
||||
| `kernelModules[i].name` | Module name as `modprobe` takes it, e.g. `nvidia_uvm`. | `string` | `""` |
|
||||
| `kernelModules[i].parameters` | Module parameters, each as a bare `key=value` string. | `[]string` | `[]` |
|
||||
| `kubelet` | Kubelet resource reservations for this pool. | `object` | `{}` |
|
||||
| `kubelet.systemReservedMemory` | Memory reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.kubeReservedMemory` | Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.systemReservedCpu` | CPU reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
|
||||
| `kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` |
|
||||
| `kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` |
|
||||
| `logSerialConsole` | Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it. | `bool` | `false` |
|
||||
| `maxUnhealthy` | Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At "50%" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two. | `string` | `50%` |
|
||||
| `nodeStartupTimeout` | Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning. | `string` | `20m` |
|
||||
| `readyUnknownTimeout` | How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart. | `string` | `5m` |
|
||||
| `readyFalseTimeout` | How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart. | `string` | `10m` |
|
||||
| `maxNodeProvisionTime` | How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted. Keep in sync with the parent kubernetes chart. | `string` | `30m` |
|
||||
| Name | Description | Type | Value |
|
||||
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------- |
|
||||
| `minReplicas` | Minimum number of replicas in this pool. Used as the cluster-autoscaler floor. | `int` | `0` |
|
||||
| `maxReplicas` | Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling. | `int` | `10` |
|
||||
| `instanceType` | KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them. | `string` | `u1.medium` |
|
||||
| `diskSize` | System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size. | `quantity` | `20Gi` |
|
||||
| `roles` | List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller. | `[]string` | `[]` |
|
||||
| `resources` | Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `"010"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `"010"` reserves what `"10"` does. | `object` | `{}` |
|
||||
| `resources.cpu` | Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores. | `quantity` | `""` |
|
||||
| `resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `podCpuLimit` | CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. | `quantity` | `""` |
|
||||
| `podCpuRequest` | CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. | `quantity` | `""` |
|
||||
| `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
|
||||
| `gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` |
|
||||
| `kernelModules` | Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. | `[]object` | `[]` |
|
||||
| `kernelModules[i].name` | Module name as `modprobe` takes it, e.g. `nvidia_uvm`. | `string` | `""` |
|
||||
| `kernelModules[i].parameters` | Module parameters, each as a bare `key=value` string. | `[]string` | `[]` |
|
||||
| `kubelet` | Kubelet resource reservations for this pool. An empty reservation is computed from the worker's size: `resources` when set, otherwise the `instanceType`. | `object` | `{}` |
|
||||
| `kubelet.systemReservedMemory` | Memory reserved for host OS. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi. | `string` | `""` |
|
||||
| `kubelet.kubeReservedMemory` | Memory reserved for kubelet and container runtime. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi. | `string` | `""` |
|
||||
| `kubelet.systemReservedCpu` | CPU reserved for host OS. When empty, 5% of the worker's CPU, clamped to 50m..500m. | `string` | `""` |
|
||||
| `kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. When empty, 5% of the worker's CPU, clamped to 50m..500m. | `string` | `""` |
|
||||
| `kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` |
|
||||
| `kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` |
|
||||
| `logSerialConsole` | Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it. | `bool` | `false` |
|
||||
| `maxUnhealthy` | Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At "50%" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two. | `string` | `50%` |
|
||||
| `nodeStartupTimeout` | Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (the Talos image import from the Image Factory, or a storage class slow to provision or clone the worker disk). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning. | `string` | `20m` |
|
||||
| `readyUnknownTimeout` | How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. | `string` | `5m` |
|
||||
| `readyFalseTimeout` | How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. | `string` | `10m` |
|
||||
| `maxNodeProvisionTime` | How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted. | `string` | `30m` |
|
||||
|
||||
|
||||
### Kubernetes Version
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- |
|
||||
| `version` | Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`. | `string` | `v1.35` |
|
||||
| Name | Description | Type | Value |
|
||||
| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- |
|
||||
| `version` | Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool. The render fails when it is ahead. Its pairing with the Talos release the pool resolves (`talos.version`, or the `osImage` override) is checked against the chart's Talos<->Kubernetes support matrix: for a Talos minor the matrix lists, a Kubernetes minor outside that minor's list is refused; a Talos minor the matrix does not list is not checked. | `string` | `v1.35` |
|
||||
|
||||
|
||||
### Talos Worker Image
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------------------------------------------------------ |
|
||||
| `osImage` | Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool. | `object` | `{}` |
|
||||
| `osImage.builtin` | Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled. | `*object` | `null` |
|
||||
| `osImage.builtin.schematicID` | Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`. | `string` | `""` |
|
||||
| `osImage.builtin.version` | Talos version of the golden to clone. Defaults to `talos.version`. | `string` | `""` |
|
||||
| `osImage.factory` | Import from an Image Factory / mirror (default). | `*object` | `null` |
|
||||
| `osImage.factory.version` | Talos version. Defaults to `talos.version`. | `string` | `""` |
|
||||
| `osImage.factory.schematicID` | Talos schematic ID. Defaults to `talos.schematicID`. | `string` | `""` |
|
||||
| `osImage.factory.imageFactoryURL` | Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry. | `string` | `""` |
|
||||
| `talos` | Talos worker image configuration. Keep in sync with the parent cluster's `talos`. | `object` | `{}` |
|
||||
| `talos.version` | Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead. | `string` | `v1.13.6` |
|
||||
| `talos.schematicID` | Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID. | `string` | `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` |
|
||||
| `talos.imageFactoryURL` | Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. | `string` | `https://factory.talos.dev` |
|
||||
| `talos.installerRepository` | OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. | `string` | `factory.talos.dev/installer` |
|
||||
| `talos.registryMirrors` | Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. | `object` | `{}` |
|
||||
| `talos.imageFactoryCA` | Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. | `string` | `""` |
|
||||
| Name | Description | Type | Value |
|
||||
| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------------------------------------------------------ |
|
||||
| `osImage` | Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool. | `object` | `{}` |
|
||||
| `osImage.builtin` | Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled. | `*object` | `null` |
|
||||
| `osImage.builtin.schematicID` | Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`. | `string` | `""` |
|
||||
| `osImage.builtin.version` | Talos version of the golden to clone. Defaults to `talos.version`. | `string` | `""` |
|
||||
| `osImage.factory` | Import from an Image Factory / mirror (default). | `*object` | `null` |
|
||||
| `osImage.factory.version` | Talos version. Defaults to `talos.version`. | `string` | `""` |
|
||||
| `osImage.factory.schematicID` | Talos schematic ID. Defaults to `talos.schematicID`. | `string` | `""` |
|
||||
| `osImage.factory.imageFactoryURL` | Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. This redirects only the worker boot-disk import; the installer image in the worker machine config is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry. | `string` | `""` |
|
||||
| `talos` | Talos worker image configuration for this pool. | `object` | `{}` |
|
||||
| `talos.version` | Default Talos release for this pool's worker OS image and for the installer image written into the worker machine config. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix lists, for each Talos minor the chart knows, the Kubernetes minors it supports and refuses a pool `version` outside that list; a Talos minor it does not list passes unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead. | `string` | `v1.13.6` |
|
||||
| `talos.schematicID` | Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID. | `string` | `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` |
|
||||
| `talos.imageFactoryURL` | Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. On `substrate: proxmox` the workers clone a VM template that already carries the OS, so this field has no effect there beyond that format check. | `string` | `https://factory.talos.dev` |
|
||||
| `talos.installerRepository` | OCI repository prefix for the Talos installer image, written into the worker machine config as `machine.install.image`. Resolved as `<installerRepository>/<schematicID>:<version>` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. | `string` | `factory.talos.dev/installer` |
|
||||
| `talos.registryMirrors` | Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. | `object` | `{}` |
|
||||
| `talos.imageFactoryCA` | Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. On `substrate: proxmox` it has no effect on the workers either: the ConfigMap is still rendered, but nothing references it. | `string` | `""` |
|
||||
|
||||
|
||||
### Substrate
|
||||
|
||||
@@ -7,12 +7,11 @@ This pool is a separate release from the control plane. Three operational notes:
|
||||
cluster that is being torn down. Delete the KubernetesNodes pools first, then
|
||||
the parent Kubernetes cluster.
|
||||
|
||||
2. Version skew. Keep this pool's `version` and `talos.*` in sync with the parent
|
||||
Kubernetes cluster. A pool on a Kubernetes/Talos combination the parent does
|
||||
not run can produce workers the control plane will not admit.
|
||||
2. Version skew. This pool's `version` may lag the parent cluster's Kubernetes
|
||||
minor during an upgrade but may not lead it: a worker kubelet ahead of the
|
||||
apiserver is refused at render time. Upgrade the parent Kubernetes CR first,
|
||||
then each pool.
|
||||
|
||||
3. Pool name. The pool name (the part after `<cluster>-` in the KubernetesNodes
|
||||
name) must not collide with a nodeGroup still managed by the parent kubernetes
|
||||
chart — most importantly the implicit default `md0`. A collision makes this
|
||||
release fail on a Helm ownership conflict against the parent-owned objects.
|
||||
The chart guards this at render time with a clear error.
|
||||
3. Pool name. The pool's objects are named `kubernetes-<cluster>-<pool>`. When a
|
||||
MachineDeployment of that name already exists and belongs to another Helm
|
||||
release, this release fails at render time rather than taking it over.
|
||||
@@ -165,10 +165,9 @@ The pool attaches to the parent Kubernetes CR named .Values.cluster, whose
|
||||
HelmRelease (and therefore CAPI Cluster / KamajiControlPlane / KubevirtCluster
|
||||
and every worker object it owns) is `kubernetes-<cluster>`. This chart does
|
||||
NOT use its own .Release.Name for CAPI wiring — the pool lives in a separate
|
||||
HelmRelease from the control plane, so every reference that the monolithic
|
||||
chart made through $.Release.Name is reconstructed here as kubernetes-<cluster>
|
||||
instead. Linkage is by name convention (mirrors vm-instance -> vm-disk), not
|
||||
ownerReference or lookup-gated render.
|
||||
HelmRelease from the control plane, so every reference to the cluster's objects
|
||||
is built from kubernetes-<cluster>. Linkage is by name convention (mirrors
|
||||
vm-instance -> vm-disk), not ownerReference or lookup-gated render.
|
||||
*/}}
|
||||
{{- define "kubernetes-nodes.clusterName" -}}
|
||||
{{- if not .Values.cluster -}}
|
||||
@@ -194,9 +193,9 @@ The node-group name for this pool, derived from the release name.
|
||||
A KubernetesNodes CR is named <cluster>-<pool> and gets the release prefix
|
||||
`kubernetes-nodes-`, so the release name is `kubernetes-nodes-<cluster>-<pool>`.
|
||||
The group name is the <pool> suffix. Enforcing the `<cluster>-` segment keeps
|
||||
every rendered object named `kubernetes-<cluster>-<pool>` — byte-identical to
|
||||
what the monolithic chart rendered for the same group — and prevents two
|
||||
clusters in one namespace from colliding on a pool named e.g. `md0`.
|
||||
every rendered object named `kubernetes-<cluster>-<pool>` — the names worker
|
||||
objects adopted by migration 56 already carry — and prevents two clusters in
|
||||
one namespace from colliding on a pool named e.g. `md0`.
|
||||
*/}}
|
||||
{{- define "kubernetes-nodes.groupName" -}}
|
||||
{{- $prefix := printf "kubernetes-nodes-%s-" .Values.cluster -}}
|
||||
@@ -208,13 +207,13 @@ clusters in one namespace from colliding on a pool named e.g. `md0`.
|
||||
|
||||
{{/*
|
||||
Fail early with a clear message if this pool's MachineDeployment already exists
|
||||
under a different Helm release — i.e. the pool name collides with a nodeGroup
|
||||
still managed by the parent kubernetes chart (most likely the default `md0`).
|
||||
under a different Helm release: a worker pool migration 56 left with the parent
|
||||
cluster's release, or another pool whose <cluster>-<pool> split reconstructs
|
||||
the same object name (cluster `a` with pool `b-c`, cluster `a-b` with pool `c`).
|
||||
Without this guard the collision surfaces as a cryptic Helm "invalid ownership
|
||||
metadata" error at install time. Inert under `helm template`/unittest (lookup
|
||||
returns nil with no cluster) and a no-op during the Phase 2b adoption
|
||||
migration, which re-annotates the MachineDeployment onto this release before
|
||||
it reconciles.
|
||||
returns nil with no cluster) and a no-op for a pool migration 56 adopted, which
|
||||
re-annotates the MachineDeployment onto this release before it reconciles.
|
||||
*/}}
|
||||
{{- define "kubernetes-nodes.assertNoForeignPool" -}}
|
||||
{{- $clusterName := include "kubernetes-nodes.clusterName" . -}}
|
||||
@@ -224,7 +223,7 @@ it reconciles.
|
||||
{{- if $existing -}}
|
||||
{{- $owner := dig "annotations" "meta.helm.sh/release-name" "" $existing.metadata -}}
|
||||
{{- if and $owner (ne $owner .Release.Name) -}}
|
||||
{{- fail (printf "kubernetes-nodes: MachineDeployment %q in namespace %q is already managed by release %q, not this pool release %q — the pool name collides with a nodeGroup still managed by the parent kubernetes chart. Rename the pool or remove it from the parent Kubernetes CR's nodeGroups first." $mdName .Release.Namespace $owner .Release.Name) -}}
|
||||
{{- fail (printf "kubernetes-nodes: MachineDeployment %q in namespace %q is already managed by release %q, not this pool release %q — two pools cannot share the object name kubernetes-<cluster>-<pool>: give this pool a name whose object name is free, or remove the existing pool from release %q first." $mdName .Release.Namespace $owner .Release.Name $owner) -}}
|
||||
{{- end -}}
|
||||
{{- /* clusterName-drift guard. The release name kubernetes-nodes-<cluster>-<pool>
|
||||
does not encode where <cluster> ends and <pool> begins: for one release
|
||||
@@ -253,8 +252,7 @@ control plane is supported (up to n-3 upstream) and is the normal state during a
|
||||
rolling upgrade — the parent Kubernetes CR is bumped first, then each pool. A
|
||||
symmetric equality check would flip every pool of a cluster to render-failure the
|
||||
moment the parent is bumped, blocking all pool operations until each pool is
|
||||
hand-edited, so the check is directional. The split removed the single `version`
|
||||
that used to feed both control plane and workers. Looks up the parent
|
||||
hand-edited, so the check is directional. Looks up the parent
|
||||
KamajiControlPlane (named like the reconstructed clusterName) and compares its
|
||||
spec.version minor against .Values.version. Skipped when the lookup is empty
|
||||
(helm template / unittest, or the parent not yet present) so it validates only
|
||||
|
||||
@@ -69,7 +69,7 @@
|
||||
{{- /* Management cluster DNS domain. Tenant Kubernetes workers run as
|
||||
KubeVirt VMs inside the management cluster's pod network and need
|
||||
to resolve management-cluster service short names ("<svc>.<ns>.svc")
|
||||
the same way an Ubuntu+kubeadm worker did via cloud-init DHCP. The
|
||||
the same way a pod of the management cluster does. The
|
||||
kubevirt-csi NFS mount path is the load-bearing case: kubelet on
|
||||
the worker host calls `mount.nfs linstor-csi-nfs.cozy-linstor.svc`
|
||||
(no domain suffix), and without searchDomains in /etc/resolv.conf
|
||||
|
||||
@@ -1,17 +1,16 @@
|
||||
{{- /*
|
||||
Worker node pool for a managed Kubernetes cluster.
|
||||
|
||||
This chart is the second half of the Kubernetes-app split (cozystack/community#8):
|
||||
the control plane (Cluster / KamajiControlPlane / KubevirtCluster / PKI) lives in
|
||||
the `kubernetes` chart, and each worker pool is a separate `kubernetes-nodes`
|
||||
release. The pool objects (KubevirtMachineTemplate, MachineDeployment,
|
||||
MachineHealthCheck, worker WorkloadMonitor) and the talos-reconcile Job that
|
||||
produces the pool's TalosConfigTemplate are rendered here, wired to the parent
|
||||
cluster `kubernetes-<cluster>` by name convention (see _helpers.tpl). Every
|
||||
reference the monolithic chart made through $.Release.Name is reconstructed here
|
||||
as the parent clusterName so the objects are byte-identical to what the parent
|
||||
rendered for the same group — the KubevirtMachineTemplate is content-hash-named,
|
||||
so any drift would roll every live worker VM.
|
||||
The control plane (Cluster / KamajiControlPlane / KubevirtCluster / PKI) lives
|
||||
in the `kubernetes` chart, and each worker pool is a separate
|
||||
`kubernetes-nodes` release. The pool objects (machine template,
|
||||
MachineDeployment, MachineHealthCheck, worker WorkloadMonitor) and the
|
||||
talos-reconcile Job that produces the pool's TalosConfigTemplate are rendered
|
||||
here, wired to the parent cluster `kubernetes-<cluster>` by name convention
|
||||
(see _helpers.tpl). Every reference to the cluster is built from that
|
||||
clusterName. Pools adopted by migration 56 keep running on objects of exactly
|
||||
these names and this content: the machine template is content-hash-named, so
|
||||
any drift in its text would roll every live worker VM.
|
||||
|
||||
Talos -> Kubernetes support matrix, against the pool default. The pool can also
|
||||
reach a different Talos release through osImage.builtin.version /
|
||||
@@ -346,7 +345,7 @@ spec:
|
||||
fails the whole release of a pool whose disks are all on the right class
|
||||
and which changed nothing. Only a pool with no disk yet resolves to
|
||||
today's default, which is the class its first clone will actually get. */}}
|
||||
{{- $cloneWorkerSC = .group.storageClass | default $.Values.storageClass | toString }}
|
||||
{{- $cloneWorkerSC = .group.storageClass | toString }}
|
||||
{{- if and (not $cloneWorkerSC) $poolDiskName }}
|
||||
{{- $poolPVC := lookup "v1" "PersistentVolumeClaim" $.Release.Namespace $poolDiskName }}
|
||||
{{- if $poolPVC }}
|
||||
@@ -527,12 +526,10 @@ spec:
|
||||
Worker VMs live-migrate, so their disks need ReadWriteMany. The
|
||||
RWX access mode is supplied by the chosen StorageClass's CDI
|
||||
StorageProfile — it is not set on the DataVolume here. linstor-csi
|
||||
rejects RWX volumes that are not on a DRBD-backed StorageClass, so
|
||||
an empty nodeGroup storageClass falls back to the application-level
|
||||
storageClass (a replicated/DRBD class) instead of the cluster
|
||||
default, which may be a non-DRBD class that the CSI driver would
|
||||
reject. Both empty still omits the field so the cluster default
|
||||
applies.
|
||||
rejects RWX volumes that are not on a DRBD-backed StorageClass,
|
||||
which is why the pool's storageClass defaults to a replicated/DRBD
|
||||
class rather than to the cluster default. An empty storageClass
|
||||
omits the field so the cluster default applies.
|
||||
|
||||
A builtin pool instead renders the class the guard above resolved for
|
||||
it. Omitting the field there would let the disk bind on whatever the
|
||||
@@ -541,12 +538,12 @@ spec:
|
||||
after the pool's earlier disks were bound turns the next clone into a
|
||||
host-assisted copy over the pod network, the transfer the guard exists
|
||||
to refuse, while the guard reports no drift because it compared the
|
||||
class those earlier disks sit on. With both levels empty and no cluster
|
||||
to resolve against the field is still omitted, so the HTTP path renders
|
||||
exactly as before and the KubevirtMachineTemplate hash named after it
|
||||
does not move.
|
||||
class those earlier disks sit on. With the class empty and no cluster
|
||||
to resolve against the field is still omitted, so the HTTP path does
|
||||
not render one and the KubevirtMachineTemplate hash named after it
|
||||
does not depend on the cluster default.
|
||||
*/}}
|
||||
{{- with $cloneWorkerSC | default (.group.storageClass | default $.Values.storageClass) }}
|
||||
{{- with $cloneWorkerSC | default .group.storageClass }}
|
||||
storageClassName: {{ . | quote }}
|
||||
{{- end }}
|
||||
template:
|
||||
@@ -1066,19 +1063,17 @@ spec:
|
||||
{{- /* MHC spec.maxUnhealthy is an IntOrString. CAPI admission accepts a
|
||||
bare integer (0, 1, ...) or a percentage string ("0%", "50%"), but
|
||||
rejects a quoted plain integer ("0"). Quote only when the operator
|
||||
supplied a percentage; otherwise render the value as a bare int so
|
||||
the admission webhook is happy with the chart-default "0". */}}
|
||||
supplied a percentage; otherwise render the value as a bare int. */}}
|
||||
{{- $maxUnhealthy := toString $.Values.maxUnhealthy }}
|
||||
{{- $muSource := "maxUnhealthy" }}
|
||||
{{- if hasSuffix "%" $maxUnhealthy }}
|
||||
{{- if not (regexMatch "^[0-9]+%$" $maxUnhealthy) }}
|
||||
{{- fail (printf "%s must be a bare integer (e.g. 0, 1) or an integer percentage (e.g. 50%%), got %q" $muSource $maxUnhealthy) }}
|
||||
{{- fail (printf "maxUnhealthy must be a bare integer (e.g. 0, 1) or an integer percentage (e.g. 50%%), got %q" $maxUnhealthy) }}
|
||||
{{- end }}
|
||||
maxUnhealthy: {{ $maxUnhealthy | quote }}
|
||||
{{- else if regexMatch "^[0-9]+$" $maxUnhealthy }}
|
||||
maxUnhealthy: {{ $maxUnhealthy | int }}
|
||||
{{- else }}
|
||||
{{- fail (printf "%s must be a bare integer (e.g. 0, 1) or an integer percentage (e.g. 50%%), got %q" $muSource $maxUnhealthy) }}
|
||||
{{- fail (printf "maxUnhealthy must be a bare integer (e.g. 0, 1) or an integer percentage (e.g. 50%%), got %q" $maxUnhealthy) }}
|
||||
{{- end }}
|
||||
nodeStartupTimeout: {{ $.Values.nodeStartupTimeout }}
|
||||
selector:
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
suite: worker pool clusterName-drift guard (assertNoForeignPool)
|
||||
|
||||
# assertNoForeignPool reconstructs the pool MachineDeployment name
|
||||
# kubernetes-<cluster>-<pool> and looks it up. Beyond the pre-existing
|
||||
# kubernetes-<cluster>-<pool> and looks it up. Beyond the
|
||||
# foreign-owner branch, it refuses the render when a live MD of that name is
|
||||
# owned by this release but its spec.clusterName disagrees with the clusterName
|
||||
# this pool renders: that is the fingerprint of a post-create .Values.cluster
|
||||
@@ -126,3 +126,28 @@ tests:
|
||||
value: MachineDeployment
|
||||
exists:
|
||||
path: metadata.name
|
||||
|
||||
# The other branch of the same guard: a live MachineDeployment of this name
|
||||
# that another release owns, here the cluster's own release, which is what a
|
||||
# pool migration 56 did not adopt still carries. The message has to point at
|
||||
# a remedy the operator can act on; the cluster's Kubernetes CR renders no
|
||||
# worker pool, so editing it cannot clear the conflict.
|
||||
- it: refuses to render over a MachineDeployment another release owns
|
||||
kubernetesProvider:
|
||||
scheme: *scheme
|
||||
objects:
|
||||
- *kcp
|
||||
- *schemeMachineSet
|
||||
- *schemeKmt
|
||||
- kind: MachineDeployment
|
||||
apiVersion: cluster.x-k8s.io/v1beta1
|
||||
metadata:
|
||||
name: kubernetes-myk8s-md0
|
||||
namespace: tenant-test
|
||||
annotations:
|
||||
meta.helm.sh/release-name: kubernetes-myk8s
|
||||
spec:
|
||||
clusterName: kubernetes-myk8s
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'already managed by release "kubernetes-myk8s", not this pool release "kubernetes-nodes-myk8s-md0" — two pools cannot share the object name kubernetes-<cluster>-<pool>: give this pool a name whose object name is free, or remove the existing pool from release "kubernetes-myk8s" first\.'
|
||||
@@ -1,10 +1,9 @@
|
||||
suite: kubelet reservation validation
|
||||
|
||||
# The kubelet-reservation branch (quantity-format regexes, positive-value and
|
||||
# eviction-threshold invariants) moved verbatim from the parent chart into
|
||||
# nodegroup.yaml. These failure-path guards reject malformed input at render
|
||||
# time before a broken machineconfig reaches a worker. Ported from the parent
|
||||
# chart's kubelet_reservation_test.
|
||||
# The kubelet-reservation branch of nodegroup.yaml (quantity-format regexes,
|
||||
# positive-value and eviction-threshold invariants). These failure-path guards
|
||||
# reject malformed input at render time before a broken machineconfig reaches
|
||||
# a worker.
|
||||
|
||||
templates:
|
||||
- templates/nodegroup.yaml
|
||||
|
||||
@@ -4,8 +4,6 @@ suite: MachineHealthCheck maxUnhealthy int-vs-percent rendering
|
||||
# but rejects a bare numeric STRING ("0"): a percentage must be a quoted string
|
||||
# and a count must be an unquoted int. nodegroup.yaml renders it as
|
||||
# maxUnhealthy: {{ if hasSuffix "%" ... }}{{ . | quote }}{{ else }}{{ . | int }}{{ end }}
|
||||
# Ported from the parent chart's mhc_rendering_test after the pool moved to
|
||||
# kubernetes-nodes.
|
||||
|
||||
templates:
|
||||
- templates/nodegroup.yaml
|
||||
|
||||
@@ -16,9 +16,8 @@ suite: MachineHealthCheck timeouts
|
||||
#
|
||||
# `nodeStartupTimeout` bounds the phase before any of that: it is the window a
|
||||
# Machine has to produce a Node at all, and the unhealthyConditions entries are
|
||||
# evaluated only once one exists. Its default is pinned here because the parity
|
||||
# gate compares this chart against its counterpart, so a default moved in both
|
||||
# at once passes there and only fails on this assertion.
|
||||
# evaluated only once one exists. Its default is pinned here because a moved
|
||||
# default changes every pool that does not set the field.
|
||||
|
||||
templates:
|
||||
- templates/nodegroup.yaml
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
suite: worker pool pod CPU limit
|
||||
|
||||
# Mirror of the parent kubernetes chart's suite, on the pool-shaped values of
|
||||
# this chart. `podCpuLimit` is the only knob that sets
|
||||
# `podCpuLimit` is the only knob that sets
|
||||
# `domain.resources.limits.cpu` on a worker VM. KubeVirt otherwise derives that
|
||||
# limit from the guest vCPU count whenever the namespace carries a ResourceQuota
|
||||
# that limits CPU, and it stops deriving as soon as the VMI declares the limit
|
||||
@@ -79,8 +78,7 @@ tests:
|
||||
asserts:
|
||||
# Absent, not empty: the KubevirtMachineTemplate is content-hash named, so
|
||||
# a `cpu:` key rendered for a pool that asked for nothing would rename the
|
||||
# template and roll every live worker VM of the pool on upgrade. The
|
||||
# render-parity gate compares those bytes against the parent chart.
|
||||
# template and roll every live worker VM of the pool on upgrade.
|
||||
- notExists:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.cpu
|
||||
- equal:
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
suite: worker pool pod CPU request
|
||||
|
||||
# Mirror of the parent kubernetes chart's suite, on the pool-shaped values of
|
||||
# this chart. `podCpuRequest` is the only knob that sets
|
||||
# `podCpuRequest` is the only knob that sets
|
||||
# `domain.resources.requests.cpu` on a worker VM, and it is the other half of
|
||||
# `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the
|
||||
# VM declares none, so without this field the ceiling and the scheduler
|
||||
@@ -100,8 +99,7 @@ tests:
|
||||
asserts:
|
||||
# Absent, not empty: the KubevirtMachineTemplate is content-hash named, so
|
||||
# a `cpu:` key rendered for a pool that asked for nothing would rename the
|
||||
# template and roll every live worker VM of the pool on upgrade. The
|
||||
# render-parity gate compares those bytes against the parent chart.
|
||||
# template and roll every live worker VM of the pool on upgrade.
|
||||
- notExists:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.requests.cpu
|
||||
- equal:
|
||||
|
||||
@@ -2,12 +2,9 @@ suite: worker node pool golden render snapshot
|
||||
|
||||
# Freezes the full rendered pool (KubevirtMachineTemplate, MachineDeployment,
|
||||
# MachineHealthCheck, worker WorkloadMonitor) for a representative node group.
|
||||
# In PR-2a this render was proven byte-identical to the monolithic kubernetes
|
||||
# chart's output via tests/render-parity.sh; PR-2b removed the pool from the
|
||||
# parent, so cross-chart parity can no longer run. This snapshot takes over as
|
||||
# the drift guard: any change to the rendered pool objects (which would rename
|
||||
# the content-hashed KubevirtMachineTemplate and churn live worker VMs on
|
||||
# upgrade) fails here until the snapshot is regenerated deliberately with
|
||||
# It is the drift guard for those objects: any change to them (which would
|
||||
# rename the content-hashed KubevirtMachineTemplate and churn live worker VMs
|
||||
# on upgrade) fails here until the snapshot is regenerated deliberately with
|
||||
# `helm unittest -u`.
|
||||
#
|
||||
# Coverage note: every case sets instanceType: "" with explicit resources,
|
||||
@@ -44,10 +41,10 @@ tests:
|
||||
asserts:
|
||||
- matchSnapshot: {}
|
||||
|
||||
# GPU and kubelet-override branches live on in nodegroup.yaml after the split
|
||||
# and are content-hash inputs to the KubevirtMachineTemplate; pin them too so
|
||||
# a change to either branch that would rename the KMT (and roll live GPU or
|
||||
# kubelet-tuned worker VMs) fails here. Value sets mirror the PR-2a parity gate.
|
||||
# The GPU and kubelet-override branches of nodegroup.yaml are content-hash
|
||||
# inputs to the KubevirtMachineTemplate; pin them too so a change to either
|
||||
# branch that would rename the KMT (and roll live GPU or kubelet-tuned worker
|
||||
# VMs) fails here.
|
||||
- it: matches the golden render of a GPU pool
|
||||
set:
|
||||
roles: []
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
suite: Talos to Kubernetes support-matrix guard (assertTalosSupportsKubernetes)
|
||||
|
||||
# The guard is keyed by Talos minor and consulted twice in nodegroup.yaml: once
|
||||
# for talos.version and once for the release osImage resolves to. These cases
|
||||
# pin the pairings it has to let through: each one fails if the matrix row loses
|
||||
# the Kubernetes version under test.
|
||||
#
|
||||
# The refusals are in hack/kubernetes-nodes-talos-matrix.bats. The values schema
|
||||
# offers only Kubernetes versions the v1.13 row lists, so no schema-valid input
|
||||
# reaches a refusal, and helm-unittest always validates the schema; the bats file
|
||||
# renders with --skip-schema-validation instead.
|
||||
|
||||
templates:
|
||||
- templates/nodegroup.yaml
|
||||
|
||||
release:
|
||||
name: kubernetes-nodes-myk8s-md0
|
||||
namespace: tenant-test
|
||||
|
||||
set:
|
||||
cluster: myk8s
|
||||
_cluster:
|
||||
cluster-domain: cozy.local
|
||||
minReplicas: 1
|
||||
maxReplicas: 3
|
||||
instanceType: ""
|
||||
diskSize: 20Gi
|
||||
storageClass: replicated
|
||||
roles: [ingress-nginx]
|
||||
resources:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
|
||||
tests:
|
||||
- it: renders the oldest Kubernetes version the schema offers against the default Talos
|
||||
set:
|
||||
version: "v1.31"
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
- hasDocuments:
|
||||
count: 4
|
||||
|
||||
- it: renders the newest Kubernetes version the schema offers against the default Talos
|
||||
set:
|
||||
version: "v1.35"
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
|
||||
- it: renders an in-window pairing reached through osImage.factory.version
|
||||
set:
|
||||
version: "v1.31"
|
||||
talos:
|
||||
version: "v1.12.0"
|
||||
osImage:
|
||||
factory:
|
||||
version: "v1.13.0"
|
||||
asserts:
|
||||
- notFailedTemplate: {}
|
||||
- equal:
|
||||
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.source.http.url
|
||||
value: https://factory.talos.dev/image/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515/v1.13.0/openstack-amd64.raw.xz
|
||||
documentSelector: {path: kind, value: KubevirtMachineTemplate}
|
||||
@@ -7,7 +7,6 @@ suite: Talos worker machineconfig (GPU labels + kubelet reservations)
|
||||
# - GPU pools must inject nodeLabels gpu=on so HAMi's device plugin
|
||||
# (nodeSelector gpu=on) schedules and advertises nvidia.com/gpu;
|
||||
# - kubelet reservation overrides must reach the machineconfig kubelet config.
|
||||
# Ported from the parent chart's gpu_node_labels_test / kubelet_reservation_test.
|
||||
|
||||
templates:
|
||||
- templates/talos-reconcile-job.yaml
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"cluster": {
|
||||
"description": "Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-\u003ccluster\u003e`; the KubernetesNodes CR must be named `\u003ccluster\u003e-\u003cpool\u003e`. The pool name (the part after `\u003ccluster\u003e-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.",
|
||||
"description": "Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-\u003ccluster\u003e`; the KubernetesNodes CR must be named `\u003ccluster\u003e-\u003cpool\u003e`. The pool's objects are named `kubernetes-\u003ccluster\u003e-\u003cpool\u003e`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over.",
|
||||
"type": "string",
|
||||
"default": "",
|
||||
"x-kubernetes-validations": [
|
||||
@@ -14,7 +14,7 @@
|
||||
]
|
||||
},
|
||||
"storageClass": {
|
||||
"description": "StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.",
|
||||
"description": "StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.",
|
||||
"type": "string",
|
||||
"default": "replicated",
|
||||
"x-kubernetes-validations": [
|
||||
@@ -38,7 +38,7 @@
|
||||
"default": 10
|
||||
},
|
||||
"instanceType": {
|
||||
"description": "Virtual machine instance type.",
|
||||
"description": "KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them.",
|
||||
"type": "string",
|
||||
"default": "u1.medium",
|
||||
"x-cozystack-options": {
|
||||
@@ -46,7 +46,7 @@
|
||||
}
|
||||
},
|
||||
"diskSize": {
|
||||
"description": "System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.",
|
||||
"description": "System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size.",
|
||||
"default": "20Gi",
|
||||
"pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$",
|
||||
"anyOf": [
|
||||
@@ -101,7 +101,7 @@
|
||||
}
|
||||
},
|
||||
"podCpuLimit": {
|
||||
"description": "CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.",
|
||||
"description": "CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content.",
|
||||
"pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$",
|
||||
"anyOf": [
|
||||
{
|
||||
@@ -114,7 +114,7 @@
|
||||
"x-kubernetes-int-or-string": true
|
||||
},
|
||||
"podCpuRequest": {
|
||||
"description": "CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.",
|
||||
"description": "CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content.",
|
||||
"pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$",
|
||||
"anyOf": [
|
||||
{
|
||||
@@ -147,7 +147,7 @@
|
||||
}
|
||||
},
|
||||
"kernelModules": {
|
||||
"description": "Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.",
|
||||
"description": "Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.",
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
@@ -170,7 +170,7 @@
|
||||
}
|
||||
},
|
||||
"kubelet": {
|
||||
"description": "Kubelet resource reservations for this pool.",
|
||||
"description": "Kubelet resource reservations for this pool. An empty reservation is computed from the worker's size: `resources` when set, otherwise the `instanceType`.",
|
||||
"type": "object",
|
||||
"default": {},
|
||||
"properties": {
|
||||
@@ -185,19 +185,19 @@
|
||||
"default": "10%"
|
||||
},
|
||||
"kubeReservedCpu": {
|
||||
"description": "CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.",
|
||||
"description": "CPU reserved for kubelet and container runtime. When empty, 5% of the worker's CPU, clamped to 50m..500m.",
|
||||
"type": "string"
|
||||
},
|
||||
"kubeReservedMemory": {
|
||||
"description": "Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.",
|
||||
"description": "Memory reserved for kubelet and container runtime. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi.",
|
||||
"type": "string"
|
||||
},
|
||||
"systemReservedCpu": {
|
||||
"description": "CPU reserved for host OS. Auto-computed from instanceType if empty.",
|
||||
"description": "CPU reserved for host OS. When empty, 5% of the worker's CPU, clamped to 50m..500m.",
|
||||
"type": "string"
|
||||
},
|
||||
"systemReservedMemory": {
|
||||
"description": "Memory reserved for host OS. Auto-computed from instanceType if empty.",
|
||||
"description": "Memory reserved for host OS. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi.",
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
@@ -213,27 +213,27 @@
|
||||
"default": "50%"
|
||||
},
|
||||
"nodeStartupTimeout": {
|
||||
"description": "Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.",
|
||||
"description": "Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (the Talos image import from the Image Factory, or a storage class slow to provision or clone the worker disk). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.",
|
||||
"type": "string",
|
||||
"default": "20m"
|
||||
},
|
||||
"readyUnknownTimeout": {
|
||||
"description": "How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.",
|
||||
"description": "How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips.",
|
||||
"type": "string",
|
||||
"default": "5m"
|
||||
},
|
||||
"readyFalseTimeout": {
|
||||
"description": "How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.",
|
||||
"description": "How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips.",
|
||||
"type": "string",
|
||||
"default": "10m"
|
||||
},
|
||||
"maxNodeProvisionTime": {
|
||||
"description": "How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted. Keep in sync with the parent kubernetes chart.",
|
||||
"description": "How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted.",
|
||||
"type": "string",
|
||||
"default": "30m"
|
||||
},
|
||||
"version": {
|
||||
"description": "Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos\u003c-\u003eKubernetes support matrix against `talos.version`.",
|
||||
"description": "Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool. The render fails when it is ahead. Its pairing with the Talos release the pool resolves (`talos.version`, or the `osImage` override) is checked against the chart's Talos\u003c-\u003eKubernetes support matrix: for a Talos minor the matrix lists, a Kubernetes minor outside that minor's list is refused; a Talos minor the matrix does not list is not checked.",
|
||||
"type": "string",
|
||||
"default": "v1.35",
|
||||
"enum": [
|
||||
@@ -268,7 +268,7 @@
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"imageFactoryURL": {
|
||||
"description": "Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.",
|
||||
"description": "Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. This redirects only the worker boot-disk import; the installer image in the worker machine config is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.",
|
||||
"type": "string"
|
||||
},
|
||||
"schematicID": {
|
||||
@@ -284,7 +284,7 @@
|
||||
}
|
||||
},
|
||||
"talos": {
|
||||
"description": "Talos worker image configuration. Keep in sync with the parent cluster's `talos`.",
|
||||
"description": "Talos worker image configuration for this pool.",
|
||||
"type": "object",
|
||||
"default": {},
|
||||
"required": [
|
||||
@@ -296,17 +296,17 @@
|
||||
],
|
||||
"properties": {
|
||||
"imageFactoryCA": {
|
||||
"description": "Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there.",
|
||||
"description": "Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. On `substrate: proxmox` it has no effect on the workers either: the ConfigMap is still rendered, but nothing references it.",
|
||||
"type": "string",
|
||||
"default": ""
|
||||
},
|
||||
"imageFactoryURL": {
|
||||
"description": "Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal.",
|
||||
"description": "Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. On `substrate: proxmox` the workers clone a VM template that already carries the OS, so this field has no effect there beyond that format check.",
|
||||
"type": "string",
|
||||
"default": "https://factory.talos.dev"
|
||||
},
|
||||
"installerRepository": {
|
||||
"description": "OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `\u003cinstallerRepository\u003e/\u003cschematicID\u003e:\u003cversion\u003e` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.",
|
||||
"description": "OCI repository prefix for the Talos installer image, written into the worker machine config as `machine.install.image`. Resolved as `\u003cinstallerRepository\u003e/\u003cschematicID\u003e:\u003cversion\u003e` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.",
|
||||
"type": "string",
|
||||
"default": "factory.talos.dev/installer"
|
||||
},
|
||||
@@ -322,7 +322,7 @@
|
||||
"default": "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
|
||||
},
|
||||
"version": {
|
||||
"description": "Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos\u003c-\u003eKubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.",
|
||||
"description": "Default Talos release for this pool's worker OS image and for the installer image written into the worker machine config. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos\u003c-\u003eKubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix lists, for each Talos minor the chart knows, the Kubernetes minors it supports and refuses a pool `version` outside that list; a Talos minor it does not list passes unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.",
|
||||
"type": "string",
|
||||
"default": "v1.13.6"
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
## @section Linkage
|
||||
##
|
||||
|
||||
## @param {string} cluster - Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool name (the part after `<cluster>-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.
|
||||
## @param {string} cluster - Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool's objects are named `kubernetes-<cluster>-<pool>`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over.
|
||||
## @immutable
|
||||
cluster: ""
|
||||
|
||||
@@ -22,7 +22,7 @@ cluster: ""
|
||||
## @section Common Parameters
|
||||
##
|
||||
|
||||
## @param {string} storageClass - StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.
|
||||
## @param {string} storageClass - StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.
|
||||
## @x-cozystack-options {source: storageclass}
|
||||
## @immutable
|
||||
storageClass: replicated
|
||||
@@ -44,10 +44,10 @@ storageClass: replicated
|
||||
## @field {[]string} [parameters] - Module parameters, each as a bare `key=value` string.
|
||||
|
||||
## @typedef {struct} Kubelet - Kubelet resource reservations for worker nodes.
|
||||
## @field {string} [systemReservedMemory] - Memory reserved for host OS. Auto-computed from instanceType if empty.
|
||||
## @field {string} [kubeReservedMemory] - Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.
|
||||
## @field {string} [systemReservedCpu] - CPU reserved for host OS. Auto-computed from instanceType if empty.
|
||||
## @field {string} [kubeReservedCpu] - CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.
|
||||
## @field {string} [systemReservedMemory] - Memory reserved for host OS. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi.
|
||||
## @field {string} [kubeReservedMemory] - Memory reserved for kubelet and container runtime. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi.
|
||||
## @field {string} [systemReservedCpu] - CPU reserved for host OS. When empty, 5% of the worker's CPU, clamped to 50m..500m.
|
||||
## @field {string} [kubeReservedCpu] - CPU reserved for kubelet and container runtime. When empty, 5% of the worker's CPU, clamped to 50m..500m.
|
||||
## @field {string} [evictionHardMemory]="7%" - Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).
|
||||
## @field {string} [evictionSoftMemory]="10%" - Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).
|
||||
|
||||
@@ -57,11 +57,11 @@ minReplicas: 0
|
||||
## @param {int} maxReplicas=10 - Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.
|
||||
maxReplicas: 10
|
||||
|
||||
## @param {string} instanceType="u1.medium" - Virtual machine instance type.
|
||||
## @param {string} instanceType="u1.medium" - KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them.
|
||||
## @x-cozystack-options {source: instancetype}
|
||||
instanceType: u1.medium
|
||||
|
||||
## @param {quantity} diskSize="20Gi" - System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.
|
||||
## @param {quantity} diskSize="20Gi" - System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size.
|
||||
diskSize: 20Gi
|
||||
|
||||
## @param {[]string} roles - List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.
|
||||
@@ -70,16 +70,16 @@ roles: []
|
||||
## @param {Resources} [resources] - Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `"010"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `"010"` reserves what `"10"` does.
|
||||
resources: {}
|
||||
|
||||
## @param {quantity} [podCpuLimit] - CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.
|
||||
## @param {quantity} [podCpuLimit] - CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content.
|
||||
|
||||
## @param {quantity} [podCpuRequest] - CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.
|
||||
## @param {quantity} [podCpuRequest] - CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content.
|
||||
|
||||
## @param {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
|
||||
gpus: []
|
||||
|
||||
## @param {[]KernelModule} [kernelModules] - Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.
|
||||
## @param {[]KernelModule} [kernelModules] - Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.
|
||||
|
||||
## @param {Kubelet} [kubelet] - Kubelet resource reservations for this pool.
|
||||
## @param {Kubelet} [kubelet] - Kubelet resource reservations for this pool. An empty reservation is computed from the worker's size: `resources` when set, otherwise the `instanceType`.
|
||||
kubelet: {}
|
||||
|
||||
## @param {bool} logSerialConsole=false - Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.
|
||||
@@ -88,16 +88,16 @@ logSerialConsole: false
|
||||
## @param {string} maxUnhealthy="50%" - Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At "50%" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two.
|
||||
maxUnhealthy: "50%"
|
||||
|
||||
## @param {string} nodeStartupTimeout="20m" - Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.
|
||||
## @param {string} nodeStartupTimeout="20m" - Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (the Talos image import from the Image Factory, or a storage class slow to provision or clone the worker disk). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.
|
||||
nodeStartupTimeout: "20m"
|
||||
|
||||
## @param {string} readyUnknownTimeout="5m" - How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.
|
||||
## @param {string} readyUnknownTimeout="5m" - How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips.
|
||||
readyUnknownTimeout: "5m"
|
||||
|
||||
## @param {string} readyFalseTimeout="10m" - How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.
|
||||
## @param {string} readyFalseTimeout="10m" - How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips.
|
||||
readyFalseTimeout: "10m"
|
||||
|
||||
## @param {string} maxNodeProvisionTime="30m" - How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted. Keep in sync with the parent kubernetes chart.
|
||||
## @param {string} maxNodeProvisionTime="30m" - How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted.
|
||||
maxNodeProvisionTime: "30m"
|
||||
|
||||
##
|
||||
@@ -111,23 +111,23 @@ maxNodeProvisionTime: "30m"
|
||||
## @value v1.32
|
||||
## @value v1.31
|
||||
|
||||
## @param {Version} version - Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`.
|
||||
## @param {Version} version - Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool. The render fails when it is ahead. Its pairing with the Talos release the pool resolves (`talos.version`, or the `osImage` override) is checked against the chart's Talos<->Kubernetes support matrix: for a Talos minor the matrix lists, a Kubernetes minor outside that minor's list is refused; a Talos minor the matrix does not list is not checked.
|
||||
version: "v1.35"
|
||||
|
||||
##
|
||||
## @section Talos Worker Image
|
||||
##
|
||||
|
||||
## @typedef {struct} WorkerImageBuiltin - Boot this pool's workers by CDI clone of a golden image from the opt-in worker image catalog (packages/system/kubernetes-worker-image) in cozy-public. The catalog package must be enabled (bundles.enabledPackages) and hold a matching (schematicID, version) entry, otherwise the render fails. A clone is a storage-layer CSI copy, so N workers no longer each stream the OS image over HTTP. Pair this with a StorageClass whose volumes are reachable from any node -- in practice a replicated/DRBD class. The copy runs between two volumes that a node-pinned class requires to be co-located, and nothing here arranges that: the golden in cozy-public has no consumer, so it binds as soon as it imports, on whichever node CDI's importer landed on, while a worker disk binds where its VM is scheduled. The platform also ships `cloneStrategyOverride: csi-clone`, which rules out the host-assisted `copy` strategy that would cross nodes. What a node-pinned class (`allowRemoteVolumeAccess: false`, WaitForFirstConsumer) actually does here has not been measured, so treat `builtin` on one as unsupported rather than slow.
|
||||
## @typedef {struct} WorkerImageBuiltin - Boot this pool's workers by CDI clone of a golden image from the opt-in worker image catalog (packages/system/kubernetes-worker-image) in cozy-public. The catalog package must be enabled (bundles.enabledPackages) and hold a matching (schematicID, version) entry, otherwise the render fails. A clone is a storage-layer CSI copy, so the workers do not each stream the OS image over HTTP. Pair this with a StorageClass whose volumes are reachable from any node -- in practice a replicated/DRBD class. The copy runs between two volumes that a node-pinned class requires to be co-located, and nothing here arranges that: the golden in cozy-public has no consumer, so it binds as soon as it imports, on whichever node CDI's importer landed on, while a worker disk binds where its VM is scheduled. The platform also ships `cloneStrategyOverride: csi-clone`, which rules out the host-assisted `copy` strategy that would cross nodes. What a node-pinned class (`allowRemoteVolumeAccess: false`, WaitForFirstConsumer) actually does here has not been measured, so treat `builtin` on one as unsupported rather than slow.
|
||||
## @field {string} [schematicID] - Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`.
|
||||
## @field {string} [version] - Talos version of the golden to clone. Defaults to `talos.version`.
|
||||
|
||||
## @typedef {struct} WorkerImageFactory - Boot this pool's workers by importing the Talos OS image over HTTP from an Image Factory (public, self-hosted, or caching mirror). This is the bring-your-own path and the default when `osImage` is omitted.
|
||||
## @field {string} [version] - Talos version. Defaults to `talos.version`.
|
||||
## @field {string} [schematicID] - Talos schematic ID. Defaults to `talos.schematicID`.
|
||||
## @field {string} [imageFactoryURL] - Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.
|
||||
## @field {string} [imageFactoryURL] - Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. This redirects only the worker boot-disk import; the installer image in the worker machine config is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.
|
||||
|
||||
## @typedef {struct} WorkerImage - Worker OS image source for this pool. Set exactly one of `builtin` or `factory`; when the whole field is omitted the workers import over HTTP using the pool's `talos.*` defaults (equivalent to an empty `factory`). The disk-system source is derived entirely from this field, so a pool with `osImage` omitted renders identically to a chart without this feature — adopting the feature never rolls existing workers. Editing this field on an existing pool changes the rendered source and therefore re-images (rolls) that pool, the same as changing any other image reference.
|
||||
## @typedef {struct} WorkerImage - Worker OS image source for this pool. Set exactly one of `builtin` or `factory`; when the whole field is omitted the workers import over HTTP using the pool's `talos.*` defaults (equivalent to an empty `factory`). The disk-system source is derived entirely from this field. Editing this field on an existing pool changes the rendered source and therefore re-images (rolls) that pool, the same as changing any other image reference.
|
||||
## @field {*WorkerImageBuiltin} [builtin] - Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled.
|
||||
## @field {*WorkerImageFactory} [factory] - Import from an Image Factory / mirror (default).
|
||||
|
||||
@@ -135,14 +135,14 @@ version: "v1.35"
|
||||
osImage: {}
|
||||
|
||||
## @typedef {struct} Talos - Talos worker image configuration.
|
||||
## @field {string} version - Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.
|
||||
## @field {string} version - Default Talos release for this pool's worker OS image and for the installer image written into the worker machine config. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix lists, for each Talos minor the chart knows, the Kubernetes minors it supports and refuses a pool `version` outside that list; a Talos minor it does not list passes unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.
|
||||
## @field {string} schematicID - Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID.
|
||||
## @field {string} imageFactoryURL - Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal.
|
||||
## @field {string} installerRepository - OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.
|
||||
## @field {string} imageFactoryURL - Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. On `substrate: proxmox` the workers clone a VM template that already carries the OS, so this field has no effect there beyond that format check.
|
||||
## @field {string} installerRepository - OCI repository prefix for the Talos installer image, written into the worker machine config as `machine.install.image`. Resolved as `<installerRepository>/<schematicID>:<version>` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.
|
||||
## @field {object} registryMirrors - Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.
|
||||
## @field {string} [imageFactoryCA] - Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there.
|
||||
## @field {string} [imageFactoryCA] - Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. On `substrate: proxmox` it has no effect on the workers either: the ConfigMap is still rendered, but nothing references it.
|
||||
|
||||
## @param {Talos} talos - Talos worker image configuration. Keep in sync with the parent cluster's `talos`.
|
||||
## @param {Talos} talos - Talos worker image configuration for this pool.
|
||||
talos:
|
||||
version: "v1.13.6"
|
||||
schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
|
||||
|
||||
@@ -8,11 +8,9 @@ Cozystack deploys and manages Kubernetes-as-a-service as standalone applications
|
||||
In Cozystack, such clusters are named tenant Kubernetes clusters, while the base Cozystack cluster is called a management or root cluster.
|
||||
Tenant clusters are fully separated from the management cluster and are intended for deploying tenant-specific or customer-developed applications.
|
||||
|
||||
Within a tenant cluster, users can take advantage of LoadBalancer services and easily provision physical volumes as needed.
|
||||
The control-plane operates within containers, while the worker nodes are deployed as virtual machines, all seamlessly managed by the application.
|
||||
Within a tenant cluster, users can take advantage of LoadBalancer services and easily provision physical volumes as needed. The control plane runs in containers in the management cluster, and the worker nodes are virtual machines, grouped into pools that are separate `KubernetesNodes` resources.
|
||||
|
||||
Kubernetes version in tenant clusters is independent of Kubernetes in the management cluster.
|
||||
Users can select the supported patch versions from 1.31 to 1.35.
|
||||
Kubernetes version in tenant clusters is independent of Kubernetes in the management cluster. Users choose the Kubernetes minor version, from 1.31 to 1.35, and the chart deploys the patch release it pins for that minor.
|
||||
|
||||
## Why Use a Managed Kubernetes Cluster?
|
||||
|
||||
@@ -58,14 +56,9 @@ Deploying it involves the following components:
|
||||
Each control plane pod includes essential components like `kube-apiserver`, `controller-manager`, and `scheduler`,
|
||||
allowing for efficient multi-tenancy and resource utilization.
|
||||
|
||||
- **Etcd Cluster**: A dedicated etcd cluster is deployed using the cozystack [etcd-operator](https://github.com/cozystack/etcd-operator) (`etcd-operator.cozystack.io/v1alpha2`).
|
||||
It provides reliable and scalable key-value storage for the Kubernetes control plane.
|
||||
- **Etcd**: the control plane keeps its state in the tenant's etcd, which this application does not deploy. It is the etcd the Tenant runs when `etcd` is enabled on it or on one of its parent tenants, built with the cozystack [etcd-operator](https://github.com/cozystack/etcd-operator), and every tenant Kubernetes cluster that uses it shares it. Until that etcd exists, the application creates only a `<release>-awaiting-etcd` ConfigMap that says so, and provisions the cluster once the etcd is there.
|
||||
|
||||
- **Worker Nodes**: Virtual Machines are provisioned to serve as worker nodes using KubeVirt.
|
||||
These nodes are configured to join the tenant Kubernetes cluster, enabling the deployment and management of workloads.
|
||||
Worker node disks automatically detect and match the underlying volume's block size
|
||||
(`blockSize.matchVolume`) to ensure compatibility with storage backends that use
|
||||
non-512-byte sectors, such as LINSTOR DRBD with 4Kn drives.
|
||||
- **Worker Nodes**: worker nodes are virtual machines running Talos Linux that join the tenant cluster. They come in pools, and each pool is a separate `KubernetesNodes` resource that names this cluster and sets its own size, disk, Talos image and scaling bounds; this application creates no worker nodes itself. With `substrate: kubevirt` the virtual machines run on KubeVirt in the management cluster, and with `substrate: proxmox` on an external Proxmox VE cluster.
|
||||
|
||||
- **Cluster API**: Cozystack is using the [Kubernetes Cluster API](https://cluster-api.sigs.k8s.io/) to provision the components of a cluster.
|
||||
|
||||
@@ -83,23 +76,11 @@ See the reference for components utilized in this service:
|
||||
- [github.com/kubernetes-sigs/cluster-api-provider-kubevirt](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt)
|
||||
- [github.com/kubevirt/csi-driver](https://github.com/kubevirt/csi-driver)
|
||||
|
||||
## Breaking Changes
|
||||
The top-level `storageClass` is the default StorageClass inside the tenant cluster and cannot be changed after the cluster is created; see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. Each worker pool sets the StorageClass of its own disks on its `KubernetesNodes` resource.
|
||||
|
||||
- **`ephemeralStorage` renamed to `diskSize`** (v1.4): The `nodeGroups[name].ephemeralStorage` field has been renamed to `nodeGroups[name].diskSize`. Existing clusters are migrated transparently by platform migration 41 during the pre-upgrade hook — no manual action is required. Newly written values should use `diskSize`. Existing VMs are rolling-updated via CAPI when the new values are applied. With the Talos worker rollover in this release the field now sizes the **single system disk** (Talos OS image streamed from `factory.talos.dev`, kubelet state, containerd image cache, local-path PVCs) — the pre-Talos `disk-kubelet` PVC layout was removed. State on that single disk persists across same-VM reboots (virt-launcher restart, guest reboot, node failure); VM replacement by CAPI (e.g. nodeGroup field change, MachineHealthCheck remediation) provisions a fresh disk.
|
||||
## Upgrading
|
||||
|
||||
- **Air-gapped tenant workers (Phase 1 Talos rollover)**: tenant worker bootstrap moves from Ubuntu containerDisk + kubeadm to a Talos OS disk image fetched over HTTP by CDI and a `.../installer/...` installer reference. Both sources are now overridable — set `talos.imageFactoryURL` (default `https://factory.talos.dev`) to a self-hosted Image Factory, a caching mirror, or an internal HTTP file server, and `talos.installerRepository` (default `factory.talos.dev/installer`) to a mirrored registry. The Helm-rendered `*-patch-containerd` Secret that previously plumbed the platform-wide `registries.mirrors` config to tenant workers (via the kubeadm template's containerd certs.d mount) has no consumer in the Talos machineconfig, so the Secret is removed together with the platform `registries` value that fed it. A Package that still sets `registries` keeps installing, the key is ignored. Registry mirrors for tenant workers, **in-guest container-image pulls** (pods pulling from registries inside the tenant cluster) included, are configured per worker pool through `talos.registryMirrors` of the `KubernetesNodes` resource.
|
||||
|
||||
- **Worker MachineHealthCheck remediation is now enabled by default** (Phase 1 Talos rollover): the worker MHC `maxUnhealthy` moved from a hard-coded `0` (remediation effectively **off**) to a configurable `nodeHealthCheck.maxUnhealthy` defaulting to `"50%"`. CAPI now auto-remediates (deletes and replaces) unhealthy worker Machines while at least 50% of a group stays healthy. `"50%"` deliberately leaves headroom for transient unhealthy nodes during the kubeadm→Talos rollover and slow first boots from `factory.talos.dev`; set `nodeHealthCheck.maxUnhealthy: "0%"` to keep the previous remediation-off behaviour until your fleet is stable on Talos workers.
|
||||
|
||||
- **GitOps-managed tenant `Kubernetes` CRs must bump `spec.version` in Git** (Phase 1 Talos rollover): platform migration 46 patches live `kuberneteses.apps.cozystack.io` objects from `v1.30` to `v1.31` (v1.30 left the Talos↔Kubernetes support matrix). When the tenant `Kubernetes` CR is reconciled from Git (Flux/Argo), the next source reconcile re-applies `version: v1.30` over migration 46's patch, which then trips the chart's `_versions.tpl` guard and the HelmRelease fails. Update `spec.version` to `v1.31` (or newer) in your Git source before/with the platform upgrade. Tenants managed via the API or dashboard need no action — migration 46 handles them.
|
||||
|
||||
- **Remote-accessible LINSTOR StorageClasses are auto-propagated to tenants** (v1.5): infra-cluster LINSTOR StorageClasses whose `linstor.csi.linbit.com/allowRemoteVolumeAccess` is not `"false"` are created inside each tenant under the same name (provisioned by `csi.kubevirt.io`). **Upgrade caveat:** delete any *manually created* tenant StorageClass whose name collides with a propagated infra class (e.g. a hand-made `replicated`) before upgrading, or the propagated class cannot be created and the tenant CSI release will not converge. Infra classes that must stay node-local need `allowRemoteVolumeAccess: "false"` set explicitly — an absent annotation is treated as remote-accessible. Propagation is evaluated at HelmRelease render time, so classes added or removed on the infra cluster after a tenant exists propagate only on that tenant's next reconcile.
|
||||
|
||||
- **Worker pools split into `KubernetesNodes` (Phase 2)**: `spec.nodeGroups`, `spec.nodeHealthCheck` and `spec.maxNodeProvisionTime` are removed from the `Kubernetes` CR — worker pools are now separate `KubernetesNodes` resources (one HelmRelease per pool; see the `kubernetes-nodes` chart), on which `spec.cluster` and `spec.storageClass` are immutable (`self == oldSelf`). Existing pools are adopted automatically on upgrade by platform migration 56 with no worker-VM churn. Consequences: (1) the `Kubernetes` CR's `WorkloadsReady` condition is now **control-plane-only** — worker health is reported on each `KubernetesNodes` resource, so a dashboard or `kubectl wait` watching the parent no longer reflects worker readiness; (2) enabling `addons.ingressNginx` now requires a pool carrying `roles: [ingress-nginx]` (the implicit default `md0` is gone — without such a pool the controller DaemonSet schedules nothing); (3) do **not** edit `Kubernetes` CRs during a platform upgrade — a mid-window parent render can transiently fail Helm ownership validation while migration 56 re-annotates worker objects, and self-heals once the new chart artifact lands; (4) when deleting a cluster, delete its `KubernetesNodes` pools **before** the parent `Kubernetes` resource, so each pool's pre-delete unpin hook can run while the tenant apiserver and the CAPI CRDs still exist — deleting the parent first can leave the pool release reconciling CAPI objects against a cluster that is being torn down (in the pre-split monolith the pools were part of the parent release and died with it; they are now separate resources with their own lifecycle); (5) the removed fields (`nodeGroups`, `nodeHealthCheck`, `maxNodeProvisionTime`) are still **accepted and stored** on an already-upgraded `Kubernetes` CR — the adoption migration deliberately leaves them in place rather than rewriting parent values mid-upgrade — but they have **no effect**: editing them (for example scaling via `spec.nodeGroups.<pool>.minReplicas`) does nothing, and the API returns an admission warning pointing you at the `KubernetesNodes` resources. A cluster name is also capped at 32 characters so every pool's `KubernetesNodes` release name fits the 53-character Helm limit. ComputePlane clusters (the `computeplane` module) are affected the same way: their worker pools are rendered as separate per-pool `KubernetesNodes` releases too (materialised directly by the module rather than adopted), so the same pool-name length limit applies to their `nodeGroups` keys.
|
||||
|
||||
- **`controlPlane.apiServer.extraArgs` refuses flags the tenant control plane owns** (Kamaji 26.9.4-edge): the tenant control plane now sets a list of kube-apiserver flags itself and drops an `extraArgs` entry naming one, where the previous build let the entry win. The entry stops taking effect once the new build reconciles the control plane, whatever the chart renders. Two of those flags have a field the control plane renders them from, and the chart moves an entry written as `--flag=value` into it while the field is empty: `--enable-admission-plugins` into `controlPlane.apiServer.admissionControllers`, `--kubelet-preferred-address-types` into `controlPlane.kubelet.preferredAddressTypes`. The KamajiControlPlane CRD accepts only the plugin names it lists, so a plugin outside that list that the apiserver leaves off by default can no longer be enabled, and a moved name outside it fails when the KamajiControlPlane is applied, with an error that names it, rather than at render. The chart refuses to render any other entry naming one of the flags, so a tenant carrying one stops reconciling until the entry is removed; the description of `controlPlane.apiServer.extraArgs` lists the flags. The same Kamaji update passes every `extraArgs` entry to the apiserver, so repeats of a flag no longer collapse to the last one: naming the kube feature gates both bare and `kube:`-prefixed across entries, or giving two `--emulated-version` entries that name kube, now stops the apiserver from starting. The chart refuses neither, in any `spec.oidc.mode`: both reach the apiserver as written. Keep one spelling for kube gate terms and one kube emulated version.
|
||||
|
||||
> The top-level `storageClass` field is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. The per-pool worker `storageClass` now lives on the `KubernetesNodes` resource (see the `kubernetes-nodes` chart).
|
||||
Changes that need action when Cozystack is upgraded are described in the upgrade notes of each release, for example [Upgrading from v1.6 to v1.7](https://cozystack.io/docs/v1.7/operations/cluster/upgrade-to-v1.7/).
|
||||
|
||||
## Proxmox substrate: credentials, privileges and tenant scope
|
||||
|
||||
@@ -133,36 +114,36 @@ A storage shared between tenants cannot be scoped: `Datastore.Allocate` on it is
|
||||
|
||||
### Application-specific Parameters
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| --------- | ---------------------------------------------------------------------------------------------- | -------- | ------- |
|
||||
| `version` | Kubernetes major.minor version to deploy | `string` | `v1.35` |
|
||||
| `host` | External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty. | `string` | `""` |
|
||||
| Name | Description | Type | Value |
|
||||
| --------- | --------------------------------------------------------------------------------------------------------- | -------- | ------- |
|
||||
| `version` | Kubernetes major.minor version to deploy | `string` | `v1.35` |
|
||||
| `host` | External hostname for Kubernetes cluster. Defaults to `kubernetes-<cluster-name>.<tenant-host>` if empty. | `string` | `""` |
|
||||
|
||||
|
||||
### Substrate
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------- |
|
||||
| `substrate` | Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one. | `string` | `kubevirt` |
|
||||
| `proxmox` | Proxmox substrate settings. | `object` | `{}` |
|
||||
| `proxmox.allowedNodes` | Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster. | `[]string` | `[]` |
|
||||
| `proxmox.dnsServers` | Nameservers written into each worker's network config. Required by the ProxmoxCluster schema. | `[]string` | `[]` |
|
||||
| `proxmox.insecure` | Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts. | `bool` | `false` |
|
||||
| `proxmox.ipv4Config` | Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional. | `object` | `{}` |
|
||||
| `proxmox.ipv4Config.addresses` | Ranges or CIDRs, e.g. `["10.0.0.120-10.0.0.170"]`. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address. | `[]string` | `[]` |
|
||||
| `proxmox.ipv4Config.gateway` | Default gateway for the workers. | `string` | `""` |
|
||||
| `proxmox.ipv4Config.prefix` | Netmask prefix length. | `int` | `24` |
|
||||
| `proxmox.ccm` | Proxmox cloud-controller-manager settings. | `object` | `{}` |
|
||||
| `proxmox.ccm.credentialsSecretName` | Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks. | `string` | `""` |
|
||||
| `proxmox.csi` | Proxmox CSI driver settings. | `object` | `{}` |
|
||||
| `proxmox.csi.enabled` | Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all. | `bool` | `true` |
|
||||
| `proxmox.csi.credentialsSecretName` | Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses` | StorageClasses to create in the tenant. | `[]object` | `[]` |
|
||||
| `proxmox.csi.storageClasses[i].name` | StorageClass name inside the tenant cluster. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].storage` | Proxmox storage id the volumes are created on (as it appears in `pvesm status`). | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].fstype` | Filesystem the driver formats volumes with: ext4 or xfs. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].ssd` | Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies. | `bool` | `false` |
|
||||
| `proxmox.csi.storageClasses[i].reclaimPolicy` | Delete or Retain. | `string` | `""` |
|
||||
| Name | Description | Type | Value |
|
||||
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------- |
|
||||
| `substrate` | Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one. | `string` | `kubevirt` |
|
||||
| `proxmox` | Proxmox substrate settings. | `object` | `{}` |
|
||||
| `proxmox.allowedNodes` | Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster. | `[]string` | `[]` |
|
||||
| `proxmox.dnsServers` | Nameservers written into each worker's network config. Required by the ProxmoxCluster schema. | `[]string` | `[]` |
|
||||
| `proxmox.insecure` | Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts. | `bool` | `false` |
|
||||
| `proxmox.ipv4Config` | Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional. | `object` | `{}` |
|
||||
| `proxmox.ipv4Config.addresses` | Ranges or CIDRs, e.g. `["10.0.0.120-10.0.0.170"]`. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address. | `[]string` | `[]` |
|
||||
| `proxmox.ipv4Config.gateway` | Default gateway for the workers. | `string` | `""` |
|
||||
| `proxmox.ipv4Config.prefix` | Netmask prefix length. | `int` | `24` |
|
||||
| `proxmox.ccm` | Proxmox cloud-controller-manager settings. | `object` | `{}` |
|
||||
| `proxmox.ccm.credentialsSecretName` | Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks. | `string` | `""` |
|
||||
| `proxmox.csi` | Proxmox CSI driver settings. | `object` | `{}` |
|
||||
| `proxmox.csi.enabled` | Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all. | `bool` | `true` |
|
||||
| `proxmox.csi.credentialsSecretName` | Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses` | StorageClasses to create in the tenant. | `[]object` | `[]` |
|
||||
| `proxmox.csi.storageClasses[i].name` | StorageClass name inside the tenant cluster. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].storage` | Proxmox storage id the volumes are created on (as it appears in `pvesm status`). | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].fstype` | Filesystem the driver formats volumes with: ext4 or xfs. | `string` | `""` |
|
||||
| `proxmox.csi.storageClasses[i].ssd` | Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies. | `bool` | `false` |
|
||||
| `proxmox.csi.storageClasses[i].reclaimPolicy` | Delete or Retain. | `string` | `""` |
|
||||
|
||||
|
||||
### Cluster Addons
|
||||
@@ -205,49 +186,37 @@ A storage shared between tenants cannot be scoped: `Datastore.Allocate` on it is
|
||||
|
||||
### Kubernetes Control Plane Configuration
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ----------- |
|
||||
| `controlPlane` | Kubernetes control-plane configuration. | `object` | `{}` |
|
||||
| `controlPlane.replicas` | Number of control-plane replicas. | `int` | `2` |
|
||||
| `controlPlane.apiServer` | API Server configuration. | `object` | `{}` |
|
||||
| `controlPlane.apiServer.resources` | CPU and memory resources for API Server. | `object` | `{}` |
|
||||
| `controlPlane.apiServer.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.apiServer.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.apiServer.resourcesPreset` | Preset if `resources` omitted. | `string` | `c1.medium` |
|
||||
| `controlPlane.apiServer.extraArgs` | Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior). | `[]string` | `[]` |
|
||||
| `controlPlane.apiServer.admissionControllers` | Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it. | `[]string` | `[]` |
|
||||
| `controlPlane.apiServer.extraVolumes` | Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default. | `[]object` | `[]` |
|
||||
| `controlPlane.apiServer.extraVolumeMounts` | Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default. | `[]object` | `[]` |
|
||||
| `controlPlane.controllerManager` | Controller Manager configuration. | `object` | `{}` |
|
||||
| `controlPlane.controllerManager.resources` | CPU and memory resources for Controller Manager. | `object` | `{}` |
|
||||
| `controlPlane.controllerManager.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.controllerManager.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.controllerManager.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.scheduler` | Scheduler configuration. | `object` | `{}` |
|
||||
| `controlPlane.scheduler.resources` | CPU and memory resources for Scheduler. | `object` | `{}` |
|
||||
| `controlPlane.scheduler.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.scheduler.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.scheduler.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.konnectivity` | Konnectivity configuration. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server` | Konnectivity Server configuration. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server.resources` | CPU and memory resources for Konnectivity. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.konnectivity.server.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.konnectivity.server.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.kubelet` | Kubelet connection settings. | `object` | `{}` |
|
||||
| `controlPlane.kubelet.preferredAddressTypes` | Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane `spec.kubelet.preferredAddressTypes`, which the control plane renders as `--kubelet-preferred-address-types`. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders `InternalIP`, `ExternalIP`, or the list of a `--kubelet-preferred-address-types=` entry in `controlPlane.apiServer.extraArgs` when there is one. | `[]string` | `[]` |
|
||||
|
||||
|
||||
### Talos Worker Image
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------ |
|
||||
| `talos` | Talos worker image configuration. | `object` | `{}` |
|
||||
| `talos.version` | Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`. | `string` | `v1.13.6` |
|
||||
| `talos.schematicID` | Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. | `string` | `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` |
|
||||
| `talos.imageFactoryURL` | Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. | `string` | `https://factory.talos.dev` |
|
||||
| `talos.installerRepository` | OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. | `string` | `factory.talos.dev/installer` |
|
||||
| `talos.registryMirrors` | Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. | `object` | `{}` |
|
||||
| Name | Description | Type | Value |
|
||||
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------- |
|
||||
| `controlPlane` | Kubernetes control-plane configuration. | `object` | `{}` |
|
||||
| `controlPlane.replicas` | Number of control-plane replicas. | `int` | `2` |
|
||||
| `controlPlane.apiServer` | API Server configuration. | `object` | `{}` |
|
||||
| `controlPlane.apiServer.resources` | CPU and memory resources for API Server. | `object` | `{}` |
|
||||
| `controlPlane.apiServer.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.apiServer.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.apiServer.resourcesPreset` | Preset if `resources` omitted. | `string` | `c1.medium` |
|
||||
| `controlPlane.apiServer.extraArgs` | Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default. | `[]string` | `[]` |
|
||||
| `controlPlane.apiServer.admissionControllers` | Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it. | `[]string` | `[]` |
|
||||
| `controlPlane.apiServer.extraVolumes` | Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default. | `[]object` | `[]` |
|
||||
| `controlPlane.apiServer.extraVolumeMounts` | Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default. | `[]object` | `[]` |
|
||||
| `controlPlane.controllerManager` | Controller Manager configuration. | `object` | `{}` |
|
||||
| `controlPlane.controllerManager.resources` | CPU and memory resources for Controller Manager. | `object` | `{}` |
|
||||
| `controlPlane.controllerManager.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.controllerManager.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.controllerManager.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.scheduler` | Scheduler configuration. | `object` | `{}` |
|
||||
| `controlPlane.scheduler.resources` | CPU and memory resources for Scheduler. | `object` | `{}` |
|
||||
| `controlPlane.scheduler.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.scheduler.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.scheduler.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.konnectivity` | Konnectivity configuration. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server` | Konnectivity Server configuration. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server.resources` | CPU and memory resources for Konnectivity. | `object` | `{}` |
|
||||
| `controlPlane.konnectivity.server.resources.cpu` | CPU available. | `quantity` | `""` |
|
||||
| `controlPlane.konnectivity.server.resources.memory` | Memory (RAM) available. | `quantity` | `""` |
|
||||
| `controlPlane.konnectivity.server.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` |
|
||||
| `controlPlane.kubelet` | Kubelet connection settings. | `object` | `{}` |
|
||||
| `controlPlane.kubelet.preferredAddressTypes` | Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane `spec.kubelet.preferredAddressTypes`, which the control plane renders as `--kubelet-preferred-address-types`. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders `InternalIP`, `ExternalIP`, or the list of a `--kubelet-preferred-address-types=` entry in `controlPlane.apiServer.extraArgs` when there is one. | `[]string` | `[]` |
|
||||
|
||||
|
||||
### OIDC
|
||||
|
||||
@@ -5,24 +5,6 @@ Expand the name of the chart.
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create a default fully qualified app name.
|
||||
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
|
||||
If release name contains chart name it will be used as a full name.
|
||||
*/}}
|
||||
{{- define "kubernetes.fullname" -}}
|
||||
{{- if .Values.fullnameOverride }}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- $name := default .Chart.Name .Values.nameOverride }}
|
||||
{{- if contains $name .Release.Name }}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create chart name and version as used by the chart label.
|
||||
*/}}
|
||||
@@ -185,29 +167,6 @@ never boots.
|
||||
{{- printf "has(claims.groups) && dyn(claims.groups).exists(g, g in [\"%s-view\", \"%s-use\", \"%s-admin\", \"%s-super-admin\"])" $ns $ns $ns $ns -}}
|
||||
{{- end }}
|
||||
|
||||
{{- /*
|
||||
Validates and returns a duration destined for a consumer that does not reject
|
||||
a bad value: the cluster-autoscaler parses its annotation with
|
||||
time.ParseDuration and silently falls back to its built-in default on a value
|
||||
it cannot parse, and the CAPI webhook does not validate unhealthyConditions
|
||||
timeouts at all, so a zero or negative one applies and remediates a Machine the
|
||||
moment its condition flips. The values schema types these fields as strings and
|
||||
stops there, which lets "30" and "-5m" reach the template. The accepted shape
|
||||
is narrower than Go's duration grammar on purpose, because that grammar also
|
||||
admits a value large enough to overflow the parser -- the silent fallback again
|
||||
-- and a fraction small enough to round to zero. The first segment must be
|
||||
positive; later segments may be zero, so the canonical rendering of a duration
|
||||
("20m0s", "1h0m0s" -- what metav1.Duration serializes to) round-trips. Every
|
||||
string this admits parses to a positive duration.
|
||||
*/ -}}
|
||||
{{- define "kubernetes.positiveDuration" -}}
|
||||
{{- $value := toString .value -}}
|
||||
{{- if not (regexMatch "^[1-9][0-9]{0,4}[smh]([0-9]{1,5}[smh]){0,2}$" $value) -}}
|
||||
{{- fail (printf "%s must be a whole number of s, m or h (e.g. 30m, 1h30m), got %q" .field $value) -}}
|
||||
{{- end -}}
|
||||
{{- $value -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/*
|
||||
The two kube-apiserver flags the tenant control plane owns that have a
|
||||
KamajiControlPlane field it renders them from, keyed to that field.
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
{{/*
|
||||
Copyright Broadcom, Inc. All Rights Reserved.
|
||||
SPDX-License-Identifier: APACHE-2.0
|
||||
*/}}
|
||||
|
||||
{{/* vim: set filetype=mustache: */}}
|
||||
|
||||
{{/*
|
||||
Return a resource request/limit object based on a given preset.
|
||||
These presets are for basic testing and not meant to be used in production
|
||||
{{ include "resources.preset" (dict "type" "nano") -}}
|
||||
*/}}
|
||||
{{- define "resources.preset" -}}
|
||||
{{- $presets := dict
|
||||
"nano" (dict
|
||||
"requests" (dict "cpu" "100m" "memory" "128Mi" "ephemeral-storage" "50Mi")
|
||||
"limits" (dict "memory" "128Mi" "ephemeral-storage" "2Gi")
|
||||
)
|
||||
"micro" (dict
|
||||
"requests" (dict "cpu" "250m" "memory" "256Mi" "ephemeral-storage" "50Mi")
|
||||
"limits" (dict "memory" "256Mi" "ephemeral-storage" "2Gi")
|
||||
)
|
||||
"small" (dict
|
||||
"requests" (dict "cpu" "500m" "memory" "512Mi" "ephemeral-storage" "50Mi")
|
||||
"limits" (dict "memory" "512Mi" "ephemeral-storage" "2Gi")
|
||||
)
|
||||
"medium" (dict
|
||||
"requests" (dict "cpu" "500m" "memory" "1Gi" "ephemeral-storage" "50Mi")
|
||||
"limits" (dict "memory" "1Gi" "ephemeral-storage" "2Gi")
|
||||
)
|
||||
"large" (dict
|
||||
"requests" (dict "cpu" "1" "memory" "2Gi" "ephemeral-storage" "50Mi")
|
||||
"limits" (dict "memory" "2Gi" "ephemeral-storage" "2Gi")
|
||||
)
|
||||
"xlarge" (dict
|
||||
"requests" (dict "cpu" "2" "memory" "4Gi" "ephemeral-storage" "50Mi")
|
||||
"limits" (dict "memory" "4Gi" "ephemeral-storage" "2Gi")
|
||||
)
|
||||
"2xlarge" (dict
|
||||
"requests" (dict "cpu" "4" "memory" "8Gi" "ephemeral-storage" "50Mi")
|
||||
"limits" (dict "memory" "8Gi" "ephemeral-storage" "2Gi")
|
||||
)
|
||||
}}
|
||||
{{- if hasKey $presets .type -}}
|
||||
{{- index $presets .type | toYaml -}}
|
||||
{{- else -}}
|
||||
{{- printf "ERROR: Preset key '%s' invalid. Allowed values are %s" .type (join "," (keys $presets)) | fail -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
@@ -1,29 +1,3 @@
|
||||
{{- /*
|
||||
Talos -> Kubernetes support matrix. Each Talos minor release supports a
|
||||
bounded window of Kubernetes minor versions; running a kubelet outside
|
||||
that window produces a silently broken Talos+kubelet combination that
|
||||
HelmRelease cannot detect. Fail the chart render instead, so the operator
|
||||
sees the mismatch in Flux/dashboards before workers roll over.
|
||||
|
||||
Upstream declares the window per Talos *minor* release, and patch releases
|
||||
never move it, so the lookup key is the minor derived from talos.version.
|
||||
Keying on the full patch version would silently disable this guard on every
|
||||
patch bump — exactly the mismatch it exists to catch.
|
||||
|
||||
The matrix only fires for Talos minors it knows about; operators who
|
||||
override talos.version to something outside this list take responsibility
|
||||
for compatibility themselves (extend the dict or upstream a PR).
|
||||
Source: https://docs.siderolabs.com/talos/v1.13/getting-started/support-matrix
|
||||
*/}}
|
||||
{{- $talosK8sSupportMatrix := dict
|
||||
"v1.13" (list "v1.31" "v1.32" "v1.33" "v1.34" "v1.35" "v1.36")
|
||||
}}
|
||||
{{- $talosMinor := regexFind "^v[0-9]+\\.[0-9]+" .Values.talos.version }}
|
||||
{{- with index $talosK8sSupportMatrix $talosMinor }}
|
||||
{{- if not (has $.Values.version .) }}
|
||||
{{- fail (printf "Kubernetes %s is not supported by Talos %s. Supported versions: %v. Bump tenant 'version' or set talos.version explicitly." $.Values.version $.Values.talos.version .) }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- $etcd := .Values._namespace.etcd }}
|
||||
{{- /* Substrate selects the infrastructure half of the cluster. The control
|
||||
plane is Kamaji either way, so this changes exactly one object — the
|
||||
@@ -66,20 +40,23 @@
|
||||
*/}}
|
||||
{{- /* Worker node pools (KubevirtMachineTemplate, MachineDeployment,
|
||||
MachineHealthCheck, worker WorkloadMonitor) and the Talos
|
||||
reconcile Job that produces each pool's TalosConfigTemplate now
|
||||
live in the separate `kubernetes-nodes` chart (one HelmRelease
|
||||
per pool). This chart renders the control plane only. The
|
||||
reconcile Job in the kubernetes-nodes release still re-patches
|
||||
this KamajiControlPlane's apiserver-Service ClusterIP-SAN and
|
||||
owns each TalosConfigTemplate via an ownerReference to this KCP,
|
||||
so garbage collection on control-plane uninstall is unchanged. */}}
|
||||
reconcile Job that produces each pool's TalosConfigTemplate are
|
||||
rendered by the `kubernetes-nodes` chart (one HelmRelease per
|
||||
pool). This chart renders the control plane only. The reconcile
|
||||
Job in the kubernetes-nodes release re-patches this
|
||||
KamajiControlPlane's apiserver-Service ClusterIP-SAN and owns
|
||||
each TalosConfigTemplate via an ownerReference to this KCP, so
|
||||
uninstalling the control plane garbage-collects them. */}}
|
||||
{{- /*
|
||||
Cluster pod/service CIDRs MUST match between the CAPI Cluster
|
||||
spec.clusterNetwork below and the Talos worker machineconfig's
|
||||
cluster.network — Talos surfaces overlaps against the host pod CIDR
|
||||
as a diagnostic at boot, and divergence between CAPI and Talos would
|
||||
manifest as silently broken kube-proxy / Cilium routing. Keep them
|
||||
in one place here.
|
||||
manifest as silently broken kube-proxy / Cilium routing. The worker
|
||||
side is a copy of these two literals in
|
||||
packages/apps/kubernetes-nodes/templates/_talosconfigtemplate.tpl, and
|
||||
templates/helmreleases/coredns.yaml pins the DNS Service IP inside the
|
||||
service CIDR, so a change here is a change in all three places.
|
||||
*/}}
|
||||
{{- $podCIDR := "10.243.0.0/16" }}
|
||||
{{- $serviceCIDR := "10.95.0.0/16" }}
|
||||
@@ -104,7 +81,7 @@ data:
|
||||
to reconcile - this HelmRelease will pick up the DataStore on its next
|
||||
5m reconcile loop and provision the cluster.
|
||||
{{- else }}
|
||||
{{- /* Phase 2 worker-pool adoption guard. Worker pools now live in per-pool
|
||||
{{- /* Worker-pool adoption guard. Worker pools live in per-pool
|
||||
kubernetes-nodes releases; the platform migration adopts each pre-split
|
||||
pool's live objects onto its child release (re-annotating
|
||||
meta.helm.sh/release-name, pinning helm.sh/resource-policy=keep) as a
|
||||
|
||||
@@ -23,9 +23,8 @@ ingress-nginx:
|
||||
{{- if and .Values.addons.ingressNginx.enabled .Values._namespace.etcd }}
|
||||
{{- /* The ingress-nginx controller DaemonSet pins itself to nodes labelled
|
||||
node-role.kubernetes.io/ingress-nginx (see cozystack.defaultIngressValues
|
||||
nodeSelector). Since Phase 2 moved worker pools into the separate
|
||||
kubernetes-nodes chart, that label is produced by a KubernetesNodes pool
|
||||
with roles: [ingress-nginx]. If no such pool exists the DaemonSet simply
|
||||
nodeSelector). That label is produced by a KubernetesNodes pool with
|
||||
roles: [ingress-nginx]. If no such pool exists the DaemonSet simply
|
||||
schedules nothing (degraded, not failed) — the requirement is documented
|
||||
in NOTES.txt and the release note rather than enforced at render time, so
|
||||
an existing cluster is never flipped from Ready to Failed on upgrade. */}}
|
||||
|
||||
@@ -6,8 +6,8 @@
|
||||
/etc/kubernetes/authentication-config/config.yaml and consumed via
|
||||
--authentication-config=. Compared to the legacy `--oidc-*` flags,
|
||||
the file form supports multiple issuers (the `jwt` block is a list)
|
||||
and inline private-CA PEM, so future BYO and Phase-2 issuers extend
|
||||
the same Secret instead of fighting the chart shape.
|
||||
and inline private-CA PEM, so further issuers extend the same Secret
|
||||
instead of fighting the chart shape.
|
||||
|
||||
System mode:
|
||||
Single JWT entry built from `_cluster.root-host` (the cozy realm
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{{- /*
|
||||
Phase 1 — System mode: render one per-cluster Keycloak client and a
|
||||
System mode: render one per-cluster Keycloak client and a
|
||||
per-cluster audience client scope inside the platform `cozy` realm.
|
||||
|
||||
The clientId is namespace-prefixed so it is globally unique across the
|
||||
|
||||
@@ -8,10 +8,9 @@
|
||||
kube-system, which the management cluster can only reach via the
|
||||
Kamaji-issued admin kubeconfig stored as <release>-admin-kubeconfig.
|
||||
|
||||
Why a main-phase resource and not a Helm post-install/post-upgrade hook
|
||||
(the earlier shape): the hook is unreliable on the very failure it was
|
||||
meant to cover. This release carries
|
||||
release.cozystack.io/helm-install-disable-wait: "true" (cozystack-api
|
||||
Why a main-phase resource and not a Helm post-install/post-upgrade hook:
|
||||
the hook is unreliable on the very failure it was meant to cover. This
|
||||
release carries release.cozystack.io/helm-install-disable-wait: "true" (cozystack-api
|
||||
sets it on both Install.DisableWait and Upgrade.DisableWait), so there
|
||||
is NO main-resource wait phase in front of the hooks — a post-install
|
||||
hook is not gated on MachineDeployment Ready and does fire on a fresh
|
||||
@@ -25,9 +24,9 @@
|
||||
<release>-admin-kubeconfig (materialised by Kamaji after the
|
||||
KamajiControlPlane is applied) is not yet available.
|
||||
|
||||
As an ordinary main-phase resource (mirroring talos-reconcile-job.yaml)
|
||||
the Job is applied on every install and every reconcile regardless of
|
||||
hook phase. It lands together with the KamajiControlPlane and
|
||||
As an ordinary main-phase resource (like talos-reconcile-job.yaml in the
|
||||
kubernetes-nodes chart) the Job is applied on every install and every
|
||||
reconcile regardless of hook phase. It lands together with the KamajiControlPlane and
|
||||
talos-secrets; a wait-for-kubeconfig init container holds the Pod until
|
||||
Kamaji publishes the admin kubeconfig Secret (the Secret volume is
|
||||
optional so kubelet does not FailedMount in the meantime), then the
|
||||
@@ -40,9 +39,9 @@
|
||||
upgrade that lands while the Job is live (or inside
|
||||
ttlSecondsAfterFinished) with `spec.template: field is immutable`, and
|
||||
RetryOnFailure would loop on it. The Job therefore carries a
|
||||
content-hash name suffix over its whole rendered spec (mirrors
|
||||
talos-reconcile-job.yaml and the KubevirtMachineTemplate idiom in
|
||||
cluster.yaml): an identical render keeps the name and Helm no-ops it;
|
||||
content-hash name suffix over its whole rendered spec (the idiom of
|
||||
talos-reconcile-job.yaml and the KubevirtMachineTemplate name in the
|
||||
kubernetes-nodes chart): an identical render keeps the name and Helm no-ops it;
|
||||
any change that flows into the spec — notably a new images/kubectl.tag —
|
||||
yields a new name so a fresh Job runs and Helm prunes the old one. The
|
||||
hash also folds in the looked-up bootstrap token — both halves,
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
suite: control-plane-only cluster.yaml render (worker pools moved to kubernetes-nodes)
|
||||
suite: control-plane-only cluster.yaml render (worker pools are kubernetes-nodes releases)
|
||||
|
||||
# Phase 2 moved the worker node pools (KubevirtMachineTemplate, MachineDeployment,
|
||||
# The worker node pools (KubevirtMachineTemplate, MachineDeployment,
|
||||
# MachineHealthCheck, worker WorkloadMonitor) and the per-pool talos-reconcile Job
|
||||
# into the separate kubernetes-nodes chart. cluster.yaml now renders the control
|
||||
# are rendered by the kubernetes-nodes chart. cluster.yaml renders the control
|
||||
# plane only, in source order: Cluster (0), KamajiControlPlane (1), the
|
||||
# control-plane WorkloadMonitor (2), KubevirtCluster (3). This suite pins that
|
||||
# surface so a regression that re-introduces pool objects into the parent chart
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
suite: tenant kube-apiserver OIDC (Phase 1)
|
||||
suite: tenant kube-apiserver OIDC
|
||||
|
||||
values:
|
||||
- values/common.yaml
|
||||
|
||||
# In cluster.yaml the KamajiControlPlane is document index 1
|
||||
# (0: Cluster, 1: KamajiControlPlane, 2: WorkloadMonitor, 3: KubevirtCluster, ...).
|
||||
# Documents at higher indices are KubevirtMachineTemplate, MachineDeployment,
|
||||
# MachineHealthCheck, WorkloadMonitor per node group.
|
||||
# (0: Cluster, 1: KamajiControlPlane, 2: WorkloadMonitor, 3: KubevirtCluster);
|
||||
# tests/control_plane_only_test.yaml pins that there are no others.
|
||||
|
||||
tests:
|
||||
###############################################
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
suite: stored Kubernetes specs that still carry a talos block
|
||||
|
||||
# The chart declares no `talos` key: worker pools are KubernetesNodes releases
|
||||
# and read their own `talos`, and the Talos to Kubernetes support matrix is
|
||||
# checked there. A Kubernetes spec stored before that still carries the block,
|
||||
# and both the values schema (no additionalProperties: false at the root) and
|
||||
# the aggregated API accept it, so the chart has to keep rendering it as if it
|
||||
# were absent.
|
||||
|
||||
templates:
|
||||
- templates/cluster.yaml
|
||||
|
||||
values:
|
||||
- values/common.yaml
|
||||
|
||||
release:
|
||||
name: test-k8s
|
||||
namespace: tenant-test
|
||||
|
||||
tests:
|
||||
- it: renders the same control plane for a stored spec carrying every talos key the chart used to declare
|
||||
set:
|
||||
talos:
|
||||
version: "v1.13.6"
|
||||
schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
|
||||
imageFactoryURL: "https://factory.example.com"
|
||||
installerRepository: "registry.example.com/installer"
|
||||
registryMirrors:
|
||||
ghcr.io:
|
||||
endpoints:
|
||||
- https://registry.example.com
|
||||
version: "v1.31"
|
||||
asserts:
|
||||
- hasDocuments:
|
||||
count: 4
|
||||
- isKind:
|
||||
of: Cluster
|
||||
documentIndex: 0
|
||||
- isKind:
|
||||
of: KamajiControlPlane
|
||||
documentIndex: 1
|
||||
@@ -1,53 +0,0 @@
|
||||
suite: Talos to Kubernetes support-matrix guard tests
|
||||
|
||||
# Each Talos minor release supports a bounded window of Kubernetes minor
|
||||
# versions; running a kubelet outside that window produces a silently
|
||||
# broken Talos+kubelet combination that HelmRelease cannot detect. The
|
||||
# chart fails render with a clear message instead, so the operator sees
|
||||
# the mismatch in Flux/dashboards before workers roll over. Source of
|
||||
# truth is the $talosK8sSupportMatrix dict at the top of cluster.yaml.
|
||||
#
|
||||
# The dict is keyed by Talos minor, and the template derives that minor from
|
||||
# talos.version, so every patch release of a supported minor stays guarded.
|
||||
#
|
||||
# Failure-path coverage is left as a follow-up: values.schema.json
|
||||
# enforces Version enum (v1.31..v1.35), which is strictly tighter than
|
||||
# the Talos v1.13 matrix entry (v1.31..v1.36), so the only way to
|
||||
# trigger the fail() guard via helm-unittest is to feed a Version the
|
||||
# schema rejects before the template ever runs. Schema-rejection happens
|
||||
# before template execution, so the chart fail() message is unreachable
|
||||
# from the test harness until either the matrix gains an exclusion the
|
||||
# schema enum still allows, or helm-unittest gains a schema-skip toggle.
|
||||
|
||||
templates:
|
||||
- templates/cluster.yaml
|
||||
|
||||
values:
|
||||
- values/common.yaml
|
||||
|
||||
tests:
|
||||
- it: accepts a Kubernetes version inside the Talos v1.13 support window
|
||||
release:
|
||||
name: test-k8s
|
||||
namespace: tenant-test
|
||||
set:
|
||||
talos:
|
||||
version: "v1.13.6"
|
||||
schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
|
||||
version: "v1.31"
|
||||
asserts:
|
||||
- hasDocuments:
|
||||
count: 4
|
||||
|
||||
- it: keeps rendering for an older patch release of the same Talos minor
|
||||
release:
|
||||
name: test-k8s
|
||||
namespace: tenant-test
|
||||
set:
|
||||
talos:
|
||||
version: "v1.13.0"
|
||||
schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
|
||||
version: "v1.31"
|
||||
asserts:
|
||||
- hasDocuments:
|
||||
count: 4
|
||||
@@ -19,5 +19,5 @@ controlPlane:
|
||||
server:
|
||||
resources: {}
|
||||
resourcesPreset: "micro"
|
||||
# Worker pools are managed via KubernetesNodes resources since the Phase 2 split;
|
||||
# the kubernetes chart renders the control plane only.
|
||||
# Worker pools are KubernetesNodes resources; the kubernetes chart renders the
|
||||
# control plane only.
|
||||
@@ -1,4 +1,4 @@
|
||||
suite: Phase 2 worker-pool adoption guard
|
||||
suite: worker-pool adoption guard
|
||||
|
||||
# The guard in templates/cluster.yaml refuses to render the control-plane-only
|
||||
# chart when a live worker MachineDeployment for this cluster is NOT owned by
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"type": "object",
|
||||
"x-cozystack-name": {
|
||||
"type": "string",
|
||||
"description": "Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes-\u003cname\u003e-\u003cpool\u003e`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.",
|
||||
"description": "Cluster name. Each worker pool is a KubernetesNodes release named `kubernetes-nodes-\u003cname\u003e-\u003cpool\u003e`, which has to fit the 53-character Helm release name limit, so a cluster name of at most 32 characters leaves at least three characters for the pool name.",
|
||||
"maxLength": 32
|
||||
},
|
||||
"properties": {
|
||||
@@ -34,12 +34,12 @@
|
||||
]
|
||||
},
|
||||
"host": {
|
||||
"description": "External hostname for Kubernetes cluster. Defaults to `\u003ccluster-name\u003e.\u003ctenant-host\u003e` if empty.",
|
||||
"description": "External hostname for Kubernetes cluster. Defaults to `kubernetes-\u003ccluster-name\u003e.\u003ctenant-host\u003e` if empty.",
|
||||
"type": "string",
|
||||
"default": ""
|
||||
},
|
||||
"substrate": {
|
||||
"description": "Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.",
|
||||
"description": "Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one.",
|
||||
"type": "string",
|
||||
"default": "kubevirt",
|
||||
"x-kubernetes-validations": [
|
||||
@@ -506,7 +506,7 @@
|
||||
}
|
||||
},
|
||||
"extraArgs": {
|
||||
"description": "Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior).",
|
||||
"description": "Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default.",
|
||||
"type": "array",
|
||||
"default": [],
|
||||
"items": {
|
||||
@@ -953,46 +953,6 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"talos": {
|
||||
"description": "Talos worker image configuration.",
|
||||
"type": "object",
|
||||
"default": {},
|
||||
"required": [
|
||||
"imageFactoryURL",
|
||||
"installerRepository",
|
||||
"registryMirrors",
|
||||
"schematicID",
|
||||
"version"
|
||||
],
|
||||
"properties": {
|
||||
"imageFactoryURL": {
|
||||
"description": "Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash.",
|
||||
"type": "string",
|
||||
"default": "https://factory.talos.dev"
|
||||
},
|
||||
"installerRepository": {
|
||||
"description": "OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `\u003cinstallerRepository\u003e/\u003cschematicID\u003e:\u003cversion\u003e`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.",
|
||||
"type": "string",
|
||||
"default": "factory.talos.dev/installer"
|
||||
},
|
||||
"registryMirrors": {
|
||||
"description": "Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap.",
|
||||
"type": "object",
|
||||
"default": {},
|
||||
"x-kubernetes-preserve-unknown-fields": true
|
||||
},
|
||||
"schematicID": {
|
||||
"description": "Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here.",
|
||||
"type": "string",
|
||||
"default": "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
|
||||
},
|
||||
"version": {
|
||||
"description": "Talos release used for worker OS image and installer. Must satisfy the chart's Talos\u003c-\u003eKubernetes support matrix against the chosen `version`.",
|
||||
"type": "string",
|
||||
"default": "v1.13.6"
|
||||
}
|
||||
}
|
||||
},
|
||||
"oidc": {
|
||||
"description": "OIDC authentication and per-user RBAC for the tenant kube-apiserver. See https://cozystack.io/docs/v1.6/kubernetes/oidc-authentication/ for the operator guide.",
|
||||
"type": "object",
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
## @name {string} - Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.
|
||||
## @name {string} - Cluster name. Each worker pool is a KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, which has to fit the 53-character Helm release name limit, so a cluster name of at most 32 characters leaves at least three characters for the pool name.
|
||||
## @maxLength 32
|
||||
|
||||
##
|
||||
@@ -79,13 +79,13 @@ storageClass: replicated
|
||||
version: "v1.35"
|
||||
|
||||
|
||||
## @param {string} host - External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty.
|
||||
## @param {string} host - External hostname for Kubernetes cluster. Defaults to `kubernetes-<cluster-name>.<tenant-host>` if empty.
|
||||
host: ""
|
||||
|
||||
##
|
||||
## @section Substrate
|
||||
|
||||
## @param {string} substrate="kubevirt" - Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.
|
||||
## @param {string} substrate="kubevirt" - Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one.
|
||||
## @immutable
|
||||
substrate: kubevirt
|
||||
|
||||
@@ -288,7 +288,7 @@ addons:
|
||||
## @typedef {struct} APIServer - API Server configuration.
|
||||
## @field {Resources} resources - CPU and memory resources for API Server.
|
||||
## @field {ResourcesPreset} resourcesPreset="c1.medium" - Preset if `resources` omitted.
|
||||
## @field {[]string} extraArgs - Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior).
|
||||
## @field {[]string} extraArgs - Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default.
|
||||
## @field {[]AdmissionController} admissionControllers - Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it.
|
||||
## @field {[]object} extraVolumes - Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default.
|
||||
## @field {[]object} extraVolumeMounts - Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default.
|
||||
@@ -342,25 +342,6 @@ controlPlane:
|
||||
kubelet:
|
||||
preferredAddressTypes: []
|
||||
|
||||
##
|
||||
## @section Talos Worker Image
|
||||
##
|
||||
|
||||
## @typedef {struct} Talos - Talos worker image configuration.
|
||||
## @field {string} version - Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`.
|
||||
## @field {string} schematicID - Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here.
|
||||
## @field {string} imageFactoryURL - Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash.
|
||||
## @field {string} installerRepository - OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.
|
||||
## @field {object} registryMirrors - Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap.
|
||||
|
||||
## @param {Talos} talos - Talos worker image configuration.
|
||||
talos:
|
||||
version: "v1.13.6"
|
||||
schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
|
||||
imageFactoryURL: "https://factory.talos.dev"
|
||||
installerRepository: "factory.talos.dev/installer"
|
||||
registryMirrors: {}
|
||||
|
||||
##
|
||||
## @section OIDC
|
||||
##
|
||||
@@ -385,7 +366,7 @@ talos:
|
||||
## @field {string} [config] - Inline AuthenticationConfiguration YAML (`apiserver.config.k8s.io/v1beta1`). The chart writes the value verbatim into a Secret mounted at `/etc/kubernetes/authentication-config/config.yaml` on the kube-apiserver. Mutually exclusive with `secretRef.name`.
|
||||
## @field {OIDCSecretRef} [secretRef] - Reference to an existing Secret in the tenant namespace carrying the `AuthenticationConfiguration`.
|
||||
|
||||
## @typedef {struct} OIDC - OIDC selector for the tenant cluster (Phase 1 — per-cluster audience binding; no per-tenant Keycloak realm).
|
||||
## @typedef {struct} OIDC - OIDC selector for the tenant cluster. There is no per-tenant Keycloak realm: `System` mode uses the platform realm with a public client and an audience of its own per cluster.
|
||||
## @field {OIDCMode} mode - Identity mode. `None`: no OIDC, only the static admin kubeconfig works. `System`: trust the platform Keycloak realm (`cozy` by default) via a per-cluster public client with audience binding; zero-config default. `CustomConfig`: trust a tenant-supplied issuer directly (BYO); `cozy` is not in the path.
|
||||
## @field {OIDCCustomConfig} [customConfig] - Tenant-supplied AuthenticationConfiguration; consumed only when `mode: CustomConfig`.
|
||||
## @field {[]OIDCUser} [users] - Users granted access to the tenant cluster; each entry produces one ClusterRoleBinding inside the tenant cluster. Works for both `System` and `CustomConfig` modes.
|
||||
|
||||
@@ -21,7 +21,7 @@ spec:
|
||||
singular: kubernetesnodes
|
||||
plural: kubernetesnodeses
|
||||
openAPISchema: |-
|
||||
{"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool name (the part after `<cluster>-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `\"010\"` is eight cores, and the autoscaler capacity countLine truncated
|
||||
{"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-<cluster>`; the KubernetesNodes CR must be named `<cluster>-<pool>`. The pool's objects are named `kubernetes-<cluster>-<pool>`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `iLine truncated
|
||||
release:
|
||||
labels:
|
||||
sharding.fluxcd.io/key: tenants
|
||||
|
||||
+2
-2
@@ -32,7 +32,7 @@ spec:
|
||||
singular: kubernetes
|
||||
plural: kuberneteses
|
||||
openAPISchema: |-
|
||||
{"title":"Chart Values","type":"object","x-cozystack-name":{"type":"string","description":"Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.","maxLength":32},"properties":{"storageClass":{"description":"Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"version":{"description":"Kubernetes major.minor version to deploy","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"host":{"description":"External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty.","type":"string","default":""},"substrate":{"description":"Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["ipv4Config"],"properties":{"allowedNodes":{"description":"Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster.","type":"array","default":[],"items":{"type":"string"}},"ccm":{"description":"Proxmox cloud-controller-manager settings.","type":"object","default":{},"required":["credentialsSecretName"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks.","type":"string","default":""}}},"csi":{"description":"Proxmox CSI driver settings.","type":"object","default":{},"required":["credentialsSecretName","enabled"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs.","type":"string","default":""},"enabled":{"description":"Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all.","type":"boolean","default":true},"storageClasses":{"description":"StorageClasses to create in the tenant.","type":"array","default":[],"items":{"type":"object","required":["name","storage"],"properties":{"fstype":{"description":"Filesystem the driver formats volumes with: ext4 or xfs.","type":"string"},"name":{"description":"StorageClass name inside the tenant cluster.","type":"string"},"reclaimPolicy":{"description":"Delete or Retain.","type":"string"},"ssd":{"description":"Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies.","type":"boolean"},"storage":{"description":"Proxmox storage id the volumes are created on (as it appears in `pvesm status`).","type":"string"}}}}}},"dnsServers":{"description":"Nameservers written into each worker's network config. Required by the ProxmoxCluster schema.","type":"array","default":[],"items":{"type":"string"}},"insecure":{"description":"Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts.","type":"boolean","default":false},"ipv4Config":{"description":"Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional.","type":"object","default":{},"required":["gateway","prefix"],"properties":{"addresses":{"description":"Ranges or CIDRs, e.g. `[\"10.0.0.120-10.0.0.170\"]`. capmox turns these into an InClusterIPPool, so they must not ovLine truncated
|
||||
{"title":"Chart Values","type":"object","x-cozystack-name":{"type":"string","description":"Cluster name. Each worker pool is a KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, which has to fit the 53-character Helm release name limit, so a cluster name of at most 32 characters leaves at least three characters for the pool name.","maxLength":32},"properties":{"storageClass":{"description":"Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"version":{"description":"Kubernetes major.minor version to deploy","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"host":{"description":"External hostname for Kubernetes cluster. Defaults to `kubernetes-<cluster-name>.<tenant-host>` if empty.","type":"string","default":""},"substrate":{"description":"Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["ipv4Config"],"properties":{"allowedNodes":{"description":"Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster.","type":"array","default":[],"items":{"type":"string"}},"ccm":{"description":"Proxmox cloud-controller-manager settings.","type":"object","default":{},"required":["credentialsSecretName"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks.","type":"string","default":""}}},"csi":{"description":"Proxmox CSI driver settings.","type":"object","default":{},"required":["credentialsSecretName","enabled"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs.","type":"string","default":""},"enabled":{"description":"Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all.","type":"boolean","default":true},"storageClasses":{"description":"StorageClasses to create in the tenant.","type":"array","default":[],"items":{"type":"object","required":["name","storage"],"properties":{"fstype":{"description":"Filesystem the driver formats volumes with: ext4 or xfs.","type":"string"},"name":{"description":"StorageClass name inside the tenant cluster.","type":"string"},"reclaimPolicy":{"description":"Delete or Retain.","type":"string"},"ssd":{"description":"Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies.","type":"boolean"},"storage":{"description":"Proxmox storage id the volumes are created on (as it appears in `pvesm status`).","type":"string"}}}}}},"dnsServers":{"description":"Nameservers written into each worker's network config. Required by the ProxmoxCluster schema.","type":"array","default":[],"items":{"type":"string"}},"insecure":{"description":"Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts.","type":"boolean","default":false},"ipv4Config":{"description":"Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional.","type":"object","default":{},"required":["gateway","prefix"],"properties":{"addresses":{"description":"Ranges or CIDRs, e.g. `[\"10.0.0.120-10.0.0.170\"]`. capmox turns these into an InClusterIPPool, so they must not overlLine truncated
|
||||
release:
|
||||
prefix: kubernetes-
|
||||
labels:
|
||||
@@ -50,7 +50,7 @@ spec:
|
||||
tags:
|
||||
- compute
|
||||
icon: PHN2ZyB3aWR0aD0iMTQ0IiBoZWlnaHQ9IjE0NCIgdmlld0JveD0iMCAwIDE0NCAxNDQiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CjxyZWN0IHdpZHRoPSIxNDQiIGhlaWdodD0iMTQ0IiByeD0iMjQiIGZpbGw9InVybCgjcGFpbnQwX2xpbmVhcl82ODFfMjg0NSkiLz4KPHBhdGggZD0iTTcxLjk5NjggMTlDNzAuMzAzOSAxOS4wMDAyIDY4LjkzMTIgMjAuNTMyMiA2OC45MzE0IDIyLjQyMjFDNjguOTMxNCAyMi40NTExIDY4LjkzNzMgMjIuNDc4OCA2OC45Mzc5IDIyLjUwNzZDNjguOTM1NCAyMi43NjQ0IDY4LjkyMzEgMjMuMDczNyA2OC45MzE0IDIzLjI5NzNDNjguOTcxNyAyNC4zODczIDY5LjIwODIgMjUuMjIxNiA2OS4zNTA2IDI2LjIyNThDNjkuNjA4NCAyOC4zNzUyIDY5LjgyNDUgMzAuMTU2OSA2OS42OTEyIDMxLjgxM0M2OS41NjE1IDMyLjQzNzUgNjkuMTAzNyAzMy4wMDg2IDY4LjY5NTYgMzMuNDA1Nkw2OC42MjM1IDM0LjcwODZDNjYuNzgzOSAzNC44NjE3IDY0LjkzMTkgMzUuMTQyMSA2My4wODIxIDM1LjU2NDFDNTUuMTIyNiAzNy4zNzk4IDQ4LjI2OTUgNDEuNDk5MSA0My4wNTIgNDcuMDYwOUM0Mi43MTM0IDQ2LjgyODggNDIuMTIxMSA0Ni40MDE5IDQxLjk0NSA0Ni4yNzEyQzQxLjM5NzcgNDYuMzQ1NCA0MC44NDQ1IDQ2LjUxNTEgNDAuMTI0MSA0Ni4wOTM1QzM4Ljc1MjIgNDUuMTY1NyAzNy41MDI4IDQzLjg4NTEgMzUuOTkxIDQyLjM0MjRDMzUuMjk4MiA0MS42MDQ0IDM0Ljc5NjYgNDAuOTAxOCAzMy45NzM1IDQwLjE5MDRDMzMuNzg2NiA0MC4wMjg5IDMzLjUwMTQgMzkuODEwNCAzMy4yOTIzIDM5LjY0NDJDMzIuNjQ4OSAzOS4xMjg4IDMxLjg5IDM4Ljg2IDMxLjE1NyAzOC44MzQ4QzMwLjIxNDcgMzguODAyNCAyOS4zMDc1IDM5LjE3MjUgMjguNzEzOCAzOS45MjA2QzI3LjY1ODQgNDEuMjUwNiAyNy45OTYzIDQzLjI4MzMgMjkuNDY3MSA0NC40NjE0QzI5LjQ4MiA0NC40NzM0IDI5LjQ5NzkgNDQuNDgyNyAyOS41MTI5IDQ0LjQ5NDNDMjkuNzE1IDQ0LjY1ODkgMjkuOTYyNSA0NC44Njk4IDMwLjE0ODMgNDUuMDA3NkMzMS4wMjE3IDQ1LjY1NTUgMzEuODE5NSA0NS45ODcyIDMyLjY4OTcgNDYuNTAxNUMzNC41MjMxIDQ3LjYzOTEgMzYuMDQzIDQ4LjU4MjMgMzcuMjQ4NiA0OS43MTk2QzM3LjcxOTQgNTAuMjIzNyAzNy44MDE2IDUxLjExMjIgMzcuODY0MyA1MS40OTY0TDM4Ljg0NjggNTIuMzc4MkMzMy41ODcyIDYwLjMzMDggMzEuMTUzIDcwLjE1MzkgMzIuNTkxNSA4MC4xNjI3TDMxLjMwNzcgODAuNTM3OEMzMC45NjkzIDgwLjk3NjggMzAuNDkxMiA4MS42Njc2IDI5Ljk5MTEgODEuODczOEMyOC40MTM4IDgyLjM3MjkgMjYuNjM4NyA4Mi41NTYyIDI0LjQ5NTYgODIuNzgxOUMyMy40ODk0IDgyLjg2NiAyMi42MjEzIDgyLjgxNTggMjEuNTU0NiA4My4wMTg4QzIxLjMxOTggODMuMDYzNSAyMC45OTI3IDgzLjE0OTEgMjAuNzM1OCA4My4yMDk3QzIwLjcyNjkgODMuMjExNiAyMC43MTg2IDgzLjIxNDIgMjAuNzA5NiA4My4yMTYyQzIwLjY5NTYgODMuMjE5NSAyMC42NzcyIDgzLjIyNjMgMjAuNjYzOCA4My4yMjk0QzE4Ljg1NyA4My42NjggMTcuNjk2MyA4NS4zMzY1IDE4LjA2OTkgODYuOTgwNUMxOC40NDM3IDg4LjYyNDggMjAuMjA4NiA4OS42MjQ4IDIyLjAyNjIgODkuMjMxMkMyMi4wMzkzIDg5LjIyODIgMjIuMDU4NCA4OS4yMjc3IDIyLjA3MiA4OS4yMjQ2QzIyLjA5MjYgODkuMjE5OSAyMi4xMTA2IDg5LjIwOTkgMjIuMTMxIDg5LjIwNDlDMjIuMzg0NCA4OS4xNDkgMjIuNzAxOSA4OS4wODY4IDIyLjkyMzYgODkuMDI3MkMyMy45NzIzIDg4Ljc0NTEgMjQuNzMxOCA4OC4zMzA2IDI1LjY3NDYgODcuOTY3N0MyNy43MDI5IDg3LjIzNjggMjkuMzgyOCA4Ni42MjYyIDMxLjAxOTUgODYuMzg4M0MzMS43MDMgODYuMzM0NSAzMi40MjMyIDg2LjgxMiAzMi43ODE0IDg3LjAxMzRMMzQuMTE3NyA4Ni43ODMxQzM3LjE5MjYgOTYuMzYxMyA0My42MzY2IDEwNC4xMDMgNTEuNzk2MyAxMDguOTYxTDUxLjIzOTYgMTEwLjMwM0M1MS40NDAzIDExMC44MjQgNTEuNjYxNiAxMTEuNTMgNTEuNTEyMSAxMTIuMDQ1QzUwLjkxNzEgMTEzLjU5NSA0OS44OTggMTE1LjIzMSA0OC43Mzc0IDExNy4wNTVDNDguMTc1NSAxMTcuODk4IDQ3LjYwMDQgMTE4LjU1MiA0Ny4wOTM0IDExOS41MTZDNDYuOTcyIDExOS43NDcgNDYuODE3NSAxMjAuMTAyIDQ2LjcwMDQgMTIwLjM0NkM0NS45MTI1IDEyMi4wMzkgNDYuNDkwNCAxMjMuOTkgNDguMDAzOCAxMjQuNzIyQzQ5LjUyNjggMTI1LjQ1OSA1MS40MTcxIDEyNC42ODIgNTIuMjM1MiAxMjIuOTg1QzUyLjIzNjQgMTIyLjk4MiA1Mi4yNDA2IDEyMi45OCA1Mi4yNDE3IDEyMi45NzhDNTIuMjQyNiAxMjIuOTc2IDUyLjI0MDkgMTIyLjk3MyA1Mi4yNDE3IDEyMi45NzFDNTIuMzU4MiAxMjIuNzMxIDUyLjUyMzMgMTIyLjQxNSA1Mi42MjE2IDEyMi4xODhDNTMuMDU2IDEyMS4xODkgNTMuMjAwNSAxMjAuMzMyIDUzLjUwNTkgMTE5LjM2NUM1NC4zMTcgMTE3LjMxOCA1NC43NjI2IDExNS4xNyA1NS44NzkxIDExMy44MzJDNTYuMTg0OSAxMTMuNDY2IDU2LjY4MzMgMTEzLjMyNSA1Ny4yMDAxIDExMy4xODZMNTcuODk0NCAxMTEuOTIyQzY1LjAwOCAxMTQuNjY1IDcyLjk3MDUgMTE1LjQwMiA4MC45MjQ1IDExMy41ODdDODIuNzM5MSAxMTMuMTczIDg0LjQ5MDggMTEyLjYzNyA4Ni4xODQzIDExMS45OTRDODYuMzc5NCAxMTIuMzQyIDg2Ljc0MiAxMTMuMDExIDg2LjgzOTMgMTEzLjE3OUM4Ny4zNjQ0IDExMy4zNTEgODcuOTM3NyAxMTMuNDM5IDg4LjQwNDcgMTE0LjEzM0M4OS4yNDAxIDExNS41NjcgODkuODExNCAxMTcuMjYzIDkwLjUwNzMgMTE5LjMxMkM5MC44MTI4IDEyMC4yNzkgOTAuOTYzOCAxMjEuMTM2IDkxLjM5ODEgMTIyLjEzNkM5MS40OTcxIDEyMi4zNjMgOTEuNjYxNCAxMjIuNjg0IDkxLjc3OCAxMjIuOTI1QzkyLjU5NDQgMTI0LjYyOCA5NC40OTA3IDEyNS40MDcgOTYuMDE1OSAxMjQuNjY5Qzk3LjUyOTIgMTIzLjkzNyA5OC4xMDc3IDEyMS45ODYgOTcuMzE5NCAxMjAuMjkzQzk3LjIwMjMgMTIwLjA0OSA5Ny4wNDEyIDExOS42OTUgOTYuOTE5OCAxMTkuNDY0Qzk2LjQxMjcgMTE4LjQ5OSA5NS44Mzc3IDExNy44NTIgOTUuMjc1OCAxMTcuMDA5Qzk0LjExNTIgMTE1LjE4NSA5My4xNTI2IDExMy42NyA5Mi41NTc1IDExMi4xMkM5Mi4zMDg3IDExMS4zMiA5Mi41OTk1IDExMC44MjMgOTIuNzkzMyAxMTAuMzAzQzkyLjY3NzIgMTEwLjE3IDkyLjQyODggMTA5LjQxNCA5Mi4yODI0IDEwOS4wNTlDMTAwLjc2MiAxMDQuMDI5IDEwNy4wMTcgOTUuOTk4NSAxMDkuOTU1IDg2LjcyMzlDMTEwLjM1MSA4Ni43ODY1IDExMS4wNDEgODYuOTA5MSAxMTEuMjY1IDg2Ljk1NDJDMTExLjcyNiA4Ni42NDg3IDExMi4xNDkgODYuMjUwMSAxMTIuOTgxIDg2LjMxNTlDMTE0LjYxNyA4Ni41NTM3IDExNi4yOTcgODcuMTY0NSAxMTguMzI2IDg3Ljg5NTNDMTE5LjI2OCA4OC4yNTgxIDEyMC4wMjggODguNjc5MyAxMjEuMDc3IDg4Ljk2MTRDMTIxLjI5OCA4OS4wMjEgMTIxLjYxNiA4OS4wNzY2IDEyMS44NjkgODkuMTMyNUMxMjEuODg5IDg5LjEzNzUgMTIxLjkwOCA4OS4xNDc1IDEyMS45MjggODkuMTUyMkMxMjEuOTQyIDg5LjE1NTMgMTIxLjk2MSA4OS4xNTU4IDEyMS45NzQgODkuMTU4OEMxMjMuNzkyIDg5LjU1MiAxMjUuNTU3IDg4LjU1MjYgMTI1LjkzIDg2LjkwODFDMTI2LjMwMyA4NS4yNjQxIDEyNS4xNDMgODMuNTk1MLine truncated
|
||||
keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "storageClass"], ["spec", "version"], ["spec", "host"], ["spec", "substrate"], ["spec", "proxmox"], ["spec", "proxmox", "allowedNodes"], ["spec", "proxmox", "dnsServers"], ["spec", "proxmox", "insecure"], ["spec", "proxmox", "ccm"], ["spec", "proxmox", "ccm", "credentialsSecretName"], ["spec", "proxmox", "csi"], ["spec", "proxmox", "csi", "enabled"], ["spec", "proxmox", "csi", "credentialsSecretName"], ["spec", "proxmox", "csi", "storageClasses"], ["spec", "proxmox", "ipv4Config"], ["spec", "proxmox", "ipv4Config", "addresses"], ["spec", "proxmox", "ipv4Config", "gateway"], ["spec", "proxmox", "ipv4Config", "prefix"], ["spec", "addons"], ["spec", "addons", "certManager"], ["spec", "addons", "certManager", "enabled"], ["spec", "addons", "certManager", "valuesOverride"], ["spec", "addons", "cilium"], ["spec", "addons", "cilium", "valuesOverride"], ["spec", "addons", "gatewayAPI"], ["spec", "addons", "gatewayAPI", "enabled"], ["spec", "addons", "ingressNginx"], ["spec", "addons", "ingressNginx", "enabled"], ["spec", "addons", "ingressNginx", "exposeMethod"], ["spec", "addons", "ingressNginx", "hosts"], ["spec", "addons", "ingressNginx", "valuesOverride"], ["spec", "addons", "gpuOperator"], ["spec", "addons", "gpuOperator", "enabled"], ["spec", "addons", "gpuOperator", "valuesOverride"], ["spec", "addons", "hami"], ["spec", "addons", "hami", "enabled"], ["spec", "addons", "hami", "valuesOverride"], ["spec", "addons", "monitoringAgents"], ["spec", "addons", "monitoringAgents", "enabled"], ["spec", "addons", "monitoringAgents", "valuesOverride"], ["spec", "addons", "verticalPodAutoscaler"], ["spec", "addons", "verticalPodAutoscaler", "valuesOverride"], ["spec", "addons", "velero"], ["spec", "addons", "velero", "enabled"], ["spec", "addons", "velero", "valuesOverride"], ["spec", "addons", "coredns"], ["spec", "addons", "coredns", "valuesOverride"], ["spec", "addons", "ouroboros"], ["spec", "addons", "ouroboros", "enabled"], ["spec", "addons", "ouroboros", "valuesOverride"], ["spec", "controlPlane"], ["spec", "controlPlane", "replicas"], ["spec", "controlPlane", "apiServer"], ["spec", "controlPlane", "apiServer", "resources"], ["spec", "controlPlane", "apiServer", "resourcesPreset"], ["spec", "controlPlane", "apiServer", "extraArgs"], ["spec", "controlPlane", "apiServer", "admissionControllers"], ["spec", "controlPlane", "apiServer", "extraVolumes"], ["spec", "controlPlane", "apiServer", "extraVolumeMounts"], ["spec", "controlPlane", "controllerManager"], ["spec", "controlPlane", "controllerManager", "resources"], ["spec", "controlPlane", "controllerManager", "resourcesPreset"], ["spec", "controlPlane", "scheduler"], ["spec", "controlPlane", "scheduler", "resources"], ["spec", "controlPlane", "scheduler", "resourcesPreset"], ["spec", "controlPlane", "konnectivity"], ["spec", "controlPlane", "konnectivity", "server"], ["spec", "controlPlane", "konnectivity", "server", "resources"], ["spec", "controlPlane", "konnectivity", "server", "resourcesPreset"], ["spec", "controlPlane", "kubelet"], ["spec", "controlPlane", "kubelet", "preferredAddressTypes"], ["spec", "talos"], ["spec", "talos", "version"], ["spec", "talos", "schematicID"], ["spec", "talos", "imageFactoryURL"], ["spec", "talos", "installerRepository"], ["spec", "talos", "registryMirrors"], ["spec", "oidc"], ["spec", "oidc", "mode"], ["spec", "oidc", "customConfig"], ["spec", "oidc", "customConfig", "config"], ["spec", "oidc", "customConfig", "secretRef"], ["spec", "oidc", "customConfig", "secretRef", "name"], ["spec", "oidc", "users"]]
|
||||
keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "storageClass"], ["spec", "version"], ["spec", "host"], ["spec", "substrate"], ["spec", "proxmox"], ["spec", "proxmox", "allowedNodes"], ["spec", "proxmox", "dnsServers"], ["spec", "proxmox", "insecure"], ["spec", "proxmox", "ccm"], ["spec", "proxmox", "ccm", "credentialsSecretName"], ["spec", "proxmox", "csi"], ["spec", "proxmox", "csi", "enabled"], ["spec", "proxmox", "csi", "credentialsSecretName"], ["spec", "proxmox", "csi", "storageClasses"], ["spec", "proxmox", "ipv4Config"], ["spec", "proxmox", "ipv4Config", "addresses"], ["spec", "proxmox", "ipv4Config", "gateway"], ["spec", "proxmox", "ipv4Config", "prefix"], ["spec", "addons"], ["spec", "addons", "certManager"], ["spec", "addons", "certManager", "enabled"], ["spec", "addons", "certManager", "valuesOverride"], ["spec", "addons", "cilium"], ["spec", "addons", "cilium", "valuesOverride"], ["spec", "addons", "gatewayAPI"], ["spec", "addons", "gatewayAPI", "enabled"], ["spec", "addons", "ingressNginx"], ["spec", "addons", "ingressNginx", "enabled"], ["spec", "addons", "ingressNginx", "exposeMethod"], ["spec", "addons", "ingressNginx", "hosts"], ["spec", "addons", "ingressNginx", "valuesOverride"], ["spec", "addons", "gpuOperator"], ["spec", "addons", "gpuOperator", "enabled"], ["spec", "addons", "gpuOperator", "valuesOverride"], ["spec", "addons", "hami"], ["spec", "addons", "hami", "enabled"], ["spec", "addons", "hami", "valuesOverride"], ["spec", "addons", "monitoringAgents"], ["spec", "addons", "monitoringAgents", "enabled"], ["spec", "addons", "monitoringAgents", "valuesOverride"], ["spec", "addons", "verticalPodAutoscaler"], ["spec", "addons", "verticalPodAutoscaler", "valuesOverride"], ["spec", "addons", "velero"], ["spec", "addons", "velero", "enabled"], ["spec", "addons", "velero", "valuesOverride"], ["spec", "addons", "coredns"], ["spec", "addons", "coredns", "valuesOverride"], ["spec", "addons", "ouroboros"], ["spec", "addons", "ouroboros", "enabled"], ["spec", "addons", "ouroboros", "valuesOverride"], ["spec", "controlPlane"], ["spec", "controlPlane", "replicas"], ["spec", "controlPlane", "apiServer"], ["spec", "controlPlane", "apiServer", "resources"], ["spec", "controlPlane", "apiServer", "resourcesPreset"], ["spec", "controlPlane", "apiServer", "extraArgs"], ["spec", "controlPlane", "apiServer", "admissionControllers"], ["spec", "controlPlane", "apiServer", "extraVolumes"], ["spec", "controlPlane", "apiServer", "extraVolumeMounts"], ["spec", "controlPlane", "controllerManager"], ["spec", "controlPlane", "controllerManager", "resources"], ["spec", "controlPlane", "controllerManager", "resourcesPreset"], ["spec", "controlPlane", "scheduler"], ["spec", "controlPlane", "scheduler", "resources"], ["spec", "controlPlane", "scheduler", "resourcesPreset"], ["spec", "controlPlane", "konnectivity"], ["spec", "controlPlane", "konnectivity", "server"], ["spec", "controlPlane", "konnectivity", "server", "resources"], ["spec", "controlPlane", "konnectivity", "server", "resourcesPreset"], ["spec", "controlPlane", "kubelet"], ["spec", "controlPlane", "kubelet", "preferredAddressTypes"], ["spec", "oidc"], ["spec", "oidc", "mode"], ["spec", "oidc", "customConfig"], ["spec", "oidc", "customConfig", "config"], ["spec", "oidc", "customConfig", "secretRef"], ["spec", "oidc", "customConfig", "secretRef", "name"], ["spec", "oidc", "users"]]
|
||||
secrets:
|
||||
exclude: []
|
||||
include:
|
||||
|
||||
@@ -2155,7 +2155,7 @@ const kubernetesKind = "Kubernetes"
|
||||
// upgraded cluster keeps them in its values until re-edited, and the adoption
|
||||
// migration deliberately leaves them in place rather than rewriting parent
|
||||
// values mid-upgrade -- but they no longer render anything on the Kubernetes CR.
|
||||
var removedKubernetesFields = []string{"nodeGroups", "nodeHealthCheck", "maxNodeProvisionTime"}
|
||||
var removedKubernetesFields = []string{"nodeGroups", "nodeHealthCheck", "maxNodeProvisionTime", "talos"}
|
||||
|
||||
// warnRemovedKubernetesFields emits a client-facing admission warning for each
|
||||
// Phase 2-removed field still present in a Kubernetes CR's values, so an
|
||||
|
||||
@@ -36,7 +36,7 @@ func TestNameCapComesFromTheShippedSchema(t *testing.T) {
|
||||
cap int
|
||||
explains string
|
||||
}{
|
||||
{"Kubernetes", "kubernetes-", "../../../../packages/system/kubernetes-rd/cozyrds/kubernetes.yaml", 32, "md0"},
|
||||
{"Kubernetes", "kubernetes-", "../../../../packages/system/kubernetes-rd/cozyrds/kubernetes.yaml", 32, "kubernetes-nodes-<name>-<pool>"},
|
||||
{"Kafka", "kafka-", "../../../../packages/system/kafka-rd/cozyrds/kafka.yaml", 44, "KRaft"},
|
||||
} {
|
||||
t.Run(tc.kind, func(t *testing.T) {
|
||||
|
||||
@@ -45,14 +45,14 @@ var nullChartCases = []nullChartCase{
|
||||
{name: "nullable schema default", schema: `{"type":"object","required":["name"],"properties":{"name":{"type":"string","nullable":true,"default":"example"}}}`, values: `{"name":"example"}`, input: `{"name":null}`},
|
||||
{name: "item schema defaults", schema: `{"type":"object","properties":{"entries":{"type":"object","additionalProperties":{"type":"object","required":["name"],"properties":{"name":{"type":"string","default":"example"}}}},"items":{"type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"type":"string","default":"example"}}}}}}`, values: `{"entries":{},"items":[{"name":"chart-default"}]}`, input: `{"entries":{"mine":{"name":null}},"items":[{"name":null}]}`},
|
||||
|
||||
{name: "nested defaults", chart: "apps/kubernetes", rd: "kubernetes", input: `{"talos":{"registryMirrors":null,"version":null},"controlPlane":{"replicas":null,"apiServer":{"resourcesPreset":null}},"nodeGroups":{"legacy":{"instanceType":null}}}`, want: `{"talos":{},"controlPlane":{"apiServer":{}},"nodeGroups":{"legacy":{"instanceType":null}}}`, paths: []string{"controlPlane.apiServer.resourcesPreset", "controlPlane.replicas", "talos.registryMirrors", "talos.version"}},
|
||||
{name: "nested defaults", chart: "apps/kubernetes", rd: "kubernetes", input: `{"oidc":{"mode":null},"addons":{"cilium":{"valuesOverride":null}},"controlPlane":{"replicas":null,"apiServer":{"resourcesPreset":null}},"nodeGroups":{"legacy":{"instanceType":null}}}`, want: `{"oidc":{},"addons":{"cilium":{}},"controlPlane":{"apiServer":{}},"nodeGroups":{"legacy":{"instanceType":null}}}`, paths: []string{"addons.cilium.valuesOverride", "controlPlane.apiServer.resourcesPreset", "controlPlane.replicas", "oidc.mode"}},
|
||||
{name: "mariadb users", chart: "apps/mariadb", rd: "mariadb", input: `{"users":{"alice":{"password":null,"maxUserConnections":null},"blank":null}}`},
|
||||
{name: "rabbitmq vhosts", chart: "apps/rabbitmq", rd: "rabbitmq", input: `{"vhosts":{"vh1":{"roles":null},"blank":null}}`},
|
||||
{name: "seaweedfs pools", chart: "extra/seaweedfs", rd: "seaweedfs", input: `{"volume":{"pools":{"ssd":{"diskType":null},"blank":null}}}`},
|
||||
{name: "vm source http", chart: "apps/vm-disk", rd: "vm-disk", input: `{"source":{"http":{"url":null}}}`},
|
||||
{name: "vm source disk", chart: "apps/vm-disk", rd: "vm-disk", input: `{"source":{"disk":{"name":null}}}`},
|
||||
{name: "vm source image", chart: "apps/vm-disk", rd: "vm-disk", input: `{"source":{"image":{"name":null}}}`},
|
||||
{name: "optional section", chart: "apps/kubernetes", rd: "kubernetes", input: `{"talos":null,"storageClass":null}`},
|
||||
{name: "optional section", chart: "apps/kubernetes", rd: "kubernetes", input: `{"oidc":null,"storageClass":null}`},
|
||||
}
|
||||
|
||||
func chartSpecSchema(t *testing.T, rd string) *structuralschema.Structural {
|
||||
|
||||
@@ -64,15 +64,16 @@ func normalize(t *testing.T, s *structuralschema.Structural, spec string) ([]str
|
||||
func TestNormalizeSpecNulls_RequiredFields(t *testing.T) {
|
||||
s := kubernetesSpecSchema(t)
|
||||
|
||||
paths, got := normalize(t, s, `{"host":"k8s","talos":{"registryMirrors":null,"version":null,"imageFactoryURL":"https://factory.example.invalid"}}`)
|
||||
paths, got := normalize(t, s, `{"host":"k8s","addons":{"cilium":{"valuesOverride":null}},"oidc":{"mode":null}}`)
|
||||
|
||||
want := []string{"talos.registryMirrors", "talos.version"}
|
||||
want := []string{"addons.cilium.valuesOverride", "oidc.mode"}
|
||||
if !reflect.DeepEqual(paths, want) {
|
||||
t.Errorf("stripped paths = %v, want %v", paths, want)
|
||||
}
|
||||
wantSpec := map[string]any{
|
||||
"host": "k8s",
|
||||
"talos": map[string]any{"imageFactoryURL": "https://factory.example.invalid"},
|
||||
"host": "k8s",
|
||||
"addons": map[string]any{"cilium": map[string]any{}},
|
||||
"oidc": map[string]any{},
|
||||
}
|
||||
if !reflect.DeepEqual(got, wantSpec) {
|
||||
t.Errorf("normalized spec = %v, want %v", got, wantSpec)
|
||||
@@ -82,12 +83,12 @@ func TestNormalizeSpecNulls_RequiredFields(t *testing.T) {
|
||||
func TestNormalizeSpecNulls_OptionalFieldsUntouched(t *testing.T) {
|
||||
s := kubernetesSpecSchema(t)
|
||||
|
||||
paths, got := normalize(t, s, `{"talos":null,"storageClass":null,"nodeGroups":null,"version":null}`)
|
||||
paths, got := normalize(t, s, `{"oidc":null,"storageClass":null,"nodeGroups":null,"version":null}`)
|
||||
|
||||
if len(paths) != 0 {
|
||||
t.Errorf("stripped paths = %v, want none: the Kubernetes schema requires none of these at the root", paths)
|
||||
}
|
||||
for _, key := range []string{"talos", "storageClass", "nodeGroups", "version"} {
|
||||
for _, key := range []string{"oidc", "storageClass", "nodeGroups", "version"} {
|
||||
v, ok := got[key]
|
||||
if !ok || v != nil {
|
||||
t.Errorf("%s = %v (present=%v), want a preserved null", key, v, ok)
|
||||
@@ -96,20 +97,20 @@ func TestNormalizeSpecNulls_OptionalFieldsUntouched(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestNormalizeSpecNulls_FreeFormSubtreeUntouched pins the boundary: inside
|
||||
// registryMirrors the keys are registry hostnames the user invents, and the
|
||||
// addons.cilium.valuesOverride the keys are Helm values the user invents, and the
|
||||
// schema declares nothing there. A null in that position may be the payload, so
|
||||
// it is left exactly as written rather than guessed at.
|
||||
func TestNormalizeSpecNulls_FreeFormSubtreeUntouched(t *testing.T) {
|
||||
s := kubernetesSpecSchema(t)
|
||||
|
||||
paths, got := normalize(t, s, `{"talos":{"registryMirrors":{"ghcr.io":null}}}`)
|
||||
paths, got := normalize(t, s, `{"addons":{"cilium":{"valuesOverride":{"ghcr.io":null}}}}`)
|
||||
|
||||
if len(paths) != 0 {
|
||||
t.Errorf("stripped paths = %v, want none", paths)
|
||||
}
|
||||
mirrors := got["talos"].(map[string]any)["registryMirrors"].(map[string]any)
|
||||
if _, ok := mirrors["ghcr.io"]; !ok {
|
||||
t.Errorf("free-form key ghcr.io was dropped: %v", mirrors)
|
||||
overrides := got["addons"].(map[string]any)["cilium"].(map[string]any)["valuesOverride"].(map[string]any)
|
||||
if _, ok := overrides["ghcr.io"]; !ok {
|
||||
t.Errorf("free-form key ghcr.io was dropped: %v", overrides)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -143,12 +144,12 @@ func TestNormalizeSpecNulls_NoDefaultOrUserCollection(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestNormalizeSpecNulls_PreservesOtherValuesExactly(t *testing.T) {
|
||||
app := &appsv1alpha1.Application{Spec: &apiextv1.JSON{Raw: []byte(`{"talos":{"registryMirrors":null},"large":9007199254740993,"negative":-9007199254740993,"text":"null","bool":false,"empty":[],"object":{}}`)}}
|
||||
app := &appsv1alpha1.Application{Spec: &apiextv1.JSON{Raw: []byte(`{"addons":{"cilium":{"valuesOverride":null}},"large":9007199254740993,"negative":-9007199254740993,"text":"null","bool":false,"empty":[],"object":{}}`)}}
|
||||
r := &REST{specSchema: kubernetesSpecSchema(t)}
|
||||
if _, err := r.normalizeSpecNulls(app); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(app.Spec.Raw) != `{"bool":false,"empty":[],"large":9007199254740993,"negative":-9007199254740993,"object":{},"talos":{},"text":"null"}` {
|
||||
if string(app.Spec.Raw) != `{"addons":{"cilium":{}},"bool":false,"empty":[],"large":9007199254740993,"negative":-9007199254740993,"object":{},"text":"null"}` {
|
||||
t.Fatalf("unrelated values changed: %s", app.Spec.Raw)
|
||||
}
|
||||
}
|
||||
@@ -182,8 +183,8 @@ func TestNormalizeSpecNulls_NoSchemaOrNoNulls(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestEmptyFieldsWarning(t *testing.T) {
|
||||
short := emptyFieldsWarning([]string{"talos.version", "talos.registryMirrors"})
|
||||
if !strings.Contains(short, "talos.version, talos.registryMirrors") ||
|
||||
short := emptyFieldsWarning([]string{"oidc.mode", "addons.cilium.valuesOverride"})
|
||||
if !strings.Contains(short, "oidc.mode, addons.cilium.valuesOverride") ||
|
||||
!strings.Contains(short, "the chart default applies") {
|
||||
t.Errorf("unexpected warning: %q", short)
|
||||
}
|
||||
|
||||
@@ -466,6 +466,12 @@ func TestWarnRemovedKubernetesFields(t *testing.T) {
|
||||
specJSON: `{"nodeGroups":{},"nodeHealthCheck":{"maxUnhealthy":"50%"},"maxNodeProvisionTime":"10m"}`,
|
||||
wantKeys: []string{"nodeGroups", "nodeHealthCheck", "maxNodeProvisionTime"},
|
||||
},
|
||||
{
|
||||
name: "kubernetes with a stored talos block warns",
|
||||
kindName: "Kubernetes",
|
||||
specJSON: `{"version":"v1.35","talos":{"version":"v1.13.6"}}`,
|
||||
wantKeys: []string{"talos"},
|
||||
},
|
||||
{
|
||||
name: "kubernetes without removed fields does not warn",
|
||||
kindName: "Kubernetes",
|
||||
|
||||
Reference in new issue
Block a user