Files
cozystack/packages/apps/kubernetes-nodes
Myasnikov Daniil bc8c54d6e6 feat(kubernetes): boot tenant worker disks from a shared golden Talos image
Every tenant worker VM imports the same Talos raw image from the Image Factory
over HTTP, once per worker: a 20-worker cluster fetches the identical ~4.15 GiB
artifact 20 times, and each fetch is a live dependency on the Factory at exactly
the moment a node joins.

packages/system/kubernetes-worker-image is a new opt-in catalog that holds one
golden DataVolume per (schematicID, version) in cozy-public, named
talos-worker-<schematicID>-<version>. A worker pool opts in with osImage.builtin
on its own KubernetesNodes resource, and its disk-system DataVolume then carries
source.pvc pointing at that golden, so CDI performs a storage-layer CSI clone
instead of an HTTP import -- one import per flavor, shared by every worker of
every tenant. Confirmed on a LINSTOR-backed cluster: the worker disk is a CSI
clone plus a resize, with no host-assisted upload pod.

Both halves are off unless asked for. The catalog is registered through the
platform's optional-package helper and disabled by default, so a cluster that
does not enable it creates no golden and keeps importing over HTTP. The disk
source is derived entirely from osImage, which is absent by default, so a pool
that does not set it renders byte-identical to the pre-feature chart and
adopting the feature never rolls existing workers. Editing osImage on an
existing pool re-images that pool, as changing any other image reference does.

What the render refuses, rather than degrading into a silent fallback:

- A pool whose golden has no catalog entry, or whose golden terminally failed to
  import. A golden that is merely mid-import is not an error; CDI holds the
  worker disk until the source populates.
- A pool whose StorageClass differs from its golden's. CDI cannot CSI-clone
  across classes and falls back to a host-assisted copy over the pod network,
  which is the transfer this feature exists to remove. An empty class on either
  side means the cluster default and is resolved before the comparison, and a
  pool that has already cloned reads its class from its own bound PVC rather
  than from today's default -- the golden's side is frozen the same way, and an
  admin moving storageclass.kubernetes.io/is-default-class must not fail the
  release of a pool whose disks are all on the right class. That resolved class
  is then written onto the worker disk, so the disk binds where the guard
  checked instead of on whatever the default has become by then -- reading one
  class and binding another is how a pool at an empty storageClass passed the
  comparison and still took the host-assisted copy. The HTTP path leaves an
  empty class to the cluster as before, which is what keeps its render, and the
  KubevirtMachineTemplate hash named after it, byte-identical.
- A diskSize below the golden's storage size, which a clone target cannot be.
- A schematicID that is not a plain lowercase alphanumeric identifier, or a
  version that is not a vMAJOR.MINOR.PATCH release. Both are interpolated into
  a DataVolume name and an image URL, and values.schema.json types them as bare
  strings, so a malformed one otherwise reaches the render. The catalog applies
  the same patterns, since both sides build the same name from the same fields.
- An images[] entry edited after its golden exists. A DataVolume spec cannot be
  patched in place, so the message names the entry and the supported move -- a
  new (schematicID, version) -- instead of leaving the API server to reject the
  patch on the next upgrade.

Goldens carry helm.sh/resource-policy: keep, so dropping an entry stops
managing a golden rather than deleting it: a missing golden aborts the whole
Kubernetes HelmRelease of every tenant pinned to it, not just the disk. Each
cloned worker disk annotates the golden it came from and the pool that owns it,
so "does anything still clone this?" is answerable before a manual delete
instead of after.

A builtin pool needs a StorageClass whose volumes are reachable from any node,
in practice a replicated/DRBD one. The golden in cozy-public has no consumer and
binds as soon as it imports, wherever CDI's importer landed, while a worker disk
binds where its VM is scheduled, and nothing here arranges for those to be the
same node. The platform ships cloneStrategyOverride: csi-clone, which rules out
the host-assisted strategy that would cross nodes. What a node-pinned class does
in that situation is not measured, so values.yaml states the requirement and not
a symptom.

ComputePlane pools forward only the per-pool shape fields, and osImage is
excluded there by name: the worker OS image, its Talos installer and its Image
Factory are operator decisions about the platform's own images.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 15:35:44 +03:00
..

Kubernetes Worker Node Pool

kubernetes-nodes manages a single worker node pool for a managed Kubernetes cluster. It is the worker half of the Kubernetes-app split (see cozystack/community#8): the control plane lives in the kubernetes chart, and each pool is a separate kubernetes-nodes release attached to a parent cluster by name.

A KubernetesNodes CR named <cluster>-<pool> (e.g. myk8s-md0) provisions a MachineDeployment + KubevirtMachineTemplate + MachineHealthCheck wired to the parent cluster kubernetes-<cluster> in the same namespace, plus a talos-reconcile Job that produces the pool's TalosConfigTemplate.

Parameters

Linkage

Name Description Type Value
cluster Name of the parent Kubernetes cluster (the Kubernetes CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster kubernetes-<cluster>; the KubernetesNodes CR must be named <cluster>-<pool>. The pool name (the part after <cluster>-) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default md0), or the render fails on ownership conflict. string ""

Common Parameters

Name Description Type Value
storageClass StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an osImage.builtin pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a builtin pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. string replicated

Pool Sizing

Name Description Type Value
minReplicas Minimum number of replicas in this pool. Used as the cluster-autoscaler floor. int 0
maxReplicas Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling. int 10
instanceType Virtual machine instance type. string u1.medium
diskSize System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's osImage: the default and osImage.factory stream the Image Factory raw artifact in over HTTP, while osImage.builtin makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A builtin pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a builtin pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus diskSize + golden storage for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. quantity 20Gi
roles List of node roles. Each role r labels nodes with node-role.kubernetes.io/<r>. Use [ingress-nginx] for a pool that hosts the tenant ingress-nginx controller. []string []
resources Explicit CPU and memory for each worker node, as an alternative to instanceType sizing. Optional: when omitted, the node is sized by instanceType. When both cpu and memory are set, they take precedence and instanceType is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both cpu and memory together or neither; setting only one is rejected at render time. object {}
resources.cpu CPU available. quantity ""
resources.memory Memory (RAM) available. quantity ""
podCpuLimit CPU limit for the worker VM's virt-launcher compute container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from resources.cpu, which sizes the guest itself. Cannot be combined with instanceType sizing: KubeVirt refuses a VM that references an instancetype and also sets domain.resources.limits.cpu, so the chart rejects that combination at render time. Sizing the pool by resources (both cpu and memory) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on Insufficient cpu. Pair it with podCpuRequest to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with podCpuRequest: 0, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. quantity ""
podCpuRequest CPU request for the worker VM's virt-launcher compute container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from resources.cpu, which sizes the guest itself. It exists to be paired with podCpuLimit: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with instanceType sizing, for the same reason as podCpuLimit: KubeVirt refuses a VM that references an instancetype and also sets domain.resources.requests.cpu, so the chart rejects that combination at render time, as it does a value above podCpuLimit. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a podCpuLimit is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. quantity ""
gpus List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). []object []
gpus[i].name Name of GPU, such as "nvidia.com/AD102GL_L40S". string ""
kubelet Kubelet resource reservations for this pool. object {}
kubelet.systemReservedMemory Memory reserved for host OS. Auto-computed from instanceType if empty. string ""
kubelet.kubeReservedMemory Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty. string ""
kubelet.systemReservedCpu CPU reserved for host OS. Auto-computed from instanceType if empty. string ""
kubelet.kubeReservedCpu CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. string ""
kubelet.evictionHardMemory Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). string 7%
kubelet.evictionSoftMemory Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). string 10%
logSerialConsole Stream each worker VM's guest serial console into a guest-console-log container beside virt-launcher, readable with kubectl logs. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide disableSerialConsoleLog, set because that container has been seen holding virt-launcher in PodInitializing (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it. bool false
maxUnhealthy Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At "50%" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two. string 50%
nodeStartupTimeout Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning. string 20m
readyUnknownTimeout How long a worker Node may report Ready=Unknown before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. Ready=Unknown is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart. string 5m
readyFalseTimeout How long a worker Node may report Ready=False before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports Ready=False, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart. string 10m
maxNodeProvisionTime How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted. Keep in sync with the parent kubernetes chart. string 30m

Kubernetes Version

Name Description Type Value
version Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against talos.version. string v1.35

Talos Worker Image

Name Description Type Value
osImage Worker OS image source for this pool. Default (omitted): import over HTTP from the talos.* Image Factory. Set osImage.builtin to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool. object {}
osImage.builtin Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled. *object null
osImage.builtin.schematicID Talos schematic ID of the golden to clone. Defaults to talos.schematicID. string ""
osImage.builtin.version Talos version of the golden to clone. Defaults to talos.version. string ""
osImage.factory Import from an Image Factory / mirror (default). *object null
osImage.factory.version Talos version. Defaults to talos.version. string ""
osImage.factory.schematicID Talos schematic ID. Defaults to talos.schematicID. string ""
osImage.factory.imageFactoryURL Base URL of the Image Factory serving the boot raw image. Defaults to talos.imageFactoryURL. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from talos.installerRepository, so an air-gapped pool must also have talos.installerRepository pointed at a reachable registry. string ""
talos Talos worker image configuration. Keep in sync with the parent cluster's talos. object {}
talos.version Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via osImage.builtin.version or osImage.factory.version; when osImage omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check talos.version would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full vMAJOR.MINOR.PATCH release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only v1.13 included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead. string v1.13.6
talos.schematicID Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via osImage.builtin.schematicID or osImage.factory.schematicID; when osImage omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on osImage.builtin also needs its own catalog entry, since the golden is named from the schematic ID. string ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515
talos.imageFactoryURL Default base URL of the Talos Image Factory that serves the worker OS disk image (the openstack-amd64.raw.xz raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and osImage.factory, which overrides it via osImage.factory.imageFactoryURL. A pool on osImage.builtin clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain http(s) URL and the render refuses anything else; a pool on osImage.builtin builds no such URL and is not held to it, so blanking this field there is legal. string https://factory.talos.dev
talos.installerRepository OCI repository prefix for the Talos installer image used by the in-guest talos-reconcile upgrade Job. Resolved as <installerRepository>/<schematicID>:<version> against whatever this pool's osImage pins, falling back to talos.schematicID and talos.version — so the installer matches the OS the pool actually booted. The prefix itself has no osImage.* override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. string factory.talos.dev/installer
talos.registryMirrors Talos machine.registries.mirrors passthrough for worker nodes: a map of upstream registry host to { endpoints: [ ... ] }. Empty by default, so workers pull container images (the Talos kubelet image included) directly from the upstream registry. Point a host such as ghcr.io at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets skipFallback: true, so a mirror alone does not enforce air-gap. Keep in sync with the parent kubernetes chart. object {}

Images

Name Description Type Value
images Optional image overrides for air-gapped or rate-limited registries. object {}
images.kubectl Image used by the talos-reconcile and pre-delete unpin Jobs (kubectl). Empty falls back to images/kubectl.tag. string ""