mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-10 14:34:58 +09:00
Aleksei Sviridkin
32c4d60b43
The talos-reconcile Job is the only thing that creates the TalosConfigTemplate the worker MachineDeployment's bootstrap.configRef points to; CAPI blocks MachineSet creation until it exists, so tenant workers cannot scale until the Job has run. The Job was a post-install/post-upgrade Helm hook, which runs only after Helm has applied the release's main resources and, when Helm waits, after they become Ready. But the same release emits in-tenant addon HelmReleases — notably cilium — whose readiness needs worker nodes, and those nodes need the TalosConfigTemplate this Job produces. So when the main-resource wait is in effect the install deadlocks: it waits for cilium to be Ready, cilium waits for workers, the workers wait for the template, and the template waits for this hook — which never runs because the install never finishes the wait that precedes the hook phase. The install times out and the MachineDeployment never scales. The release sets helm-install-disable-wait to skip that wait, but its effect is unreliable, which makes the failure bimodal: the template lands in ~46s when the wait is skipped, and the install is stranded when it is not. Render the Job and its RBAC/ServiceAccount/egress policy as ordinary main-phase resources. The Job is then created during the apply itself, not after the wait, and runs on the management cluster where nodes exist (unlike the tenant cilium-operator), so it produces the template, the MachineDeployment scales, workers join, cilium goes Ready, and the wait (if any) closes within the install budget — independent of whether helm-install-disable-wait takes effect. The Job is immutable, so it carries a content-hash name suffix over its rendered spec (mirrors the KubevirtMachineTemplate idiom in cluster.yaml): an identical render keeps the name and Helm no-ops it, while any spec change yields a new name and a fresh Job, so no spec change ever lands on the same name and triggers an immutable-field patch error. The applied TalosConfigTemplate keeps its stable <release>-<group> name, so an unchanged render re-applies a byte-identical template — a CAPI no-op that never rolls existing workers. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
83 lines
2.3 KiB
YAML
83 lines
2.3 KiB
YAML
suite: GPU node-group Talos worker nodeLabels tests
|
|
|
|
# The TalosConfigTemplate that defines worker nodeLabels is not rendered
|
|
# directly by helm — the kubernetes chart emits a main-phase Job
|
|
# (templates/talos/talos-reconcile-job.yaml) whose container command
|
|
# carries the TCT YAML as a string literal that the Job applies at
|
|
# runtime. helm-unittest therefore asserts on the rendered Job's script
|
|
# via matchRegex / notMatchRegex, looking for the machine.nodeLabels
|
|
# block GPU node-groups must inject so HAMi's hami-device-plugin
|
|
# DaemonSet (nodeSelector gpu=on) schedules and advertises nvidia.com/gpu.
|
|
# Without the label the plugin stays at DESIRED=0 and no GPUs are
|
|
# exposed to the tenant — same regression main avoids via kubeadm
|
|
# kubeletExtraArgs node-labels="gpu=on".
|
|
|
|
templates:
|
|
- templates/talos/talos-reconcile-job.yaml
|
|
|
|
tests:
|
|
- it: labels GPU node-groups with gpu=on
|
|
release:
|
|
name: test-k8s
|
|
namespace: tenant-test
|
|
set:
|
|
_namespace:
|
|
etcd: etcd
|
|
ingress: nginx
|
|
host: example.com
|
|
_cluster:
|
|
cluster-domain: cozy.local
|
|
nodeGroups:
|
|
md0:
|
|
minReplicas: 0
|
|
maxReplicas: 10
|
|
instanceType: ""
|
|
diskSize: 20Gi
|
|
roles:
|
|
- worker
|
|
resources:
|
|
cpu: 8
|
|
memory: 16Gi
|
|
gpus:
|
|
- name: nvidia.com/GH200
|
|
kubelet: {}
|
|
asserts:
|
|
- matchRegex:
|
|
path: spec.template.spec.containers[0].command[2]
|
|
pattern: 'nodeLabels:\s*\n\s+gpu: "on"'
|
|
documentSelector:
|
|
path: kind
|
|
value: Job
|
|
|
|
- it: does not label non-GPU node-groups with gpu=on
|
|
release:
|
|
name: test-k8s
|
|
namespace: tenant-test
|
|
set:
|
|
_namespace:
|
|
etcd: etcd
|
|
ingress: nginx
|
|
host: example.com
|
|
_cluster:
|
|
cluster-domain: cozy.local
|
|
nodeGroups:
|
|
md0:
|
|
minReplicas: 0
|
|
maxReplicas: 10
|
|
instanceType: ""
|
|
diskSize: 20Gi
|
|
roles:
|
|
- worker
|
|
resources:
|
|
cpu: 2
|
|
memory: 4Gi
|
|
gpus: []
|
|
kubelet: {}
|
|
asserts:
|
|
- notMatchRegex:
|
|
path: spec.template.spec.containers[0].command[2]
|
|
pattern: 'nodeLabels:\s*\n\s+gpu: "on"'
|
|
documentSelector:
|
|
path: kind
|
|
value: Job
|