mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-29 16:17:20 +09:00
Mattia Eleuteri
669fb2416a
Closes the gpu-operator half of #3563. Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA driver comes from a system extension baked into the worker image rather than from the operator's driver container. The addon default still described the Ubuntu shape, so a GPU node group came up with the operator fighting the OS for the driver and no path at all for CUDA libraries to reach a tenant pod. Three things change, each fixing one silent failure: driver.enabled and toolkit.enabled go false. The driver container would load its own module and clash with the extension's; the toolkit container would rewrite /etc/containerd/config.toml, which Talos owns and which already registers the nvidia runtime with enable_cdi = true and cdi_spec_dirs = ['/run/cdi']. validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and disables /dev/char symlink creation. The driver validator otherwise looks for a driver container install, finds none, never writes its sentinel, and leaves the device plugin blocked behind toolkit-validation — the loop on "Attempting to validate a driver container installation". devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need one on Ubuntu, where the toolkit made nvidia the default containerd runtime; Talos does not, so CDI is the only route for the CUDA libraries. DEVICE_ID_STRATEGY, NVIDIA_DRIVER_ROOT and CDI_ANNOTATION_PREFIX are set and tested alongside it: CDI device names key on the UUID strategy, the plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation prefix has to be the one containerd reads, so a regression in any of them breaks device resolution on its own. The previous default fed NVreg_NvLinkDisable=1 to the driver container through the nvidia-kernel-module-params ConfigMap. With that container off the ConfigMap is mounted by nothing, so the default is removed. On Talos a module parameter belongs to the worker machine config, which the kernelModules field of the KubernetesNodes app (packages/apps/kubernetes-nodes, #3571) writes. The wrapper chart keeps the ConfigMap for an operator who re-enables the driver container. Every value here is the configuration validated on a production Talos GPU node group, minus the parts that only mattered while the driver container was in play. Not covered: HAMi. When addons.hami.enabled the operator's device plugin stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent CDI wiring on Talos that this default does not attempt. Refs: #3563 Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io>