mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-30 08:37:31 +09:00
Mattia Eleuteri
669fb2416a
Closes the gpu-operator half of #3563. Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA driver comes from a system extension baked into the worker image rather than from the operator's driver container. The addon default still described the Ubuntu shape, so a GPU node group came up with the operator fighting the OS for the driver and no path at all for CUDA libraries to reach a tenant pod. Three things change, each fixing one silent failure: driver.enabled and toolkit.enabled go false. The driver container would load its own module and clash with the extension's; the toolkit container would rewrite /etc/containerd/config.toml, which Talos owns and which already registers the nvidia runtime with enable_cdi = true and cdi_spec_dirs = ['/run/cdi']. validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and disables /dev/char symlink creation. The driver validator otherwise looks for a driver container install, finds none, never writes its sentinel, and leaves the device plugin blocked behind toolkit-validation — the loop on "Attempting to validate a driver container installation". devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need one on Ubuntu, where the toolkit made nvidia the default containerd runtime; Talos does not, so CDI is the only route for the CUDA libraries. DEVICE_ID_STRATEGY, NVIDIA_DRIVER_ROOT and CDI_ANNOTATION_PREFIX are set and tested alongside it: CDI device names key on the UUID strategy, the plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation prefix has to be the one containerd reads, so a regression in any of them breaks device resolution on its own. The previous default fed NVreg_NvLinkDisable=1 to the driver container through the nvidia-kernel-module-params ConfigMap. With that container off the ConfigMap is mounted by nothing, so the default is removed. On Talos a module parameter belongs to the worker machine config, which the kernelModules field of the KubernetesNodes app (packages/apps/kubernetes-nodes, #3571) writes. The wrapper chart keeps the ConfigMap for an operator who re-enables the driver container. Every value here is the configuration validated on a production Talos GPU node group, minus the parts that only mattered while the driver container was in play. Not covered: HAMi. When addons.hami.enabled the operator's device plugin stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent CDI wiring on Talos that this default does not attempt. Refs: #3563 Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
47 lines
2.4 KiB
YAML
47 lines
2.4 KiB
YAML
# Defaults applied to every variant of the gpu-operator package.
|
|
# The variant-specific files (values-passthrough.yaml, values-vgpu.yaml,
|
|
# values-container.yaml) layer on top.
|
|
gpu-operator:
|
|
# Upstream chart v26.x flips ccManager.enabled to true and
|
|
# ccManager.defaultMode to "on". On Hopper-class hardware
|
|
# (H100/H200/B100) with Confidential Computing capability this would
|
|
# auto-enable CC mode after a chart upgrade — a non-trivial behavior
|
|
# change. On non-CC-capable GPUs (L40S, A100, T4, …) the DaemonSet
|
|
# deploys, probes the hardware, and no-ops, but it still enters the
|
|
# cluster's resource graph. Pin to false so the default footprint
|
|
# stays as it was on v25.x; users who want CC enable it explicitly.
|
|
ccManager:
|
|
enabled: false
|
|
# Upstream v26.x also flips vgpuDeviceManager.enabled to true. The
|
|
# primary reason this is pinned back to false: the device manager
|
|
# walks /sys/class/mdev_bus/, which does not exist on Ada Lovelace
|
|
# (L4/L40/L40S, …) or Blackwell (B100, …) on the vGPU 17/20 driver
|
|
# branch — it errors with «no parent devices found». Keeping the
|
|
# default-on would crashloop the DaemonSet on any modern GPU.
|
|
# Secondary consideration: in passthrough mode the operator does not
|
|
# apply the nvidia.com/gpu.deploy.vgpu-device-manager=true label, so
|
|
# the DaemonSet would stay at zero pods, but it would still render
|
|
# into the cluster. The vgpu variant explicitly re-affirms this in
|
|
# values-vgpu.yaml.
|
|
vgpuDeviceManager:
|
|
enabled: false
|
|
|
|
# Optional ConfigMap holding extra nvidia kernel module parameters.
|
|
# The upstream chart only references an existing ConfigMap by name via
|
|
# driver.kernelModuleConfig.name — it never creates it. When create is
|
|
# true this wrapper chart renders the ConfigMap into the release
|
|
# namespace so the params actually land. The driver entrypoint reads
|
|
# ${base}/nvidia.conf and appends each line to the nvidia modprobe
|
|
# options, so it only does anything when driver.enabled is true.
|
|
#
|
|
# Tenant Kubernetes clusters run this chart with the driver container off
|
|
# and take module parameters from the worker pool's kernelModules in the
|
|
# KubernetesNodes app (packages/apps/kubernetes-nodes). An operator who
|
|
# re-enables the driver container through valuesOverride also has to set
|
|
# create: true here.
|
|
kernelModuleConfig:
|
|
create: false
|
|
name: nvidia-kernel-module-params
|
|
content: |
|
|
NVreg_NvLinkDisable=1
|