mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 02:27:01 +09:00
Aleksei Sviridkin
9352d3ee0e
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.
The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.
- driver.enabled=false use the host driver
- toolkit.enabled=false host already has nvidia-container-toolkit;
enabling overwrites the containerd
config.toml the host configured via
nvidia-ctk runtime configure
- devicePlugin.enabled=true publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false do not unbind the host driver
- cdi.enabled=false CDI spec generation is serviced by the
disabled toolkit; an apt host uses the
legacy nvidia-container-runtime hook, so
pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false regular pod workloads, not VMs
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
54 lines
2.7 KiB
YAML
54 lines
2.7 KiB
YAML
# Container variant: GPU workloads run in regular pods on hosts that
|
|
# already have the NVIDIA driver and nvidia-container-toolkit installed
|
|
# (typical Ubuntu/Debian shape: apt-installed driver + apt-installed
|
|
# nvidia-container-toolkit). The third architectural mode that upstream
|
|
# NVIDIA gpu-operator supports — sandboxWorkloads disabled,
|
|
# devicePlugin enabled — surfaced here as a Cozystack variant.
|
|
gpu-operator:
|
|
sandboxWorkloads:
|
|
# Default upstream workload on a node without the per-node
|
|
# 'nvidia.com/gpu.workload.config' label is 'container' when
|
|
# sandboxWorkloads is off — which is exactly what this variant
|
|
# wants. No defaultWorkload override needed; the counterpart
|
|
# overrides in values-passthrough.yaml / values-vgpu.yaml exist
|
|
# only because those variants flip sandboxWorkloads to true.
|
|
enabled: false
|
|
driver:
|
|
# Host already runs the NVIDIA driver (apt-installed). The
|
|
# operator's driver DaemonSet would attempt to load its own
|
|
# kernel module and clash with the host one.
|
|
enabled: false
|
|
devicePlugin:
|
|
# Publish nvidia.com/gpu to the kubelet so pods can request GPUs.
|
|
enabled: true
|
|
vfioManager:
|
|
# vfioManager unbinds the host driver to bind vfio-pci, which must
|
|
# not happen when the host driver is in use and devicePlugin is on.
|
|
# Upstream renders vfioManager.enabled=true into the ClusterPolicy
|
|
# unconditionally; it only stays at zero pods because the DaemonSet
|
|
# schedules solely onto nodes labeled
|
|
# nvidia.com/gpu.workload.config=vm-passthrough, and no node carries
|
|
# a sandbox assignment when sandboxWorkloads is off. Pin it off so
|
|
# the rendered policy matches intent regardless of upstream
|
|
# scheduling changes.
|
|
enabled: false
|
|
toolkit:
|
|
# Host already has nvidia-container-toolkit installed (the typical
|
|
# apt-installed shape). Running the operator's toolkit DaemonSet
|
|
# would overwrite /etc/containerd/config.toml that the host package
|
|
# configured (via `nvidia-ctk runtime configure`), breaking the
|
|
# host runtime wiring. NVIDIA's documentation recommends disabling
|
|
# this when the toolkit is preinstalled on the node.
|
|
enabled: false
|
|
cdi:
|
|
# Upstream defaults cdi.enabled=true, but CDI spec generation and
|
|
# runtime wiring are serviced by the operator-managed toolkit, which
|
|
# this variant disables. An apt-installed nvidia-container-toolkit
|
|
# does not generate CDI specs on its own (`nvidia-ctk cdi generate`
|
|
# is a manual step) and uses the classic nvidia-container-runtime
|
|
# hook path. Leaving CDI on would point the device plugin at a
|
|
# CDI-injection mode with no specs on the host and silently break
|
|
# GPU allocation. Pin off so device allocation matches the host
|
|
# runtime wiring.
|
|
enabled: false
|