mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-04 02:57:27 +09:00
Aleksei Sviridkin
b6108039b7
## What this PR does Add a third architectural variant to `cozystack.gpu-operator` for hosts where the NVIDIA driver and `nvidia-container-toolkit` are already installed by the distro package manager (the typical Ubuntu/Debian GPU node shape: apt-installed driver + apt-installed CT). The existing variants do not produce a working setup in that shape. `default` (passthrough) runs `vfioManager`, which unbinds the host driver to bind `vfio-pci` — with the host driver active, `k8s-driver-manager` early-exits and the bind never happens, so the GPU never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host driver and a license server; the user-installed open or proprietary driver does not satisfy it. The new `container` variant maps onto upstream gpu-operator's third mode (`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with every component that would conflict with the host install switched off: - `driver.enabled=false` — use the host driver. - `toolkit.enabled=false` — host already has `nvidia-container-toolkit`. Enabling the operator's toolkit DaemonSet would overwrite the `/etc/containerd/config.toml` the host package configured (via `nvidia-ctk runtime configure`), breaking the host runtime wiring. Matches NVIDIA's documented guidance and the existing `examples/values-native-talos.yaml` reference, which pins the same flag for the same reason. - `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet. - `vfioManager.enabled=false` — do not unbind the host driver. Upstream renders this `true` unconditionally; it only stays at zero pods because no node carries a `vm-passthrough` sandbox assignment when `sandboxWorkloads` is off — pinned off so the rendered policy matches intent. - `cdi.enabled=false` — upstream defaults this `true`, but CDI spec generation is serviced by the operator-managed toolkit DaemonSet this variant disables. An apt host doesn't auto-generate CDI specs (`nvidia-ctk cdi generate` is a manual step) and uses the classic `nvidia-container-runtime` hook, so leaving CDI on would point the device plugin at CDI injection with no specs on the host and silently break allocation. - `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs. With this off, upstream `defaultWorkload` is already `container`, so no override is needed. Host prerequisite: because the operator's toolkit component is disabled, the node's containerd must already have the `nvidia` runtime registered (`nvidia-ctk runtime configure --runtime=containerd`); `apt install nvidia-container-toolkit` installs binaries only and does not wire containerd. With `driver.enabled=false` the operator uses the pre-installed host driver at its standard location, so a stock apt install needs no `hostPaths.driverInstallDir` override (Talos lands the driver under a non-standard prefix — see `examples/values-native-talos.yaml`). The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`) already enables `cozystack.gpu-operator` as an optional package — no bundle changes required. Users opt into this variant by setting `variant: container` in their `Package` CR. Verified locally with `helm template` against `values.yaml` + `values-container.yaml`: ClusterPolicy renders with `driver.enabled=false`, `toolkit.enabled=false`, `devicePlugin.enabled=true`, `vfioManager.enabled=false`, `cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm unittest` and the `gpu-operator` variant bats suite are green. Not yet smoke-tested on a real Ubuntu GPU node — the config matches the documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI off), but an end-to-end CUDA-pod run on hardware is still owed. Closes #2764 ### Release note ```release-note feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a "container" GPU operator variant to support containerized GPU workloads on hosts with the NVIDIA driver and nvidia-container-toolkit. * **Documentation** * Updated variants documentation and OS support table to include the container variant and per-host requirements (including Ubuntu and Talos notes). * Clarified Talos native pod example wording. * **Tests** * Added regression tests to verify container-variant configuration and component enablement/disablement. <!-- end of auto-generated comment: release notes by coderabbit.ai -->