Aleksei Sviridkin
f0f1431546
## What this PR does Makes GPU passthrough work out-of-the-box for tenant Kubernetes clusters (the `kubernetes` app), so operators no longer have to hand-fix it. Two defaults: 1. **Tenant gpu-operator loads the nvidia driver with `NVreg_NvLinkDisable=1`.** 2. **Nodes in GPU node-groups are labeled `gpu=on` automatically.** ### Root cause (validated empirically on a live cluster) cozystack passes through individual SXM GPUs (e.g. H200) into tenant worker VMs **without** the NVSwitches. Such a GPU's driver then waits for an NVLink fabric that can never come up — there is no NVSwitch and fabricmanager has nothing to manage — so `nvidia-smi` reports `Fabric State: In Progress` indefinitely. As a result every CUDA call fails with `system not yet initialized`, `nvidia-operator-validator` (cuda-validation) crashloops, and no GPUs are advertised. Loading the kernel module with `NVreg_NvLinkDisable=1` flips `Fabric State` to `N/A`, CUDA initializes, and the validators pass. This is the correct setting for per-GPU passthrough because there is no in-VM NVLink/NVSwitch peering — the GPU is alone in the VM, so there is nothing to disable that the workload needs. The second issue: HAMi's `hami-device-plugin` DaemonSet has `nodeSelector: gpu=on` and stays at `DESIRED=0` because GPU worker nodes lack that label, so `nvidia.com/gpu` is never advertised. Labeling the GPU nodes `gpu=on` lets the device plugin schedule and advertise the GPUs. ### How it works - **`gpu=on` label:** node-groups in `packages/apps/kubernetes/templates/cluster.yaml` that declare `gpus` get `--node-labels=gpu=on` appended to their kubelet args (in the `KubeadmConfigTemplate` `joinConfiguration`). Non-GPU node-groups are untouched. - **`NVreg_NvLinkDisable=1`:** the upstream gpu-operator chart only references an existing ConfigMap by name (`driver.kernelModuleConfig.name`) and never creates it. The gpu-operator wrapper chart (`packages/system/gpu-operator`) now renders that ConfigMap (opt-in via `kernelModuleConfig.create`) so it deploys into the tenant `cozy-gpu-operator` namespace alongside the operator. The tenant gpu-operator HelmRelease defaults enable it and point `driver.kernelModuleConfig.name` at it. Both defaults remain overridable: a user setting `addons.gpuOperator.valuesOverride` still wins via the existing `mergeOverwrite(defaults, overrides)` in the HelmRelease template. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note feat(kubernetes): enable GPU passthrough out-of-the-box for tenant clusters. GPU node-groups are now labeled `gpu=on` so HAMi's device plugin schedules and advertises `nvidia.com/gpu`, and the tenant gpu-operator loads the nvidia driver with `NVreg_NvLinkDisable=1` so single-SXM-GPU passthrough (no NVSwitch in the VM) no longer hangs on `Fabric State: In Progress` / `system not yet initialized`. Both defaults are overridable via `addons.gpuOperator.valuesOverride`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * GPU-enabled node groups now get the expected GPU label during node registration. * GPU operator supports optional kernel-module parameter configuration with a default parameter set. * **Tests** * Added tests verifying GPU node labeling is applied only for GPU-enabled node groups. * Added tests validating kernel-module config creation, naming, and content behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have Feature Requests please use the Discussion's Feature Request section.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.