mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-05 03:27:07 +09:00
Aleksei Sviridkin
1d747b3d8d
Closes #2765. ## What this PR does Makes the passthrough and vGPU variants of `cozystack.gpu-operator` produce a KubeVirt CR that is immediately usable, without an out-of-band `kubectl patch`. Today the gpu-operator stack reports green, `nvidia.com/<model>` shows up under node `.status.allocatable`, but every `VirtualMachine` referencing the resource via `domain.devices.hostDevices` is rejected by the admission webhook because `HostDevices` is not in `developerConfiguration.featureGates` and the resource is not listed under `permittedHostDevices`. That manual reconciliation is now driven by the platform. The change has three parts. `packages/system/kubevirt` now parameterizes the KubeVirt CR template. `developerConfiguration.featureGates` keeps its existing static defaults and appends `.Values.extraFeatureGates`; `spec.configuration.permittedHostDevices` renders verbatim from `.Values.permittedHostDevices` when set; `spec.configuration.mediatedDevicesConfiguration` does the same. Defaults stay empty so the rendered CR is byte-identical to the previous template when no overrides are supplied. The chart's `make update` target re-applies the four sed transforms after fetching the upstream CR, so refreshing the KubeVirt release version stays a single command. `packages/core/platform` adds two defaults tables under `files/`. `gpu-passthrough-defaults.yaml` covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and Volta (V100/V100S); PCI vendor:device pairs are stable, `resourceName` slugs follow the `<arch>_<model>_<form>_<mem>` convention `nvidia-sandbox-device-plugin` generates as of v25.x, and `externalResourceProvider: true` is set everywhere because those resources are advertised by the sandbox plugin rather than KubeVirt's in-tree device plugin. `gpu-vgpu-defaults.yaml` covers the Pascal–Ampere mdev profile range (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family span the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and Blackwell SR-IOV vGPU is intentionally out of scope for the default table and is expected to be advertised via the user-override hook described below. `templates/bundles/iaas.yaml` now selects between the two tables via a new `bundles.iaas.gpuOperatorVariant` knob (`default` for passthrough, `vgpu` for mediated; default is passthrough). The same variant flows through to the gpu-operator Package CR, replacing the previous always-default call. When the package is enabled, the platform injects `HostDevices` into `extraFeatureGates` plus the chosen defaults into `permittedHostDevices` for the kubevirt component's values. Operators extend or replace via top-level platform values: ```yaml gpu: replaceDefaults: false # append (default) vs. wipe-and-replace permittedHostDevices: pciHostDevices: [] # cluster-specific PCI passthrough entries mediatedDevices: [] # cluster-specific vGPU entries mediatedDevicesConfiguration: {} # extra mediated-device types / per-node mappings ``` `docs/gpu-vgpu.md` is updated to drop the manual `kubectl patch` step and describe the auto-wiring contract, the per-variant defaults, the escape hatches, and the sandbox-device-plugin `resourceName` slug convention so an operator whose node publishes a different name knows where to override. ## Test coverage `helm-unittest` covers eight branches in `packages/core/platform/tests/bundles_gpu_kubevirt_wiring_test.yaml`: gpu-operator disabled emits no extras; gpu-operator enabled at default variant emits the HostDevices gate plus `pciHostDevices`; vgpu variant switches to `mediatedDevices` plus `mediatedDevicesConfiguration`; user overrides append by default; `replaceDefaults: true` wipes the platform table; an unknown variant fails fast with a descriptive error. `packages/system/kubevirt/tests/kubevirt_cr_test.yaml` pins the empty-values render and the three populated shapes at the CR template level. `make helm-unit-tests` passes locally on the full repo. ### Release note ```release-note feat(platform): the KubeVirt CR is now auto-configured for GPU host-device passthrough (HostDevices feature gate plus permittedHostDevices populated from NVIDIA defaults) whenever cozystack.gpu-operator is in bundles.enabledPackages. Operators no longer need a manual kubectl patch to make VMs schedule against a GPU. The variant is selected via bundles.iaas.gpuOperatorVariant (default: passthrough; vgpu: mediated). Extend or replace the NVIDIA defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults. **Upgrade note:** the bundle now OWNS spec.configuration.permittedHostDevices on the KubeVirt CR. If you previously hand-edited it (kubectl edit kubevirt), the first reconcile after upgrade overwrites your manual entries with the rendered NVIDIA default table. Before upgrading, move any custom entries into .gpu.permittedHostDevices (set .gpu.replaceDefaults: true to keep only your list) and verify each resourceName against what your nodes advertise (kubectl describe node <node> | grep nvidia.com/) — the default table now carries the exact slug the sandbox device plugin generates, which upper-cases the card's PCI-IDs name and keeps the Tesla/GL tokens (e.g. a Tesla T4 is nvidia.com/TU104GL_TESLA_T4), but a different plugin build can publish a different string; a mismatch makes the admission webhook reject GPU VMs on their next restart or migration. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Bundle-managed GPU Operator with selectable passthrough vs vGPU variants; platform now injects GPU settings into KubeVirt and forwards vGPU manager overrides when enabled. * Default platform mappings for PCI passthrough and vGPU mediated-device resources added. * **Documentation** * Expanded GPU/vGPU deployment guide with bundle-managed vs manual guidance, wiring/precedence rules, resource-name conventions, and verification steps. * **Tests** * New and expanded tests for bundle→KubeVirt wiring, variant behaviors, default/override merging, and idempotent template parameterization. * **Chores** * Added Helm test target and hardened KubeVirt template update/idempotency checks. <!-- end of auto-generated comment: release notes by coderabbit.ai -->