Closes the gpu-operator half of #3563. Tenant clusters have run Talos
workers since 1.6.0, where the NVIDIA driver comes from a system extension
baked into the worker image rather than from the operator's driver
container. The addon default still described the Ubuntu shape, so a GPU
node group came up with the operator fighting the OS for the driver and no
path at all for CUDA libraries to reach a tenant pod.
Three things change, each fixing one silent failure:
driver.enabled and toolkit.enabled go false. The driver container would
load its own module and clash with the extension's; the toolkit container
would rewrite /etc/containerd/config.toml, which Talos owns and which
already registers the nvidia runtime with enable_cdi = true and
cdi_spec_dirs = ['/run/cdi'].
validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and
disables /dev/char symlink creation. The driver validator otherwise looks
for a driver container install, finds none, never writes its sentinel, and
leaves the device plugin blocked behind toolkit-validation — the loop on
"Attempting to validate a driver container installation".
devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the
host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need
one on Ubuntu, where the toolkit made nvidia the default containerd
runtime; Talos does not, so CDI is the only route for the CUDA libraries.
DEVICE_ID_STRATEGY, NVIDIA_DRIVER_ROOT and CDI_ANNOTATION_PREFIX are set
and tested alongside it: CDI device names key on the UUID strategy, the
plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation
prefix has to be the one containerd reads, so a regression in any of them
breaks device resolution on its own.
The previous default fed NVreg_NvLinkDisable=1 to the driver container
through the nvidia-kernel-module-params ConfigMap. With that container off
the ConfigMap is mounted by nothing, so the default is removed. On Talos a
module parameter belongs to the worker machine config, which the
kernelModules field of the KubernetesNodes app
(packages/apps/kubernetes-nodes, #3571) writes. The wrapper chart keeps
the ConfigMap for an operator who re-enables the driver container.
Every value here is the configuration validated on a production Talos GPU
node group, minus the parts that only mattered while the driver container
was in play.
Not covered: HAMi. When addons.hami.enabled the operator's device plugin
stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent
CDI wiring on Talos that this default does not attempt.
Refs: #3563
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
## What this PR does
`bundles.iaas.gpuOperatorVariant: container` failed the whole platform
render. The `cozystack.gpu-operator` PackageSource ships that variant,
`values-container.yaml` configures it deliberately, and
`hack/check-gpu-operator-variants.bats` covers it, but the `iaas` bundle
guard only accepted `default` and `vgpu`.
The exclusion made sense when it was written and stopped making sense a
few minutes later. The variant and the guard came from parallel
branches, neither an ancestor of the other, variant merged first. Before
the guard the bundle hardcoded `default`, so `container` was selected by
hand-writing a Package CR, and leaving it out of a bundle-side list cost
nothing. The guard made the bundle the variant owner and closed that
route without noticing it was one. Its comment calls itself typo
protection and names `passthrough` as the example typo.
Allowing the value is not the whole fix. The same file puts
`extraFeatureGates: [HostDevices]` on the KubeVirt CR whenever
gpu-operator is enabled, and builds a `permittedHostDevices` table under
it. Both are VM-facing. The container variant keeps the host driver
bound and pins `vfioManager` off, so no GPU reaches a VM: the gate would
advertise something the node cannot serve, and the table would name
resources no sandbox device plugin registers there. Both now render for
`default` and `vgpu` only, and `container` gets the gpu-operator Package
with the kubevirt component untouched.
The helm-unittest case that pinned the rejection is gone. Two replace
it: the Package renders at the container variant with no component
overrides, and the KubeVirt wiring stays absent even with
`gpu.permittedHostDevices` and `gpu.mediatedDevicesConfiguration` set.
Comments and docs that called the mirroring unconditional on variant, or
the container variant unmerged, are fixed in the same pass.
No issue for this one, so the history is in the body.
### Screenshots
No UI change.
### Downstream repositories
Walked the trigger map against the diff. The only file it touches that
appears there is `packages/core/platform/values.yaml`, and the website
hit is not the hand-written key table (that page has no GPU section): it
is the documented workaround this change makes obsolete.
`content/en/docs/next/operations/gpu-container-workloads.md` currently
tells readers the bundle rejects `container` and routes them to a
hand-crafted Package CR, and `virtualization/gpu.md` / `vgpu.md`
describe the KubeVirt mirroring as unconditional on variant.
Filed as an issue rather than a PR because the docs must not land ahead
of this change, and the `v1.6` copies describe released behaviour and
stay correct as they are.
Nothing else matches: no package added, renamed or removed, no
`values.schema.json` or version enum touched, no `ApplicationDefinition`
semantics, no installer values key, no `hack/` layout change, no
platform variant or bundle recomposed. `gpuOperatorVariant` appears in
none of `ccp`, `ansible-cozystack` or `terraform-provider-cozystack`.
- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/685
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(platform): accept `container` as a value for `bundles.iaas.gpuOperatorVariant`, so the gpu-operator variant the platform already ships can be selected from platform values instead of a hand-crafted Package CR. The KubeVirt `HostDevices` feature gate and `permittedHostDevices` table are now emitted for the `default` and `vgpu` variants only; `container` serves GPUs to pods and gets neither.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added the `container` GPU operator variant for workloads that do not
require VM passthrough.
* Container workloads no longer receive KubeVirt host-device or
mediated-device GPU wiring.
* Clarified GPU configuration behavior for passthrough, vGPU, SR-IOV,
and container workloads.
* **Documentation**
* Updated passthrough guidance, configuration references, examples, and
KubeVirt wiring instructions for all supported variants.
* Added details for manually configured package resources and
platform-specific container setup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The cozystack.gpu-operator PackageSource ships three variants: default,
vgpu and container. The iaas bundle's variant guard accepted only the
first two, so selecting container failed the render on a value the
package ships, values-container.yaml configures deliberately, and the
variant bats suite already covers.
The exclusion was coherent when it was written and stopped being so
immediately. The container variant and the guard landed within minutes
of each other from parallel branches, neither an ancestor of the other,
variant first. Until the guard, the bundle hardcoded the default variant
and container was selected by hand-crafting a Package CR, so leaving it
out of a bundle-side list cost nothing. The guard then made the bundle
the variant owner and closed that route without noticing it was one; its
comment describes itself as typo protection and names passthrough as the
example typo.
The KubeVirt side is not a bare allowlist edit. The container variant
serves GPUs to pods through the device plugin and runs no VM-facing
GPU stack: sandboxWorkloads and vfioManager are both off and the host
driver stays bound. Both KubeVirt-side pieces the bundle wires are
VM-facing — the HostDevices feature gate advertises a capability such
a node cannot serve, and permittedHostDevices lists resources no
sandbox device plugin registers there. Both are now emitted only for
default and vgpu, and container yields the gpu-operator Package with
the kubevirt component untouched.
Comments and docs that described the KubeVirt mirroring as
unconditional on variant, or the container variant as unmerged and
unavailable, are corrected to match.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
hack/helm-unit-tests.sh decides whether to enter a package by asking
`make -C <pkg> -n test`, and packages/system/gpu-operator declared only
`update:`. The sweep skipped it in silence, so the three test cases in
tests/kernel_module_config_test.yaml had never executed. They pass, so
this is dead coverage rather than a hidden red -- but the directory read
as coverage to anyone opening it, and a suite that has never run is
where a stale assertion hides.
Declare the target phony: hack/package.mk's .PHONY line lists only the
targets that file defines, so a path named `test` next to the Makefile
would otherwise make the target up to date and skip the recipe.
Pin the class in hack/, because every other guard in the sweep -- the
shadowed-target check, the zero-suite check, the positive-evidence
check -- runs inside a directory the sweep already reached, and a
package missing the target escapes all of them at once. The new case
probes with `make -n` rather than grepping for `test:`, so it asks the
same question the sweep asks, and it reports make's own diagnostic
rather than asserting the cause: an unparseable Makefile fails the same
probe as a missing rule and wants a different fix. Its predicate is
"ships at least one tests/*_test.yaml", not "has a tests/ directory":
that glob is what helm-unittest collects, and packages/system/nfs-driver
keeps a PersistentVolumeClaim and a StorageClass under tests/ to apply
by hand, with no suite for a target to run.
Of the 84 packages holding a tests/ directory, 83 ship a suite and
gpu-operator was the only one of those without the target.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.
The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.
- driver.enabled=false use the host driver
- toolkit.enabled=false host already has nvidia-container-toolkit;
enabling overwrites the containerd
config.toml the host configured via
nvidia-ctk runtime configure
- devicePlugin.enabled=true publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false do not unbind the host driver
- cdi.enabled=false CDI spec generation is serviced by the
disabled toolkit; an apt host uses the
legacy nvidia-container-runtime hook, so
pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false regular pod workloads, not VMs
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The wrapper chart named the kernel-module ConfigMap from its own
kernelModuleConfig.name key, while the NVIDIA driver mounts the ConfigMap
referenced by gpu-operator.driver.kernelModuleConfig.name. The two keys
defaulted to the same value but were never linked, so overriding only the
driver reference left the driver pointing at a ConfigMap that is never
rendered and the NVreg_NvLinkDisable=1 parameters were silently dropped.
Derive the ConfigMap name from the driver's effective reference, falling
back to the wrapper key when unset, so the rendered ConfigMap and the
driver reference can no longer diverge. Add helm-unittest coverage for the
rendered name, namespace, data key, and the create=false no-op.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When a single SXM GPU (e.g. H200) is passed through into a tenant worker
VM without the NVSwitches, the nvidia driver waits forever for an NVLink
fabric that can never come up (Fabric State: In Progress). Every CUDA
call then fails with 'system not yet initialized' and the
nvidia-operator-validator crashloops. Loading the kernel module with
NVreg_NvLinkDisable=1 flips Fabric State to N/A so CUDA initializes —
correct for per-GPU passthrough, which has no in-VM NVLink/NVSwitch
peering anyway.
The upstream gpu-operator chart only references an existing ConfigMap by
name via driver.kernelModuleConfig.name and never creates it. Render
that ConfigMap from the gpu-operator wrapper chart (opt-in via
kernelModuleConfig.create) so it lands in the tenant cozy-gpu-operator
namespace alongside the operator, and wire the tenant HelmRelease
defaults to enable it and point the driver at it. Remains overridable
via addons.gpuOperator.valuesOverride.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Adds hack/check-gpu-operator-variants.bats. The test renders both
package variants with helm template and asserts the resulting
ClusterPolicy spec for the values that this PR cares about:
* ccManager.enabled=false (pinned in values.yaml; upstream v26 default
was flipped to true)
* vgpuDeviceManager.enabled=false (same — and on Ada+/Blackwell the
DaemonSet would crashloop because /sys/class/mdev_bus is empty)
* sandboxWorkloads.defaultWorkload — vm-passthrough on the default
variant, vm-vgpu on the vgpu variant; without these the upstream
'container' default makes either variant a silent no-op on
unlabelled GPU nodes
* vgpuManager.enabled=true on the vgpu variant; driver / devicePlugin
enabled=false on both variants (proprietary host driver path is
out of scope for both passthrough and SR-IOV vGPU)
Negative-tested locally: deleting any of these settings makes the
corresponding test fail, so it actually guards regressions on the
next chart bump.
Also addresses last-pass review comments:
* values-passthrough.yaml: set sandboxWorkloads.defaultWorkload=
vm-passthrough so the default variant is symmetric with the vgpu
variant — neither relies on per-node labels for the variant to
activate. Without this the upstream default 'container' would
apply on unlabelled GPU nodes and vfio-manager would never start.
* values.yaml: rewrite the vgpuDeviceManager comment to lead with
the load-bearing reason (mdev path missing on Ada+/Blackwell)
instead of the secondary 'stays at zero pods on passthrough'
rationalisation.
* docs/gpu-vgpu.md: add a 'Migrating from chart v25.x' subsection
in the Licensing section pointing at the configMapName ->
secretName rename. Old key still works upstream but emits a
deprecation warning; PR body release note already calls this out
but the docs were missing the migration breadcrumb.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
* examples/README.md and examples/values-native-talos.yaml header
comment still pointed at values-talos.yaml after the rename to
values-passthrough.yaml. Update both to the new name.
* values-vgpu.yaml: drop the leftover vgpuDeviceManager.config block.
The chart only renders the config when vgpuDeviceManager.enabled is
true, and we keep it false on Ada+; the orphan config.default key
was scaffolding without effect.
* docs/gpu-vgpu.md and values.yaml comment: stop grouping B100 under
Ada Lovelace. B100 is Blackwell; the SR-IOV vGPU mechanism applies
to both architectures but they should be named correctly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Addresses second review pass. Three blockers + one polish:
* values-vgpu.yaml: set sandboxWorkloads.defaultWorkload=vm-vgpu.
Without this the upstream default 'container' applies on any node
without the per-node nvidia.com/gpu.workload.config label, the
operator selects the container workload (driver/devicePlugin
DaemonSets), and those are disabled in this variant — so nothing
runs on the node and vgpu-manager never starts. The cluster-wide
default makes the variant functional out of the box; per-node
labels still override it for mixed clusters.
* docs/gpu-vgpu.md: imagePullSecrets is per-component on this chart
(vgpuManager.imagePullSecrets, driver.imagePullSecrets, …). The
Package CR example previously placed the field at gpu-operator: root
where it would be silently ignored at render. Move it under
vgpuManager: where the chart actually reads it.
* docs/gpu-vgpu.md, release note: mark the vgpu variant as
experimental and call out the missing profile-assignment loop
explicitly. Until an SR-IOV-aware controller writes
current_vgpu_type per VF (and re-applies after every reboot —
PCIe re-enumeration resets the value), the variant is not safe
for production. The docs now lead with that warning instead of
burying it in a caveat.
* Rename packages/system/gpu-operator/values-talos.yaml to
values-passthrough.yaml. The overrides (sandboxWorkloads.enabled,
driver and devicePlugin disabled) apply to any host OS using the
vfio-pci passthrough path — not Talos specifically. Update
PackageSource ref accordingly. Comment in the file documents the
rename so future readers understand the prior name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Addresses follow-up review of the vGPU variant + chart bump.
* values.yaml: pin ccManager.enabled=false and vgpuDeviceManager.enabled=false.
Upstream chart v26.x flipped both to true. ccManager auto-enables CC
mode on Hopper-class hardware after upgrade — a non-trivial behavior
change that should be opt-in. vgpuDeviceManager walks
/sys/class/mdev_bus/, which does not exist on Ada+ (L4/L40/L40S/B100
with vGPU 17/20). Pinning both off keeps the default footprint as it
was on v25.x; users explicitly opt in.
* hack/dcgm-default-counters.csv: refresh from dcgm-exporter tag
4.5.1-4.8.0 (matches the image bumped earlier in this PR). The
previous snapshot was pinned to 4.1.1-4.0.4. Adds DCGM_FI_DEV_FB_RESERVED
and other fields introduced upstream. Header comment updated.
hack/check-gpu-recording-rules.bats passes against the refreshed
fixture.
* docs/gpu-vgpu.md (Licensing section):
- Drop the wrong recommendation to wire the licensing Secret through
gpu-operator's driver.licensingConfig.secretName. That mounts into
the host driver pod, where SR-IOV vGPU does not consume a license.
Document guest-side delivery (cloud-init write_files for
/etc/nvidia/ClientConfigToken/ and /etc/nvidia/gridd.conf) end to
end.
- Correct FeatureType comment. Per the NVIDIA Virtual GPU Software
Licensing User Guide, FeatureType=0 is the unlicensed state, not
auto-detect; FeatureType=1 is the value that asks the driver to
auto-select the license type from the configured vGPU profile.
Switch the example to FeatureType=1 and list all four valid values
with their meaning.
- Add nvidia-smi verification snippet inside the guest.
* docs/gpu-vgpu.md (KubeVirt section): note that L40S SR-IOV VFs share
the PF device ID (10de:26b9), so a single pciVendorSelector matches
both. Recommend lspci -nn -d 10de: as a sanity check on other
generations where PF/VF IDs may differ.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Rewrites docs/gpu-vgpu.md to match the actual NVIDIA driver model on
current data-centre GPUs. Key changes:
- Drop the implicit assumption that vGPU uses mediated devices. Ada
Lovelace and newer (L4, L40, L40S, B100) on the vGPU 17/20 branch
use SR-IOV with per-VF NVIDIA sysfs (current_vgpu_type); KubeVirt
advertises them via permittedHostDevices.pciHostDevices.
- Document KubeVirt PR #16890 as a hard dependency for the SR-IOV
path. The PR is in main since 2026-04-10 and will ship in v1.9.0
(ETA July 2026); released tags v1.6.x/1.7.x/1.8.x do not include
it and backports are not planned.
- Replace the legacy NLS configuration block (ServerAddress=,
ServerPort=7070, FeatureType=1) with the current DLS flow:
ClientConfigToken file in /etc/nvidia/ClientConfigToken/, gridd.conf
with FeatureType=0 (auto-detect).
- Switch the build instructions from the archived
gitlab.com/nvidia/container-images/driver to the upstream
github.com/NVIDIA/gpu-driver-container.
- Switch sample VirtualMachine spec from gpus: (mdev resource) to
hostDevices: (PCI resource).
- Note explicitly that Talos is not recommended for the vGPU variant
— the proprietary .run cannot be redistributed in a Talos system
extension, and siderolabs/extensions#461 was closed won't-fix.
Passthrough on Talos is unaffected.
Rename packages/system/gpu-operator/values-talos-vgpu.yaml to
values-vgpu.yaml — the overrides (sandbox workloads, pod-side driver
and device plugin disabled) are not Talos-specific, they apply to
any VM-only GPU node. Update PackageSource accordingly.
Default vgpuDeviceManager.enabled: false. The DaemonSet walks
/sys/class/mdev_bus/, which does not exist on Ada+ — keeping it on
by default would crashloop modern hardware. Operators of older GPUs
(Pascal/Volta/Turing/Ampere) can flip the flag back on.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pull latest patch release: GDRCopy v2.5.2, Kata Sandbox Device Plugin
v0.0.3, multi-arch digest fix for KubeVirt GPU Device Plugin (no longer
amd64-only), and OS release name detection fix. NFD remains v0.18.3.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add vGPU mode as a new variant alongside the existing passthrough (default) variant.
This enables sharing a single physical GPU across multiple VMs using NVIDIA mediated devices.
Changes:
- Add values-talos-vgpu.yaml with vGPU Manager and vGPU Device Manager enabled
- Add 'vgpu' variant to PackageSource
- Add documentation for vGPU setup (driver image, NLS licensing, profiles, KubeVirt)
Users select the vgpu variant via Package CR and provide their proprietary
vGPU Manager image coordinates through component values.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from coderabbitai on packages/system/gpu-operator/examples/nvidia-driver-compat.yaml:77
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Explain why tolerations: [{operator: Exists}] is safe on the driver
compat DaemonSet: the nodeSelector already confines scheduling to GPU
nodes, so the blanket toleration only kicks in when those nodes carry
the dedicated=gpu / nvidia.com/gpu taints that the GPU Operator's
default policy and many deployments apply.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
gpu-fleet.json references DCGM_FI_DEV_POWER_MGMT_LIMIT for its
"TDP vs draw" panel, but the custom DCGM Exporter CSV did not declare
it, so the panel silently rendered "No data" on clusters using that
config. Declare the counter, fix the dashboards table in the
gpu-operator examples README, and add a bats test that cross-checks
every DCGM_FI_* reference in tracked dashboards and recording rules
against the union of the upstream default set (snapshotted under
hack/) and the project's custom CSV.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-fleet: cluster-wide admin view — inventory, capacity (total /
allocated / free), per-node utilization and power, throttling,
temperatures, XID errors.
- gpu-tenants: per-namespace view — live allocation, utilization,
tensor saturation, power, and 24h GPU-hours / kWh integrations for
billing inputs.
Register both under gpu/* in dashboards-infra.list so they ship as
GrafanaDashboard CRs and fall under the bats cross-check introduced
earlier on this branch.
Update examples/README to spell out which DCGM counters each of the
five gpu/* dashboards actually needs on top of the upstream default
CSV — gpu-performance needs profiling and throttling counters,
gpu-efficiency needs profiling, gpu-tenants needs only
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE for its tensor panel, and gpu-fleet
and gpu-quotas work on the default counter set alone.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The previous wording implied that the entire custom DCGM CSV was
required by the recording rules. In fact only the profiling counters
(DCGM_FI_PROF_*) need to be added on top of the upstream defaults —
everything else the rules consume is already in default-counters.csv.
Add a Verification status block flagging that the minimum-set claim is
derived from the DCGM Exporter version pinned in the currently shipped
gpu-operator package and must be re-checked when that package moves to
a newer release.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Restrict the nvidia-driver-compat DaemonSet to nodes labelled
nvidia.com/gpu.present=true (NFD/GPU Operator label). Without the
nodeSelector it was scheduling onto every node — control-plane and
CPU-only workers included — burning a privileged pod slot per host
for no benefit.
Add resource requests and limits to the init and pause containers so
the DaemonSet stays within control-plane budget on small clusters.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add reference manifests (not templates) under
packages/system/gpu-operator/examples/ documenting one working
configuration for running CUDA workloads directly in pods on a Talos
cluster, with DCGM metrics that drive the gpu/gpu-performance
dashboard.
- values-native-talos.yaml: Cozystack Package values that disable the
sandbox path, enable the device plugin, and wire DCGM to the custom
metrics ConfigMap.
- dcgm-custom-metrics.yaml: ConfigMap extending the default DCGM CSV
with profiling, ECC, throttling and energy counters used by the
dashboard and recording rules.
- nvidia-driver-compat.yaml: DaemonSet that stages libnvidia-ml.so.1
and nvidia-smi from the Talos glibc tree into a location the
gpu-operator validator inspects. Workaround for
NVIDIA/gpu-operator#1687.
- README.md: explains why these are shipped as references rather than
first-class templates (sandbox vs native is a deployment choice),
and how the pieces connect.
The out-of-the-box values-talos.yaml still targets the sandbox (VFIO
passthrough) scenario. Operators who want native pod GPU workloads can
start from these references.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Mattia Eleuteri