GPUNode is the hoster's per-node view of GPUs and the binder's source
of device health, so something on each node has to report it. The
agent reads the GPU functions from sysfs and what kubelet's device
plugins make of them from the pod-resources API.
Plugins report devices by PCI address, mdev UUID or IOMMU group, so each
ID is resolved through sysfs by its shape rather than by knowing the
plugin. A group counts only through functions bound to vfio, so the
plain indexes NVIDIA's container plugin may report do not read as
groups. GPUs of NVIDIA's container plugin, reported by NVML UUID, cannot
be placed through sysfs, but their resource name still joins
resourceNames, since pods request GPUs by it. pod-resources carries no
health; kubelet lists only healthy devices as allocatable, which is what
the agent reads health from, and it counts them per function, because an
mdev parent serves several. A device named by its IOMMU group counts
once, on the group's lowest-addressed GPU, however many GPUs the group
holds. Pods in a final phase are skipped because kubelet 1.32 and 1.33
still list them with devices a new pod may hold.
The binder writes conditions into the same status, so the agent only
merge-patches its own fields, and it writes no GPUNode but its own
node's.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
GPUs become a catalog of hoster offers, tenant claims against them and a
per-node inventory, so a tenant can hold a GPU while its VM is off and
an order that does not fit is refused at create time.
The CRDs enforce what the apiserver alone can: a class name is a DNS
label other than total because it becomes part of a quota key, the
source is a union with dra reserved for a later member, a class's mode
never changes because a pool that names the class takes its worker
template from it, a claim's spec is immutable and orders at least one
GPU, and bound claims are listable per node through a field selector on
status.nodeName. A class resource name must have the shape of an
extended resource name and lie outside gpu.cozystack.io, where it would
read as a GPU budget key. The CRDs are kept on chart uninstall, since
they ship as plain templates and deleting GPUClaim would drop every hold
and usage record.
A class carries the tolerations of its VMs. A claim is bound when it is
ordered, before any pod exists, so a dedicated taint on GPU nodes can
only be checked against the class; without them such a node would serve
no class at all. The tolerations are copied onto launcher pods, so the
class takes only what pod validation accepts on every cluster: Exists
and Equal, since Gt and Lt are refused where their feature gate is off,
and the key, value, effect and tolerationSeconds rules of a pod
toleration. They are a type of their own rather than the core one,
because the core type carries no schema to put length limits on, and
without them the checks would not fit the CEL cost budget.
A claim records the nodes that refused its set's pinned launcher pod,
each with an expiry, in status, so a leader change does not forget them
and the controller does not pin the set to the same node again on its
next pass.
maxCount carries no schema default: it defaults to 1 for vgpu classes
and to no limit for passthrough, which a structural default cannot
express, so readers resolve it through the shared model.
A GPUNode device reports how many device-plugin devices kubelet counts
as allocatable for it, because an mdev parent serves several from one
PCI function and one healthy entry per function would undercount the
node.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin