On a fresh install the platform turns GPUs for VMs on, and so passes
--enable-gpu, while KubeVirt's operator may not have installed the VMI,
VM and KubeVirt CRDs yet. An informer on a kind no CRD serves cannot
start, and cozystack-controller exited on it, taking every reconciler
down with the GPU paths.
Now nothing outside the GPU paths waits for KubeVirt. The binder starts
with its other watches and reads no VMI, VM or KubeVirt object while
their kinds are not served, so finalizers, usage records and placement
keep running, and joins the KubeVirt watches from a runnable once the
kinds appear. While a device is attached, an unserved kind is a gap in
discovery rather than an absent VMI, and the pass fails instead of
detaching a running VM's device.
Every replica warms the informers the webhooks read, and readiness
fails until those that need no KubeVirt have synced. Readiness does not
wait for KubeVirt itself: cozystack-api, the dashboard and the lineage
webhook wait for cozystack-controller to be ready, and must not wait
for KubeVirt, nor hang when it is not installed. The VMI and KubeVirt
informers start once their kinds are served.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The NVIDIA and mediated entries of the KubeVirt permittedHostDevices
table are GPU resource names besides the hoster's gpu.resourceNames
list, whether the hoster sets one or not. Without that, a
GPU name only the table carries, on a node whose agent reports nothing
and with no class of its own, would not be a GPU resource name: under
forbidden the raw-name policy would let a VM or a pod request it
directly, and the VMI webhook would take it for a non-GPU host device.
The binder adds those names to the effective set it publishes in the
settings ConfigMap, which the policy and cozystack-api read, and the VMI
webhook adds them when it looks for a VMI's own GPU entries. A PCI entry
of another vendor, such as a ConnectX VF, stays out. Whether a class's
resource name is in the table at all is still checked against every
PCI and mediated entry.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binder, the VMI webhook, the catalog option source and the usage
records each leave out of a pod's GPU usage the units its VMI's devices
already hold. They matched the pod to the VMI by its owner reference,
the usage records also by the device-reference label, and whoever
creates the pod writes both, so a pod naming a VMI with attached
devices could hide its GPUs from capacity, raw-key budgets and the
usage records.
A pod now counts as a VMI's launcher only when that VMI lists the pod's
UID in status.activePods; any other pod's GPUs count as raw-name usage.
One shared reader decides it for all four. The binder takes the active
pods from the VMIs it already lists, the webhook from the VMI informer
every replica warms, the usage records from a read of the named VMI,
and cozystack-api reads the named VMI only for a pod that requests a
raw GPU name, for which it gains get on virtualmachineinstances.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The chart writes the device references and the webhook reads them, in
two packages with no shared code, so a renamed field on either side
would leave VMs the webhook refuses to start. The vm-instance
chart tests already pin the rendered annotation; this test reads each
pinned value back through the webhook's parser.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binder, the admission webhooks, the catalog option source and the
tenant quota gate carried copies of the tenant release name mapping,
the resourceQuotas reader, the permittedHostDevices reader and the
per-VMI device coverage, and the copies had started to drift: the
catalog skipped a budget it could not read, where the others failed.
The VMI webhook still counted one unit per healthy GPU function when a
node had lost capacity, so it refused a VM whose devices an mdev
parent's remaining slots still served while the binder would have kept
them; the units a node serves are now counted once, for both.
They now live in the shared model, next to the tenant walk they depend
on. Coverage keeps two named readers on purpose: for budgets every
attached device covers its pod, since a Lost device still counts
itself, while for node capacity only a device holding units does.
Merging them would count a Lost device twice or not at all.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
With raw names forbidden a GPU reaches a VM only through a GPUDevice
reference, which makes a held device a hard guarantee. A VMI carrying a
GPU entry (any gpus entry, a hostDevices entry or a resource of a GPU
resource name) without the device-reference label is refused, which
covers a VM written by a tenant super-admin, a VMIReplicaSet and a
worker pool. A pod requesting a GPU resource name is refused unless
kubevirt-controller creates it with a VMI owner, as it does for
virt-launcher and migration target pods.
The GPU resource names come from the settings ConfigMap the controller
publishes. The binding fails closed without it, and since the parameter
is resolved before any expression runs, the bindings select tenant
namespaces only; anything wider would block every pod in the cluster,
the controller's own included, until the ConfigMap exists. The
usage-monitor label policy now selects tenant namespaces the same way,
so the two cannot disagree on what a tenant namespace is, and its
ratchet also refuses changing the label's value, which could make an
object labelled before the policy start matching the monitor.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The GPU admission webhooks need a Service, a serving certificate and
webhook configurations, and the binder's metrics need a port a scrape
can reach. All of it renders only with GPU enabled, which the platform
sets where KubeVirt serves GPUs, so clusters without KubeVirt change
nothing.
The webhooks fail closed. The two mutating ones select only VMIs and
virt-launcher pods carrying the device-reference label, the pod webhook
also requiring the virt-launcher label, so of VMIs and pods a
controller outage blocks only the start of VMs with device references;
the GPUDevice webhook blocks new orders as well. The controller then
runs two replicas behind a PDB, since a single fail-closed replica
would block those starts on every rollout.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A tenant references held devices and device templates on the VMI
template; something has to turn those references into GPUs on one node
before KubeVirt renders the launcher pod, and refuse a start that cannot
work instead of leaving a VMI Pending with no pod.
The GPUDevice webhook refuses an order that cannot be met now: an
unknown or deleted class, maxCount, the class's free units or one
node's, a budget of the tenant or an ancestor, or a vGPU class without
the entitlement.
The VMI webhook runs every refusal check before it writes anything, so
a dry run and a refused start leave no trace. It then creates the
template devices, attaches the whole set with optimistic concurrency so
two VMIs never take one device, and waits a bounded time for the
leader's binder to place the set on one node. The API server calls it
again after virt-api's mutator changes the VMI, so a recent answered
request for the same VMI is taken as its own and nothing is written
twice. It adds a gpus entry per GPU and never a resources key as
well, since the launcher pod would request the sum.
The pod webhook pins the launcher pod to the devices' node with a
required node affinity on the pod, because LiveUpdate rewrites the
VMI's affinity from the VM template.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
RBAC cannot name the caller's node, so the agent's role lets every
agent write every GPUNode, and a compromised node could rewrite the
inventory, and through resourceNames the cluster's GPU resource-name
set, of any other node. The pod-bound service account token carries
the node name as a user extra, which a ValidatingAdmissionPolicy can
compare with the object name.
A token without that extra is refused rather than let through: the
policy matches only the agent's service account, so failing closed
stops the agent and nobody else.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
GPUNode is the hoster's per-node view of GPUs and the binder's source
of device health, so something on each node has to report it. The
agent reads the GPU functions from sysfs and what kubelet's device
plugins make of them from the pod-resources API.
Plugins report devices by PCI address, mdev UUID or IOMMU group, so each
ID is resolved through sysfs by its shape rather than by knowing the
plugin. A group counts only through functions bound to vfio, so the
plain indexes NVIDIA's container plugin may report do not read as
groups. GPUs of NVIDIA's container plugin, reported by NVML UUID, cannot
be placed through sysfs, but their resource name still joins
resourceNames, since pods request GPUs by it. pod-resources carries no
health; kubelet lists only healthy devices as allocatable, which is what
the agent reads health from, and it counts them per function, because an
mdev parent serves several. Pods in a final phase are skipped because
kubelet 1.32 and 1.33 still list them with devices a new pod may hold.
The binder writes conditions into the same status, so the agent only
merge-patches its own fields, and it writes no GPUNode but its own
node's.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binding contract said the binder either gathers a set onto one
eligible node or refuses it. A set held entirely on one cordoned node is
answered Succeeded without a move in two cases the design requires: a
cordon alone does not move a set with no room elsewhere, and a device of
a class with onCordon Stay keeps its node. The admission webhook pins
the pod to the devices' current node, so it relies on this answer; the
contract now says so.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The GPU reconciler always runs, so GPU finalizers and usage records are
handled on every cluster, including ones that later turn GPU off;
--enable-gpu adds placement and the KubeVirt watches. GPU admission
webhooks are registered only with it, because asking the manager for
its webhook server starts it and the server needs a serving
certificate. The hoster's resource-name list and raw-name policy come
in as flags and are written to the settings ConfigMap for the
admission policies to read.
GPU admission waits for the binder only seconds, so every replica has
to answer from warm caches: with GPU enabled, the informers it needs
start before the manager on every replica rather than at the first
admission, readiness waits for exactly those to sync, and the leader
releases its lease on shutdown so a rollout does not leave bind
requests unanswered for a lease duration. The ConfigMap cache holds
only the settings object. Non-positive grace periods are refused, since
a zero attachment grace would clear every attachment before its VMI
appears.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
The binder is the only writer of a device's placement, so budget and
capacity checks that admission already ran cannot be raced past by two
replicas or two webhook calls. Every event collapses into one pass over
all GPU objects, which serializes the decisions without per-object
locks, and devices are read past the cache so a unit booked in the last
pass is never booked again from a stale view.
A device that carries an unanswered bind request also queues a pass
that only answers requests, so the webhook, which waits seconds, is not
held behind moves, usage records and status of the whole cluster. The
full pass moves held sets off nodes that cannot keep them, rebinds Lost
sets before new orders, and binds new orders onto the fullest node with
room. Finalizers and usage records run with the feature disabled too,
so turning GPU off never hangs a namespace or class deletion.
The devices one VM names move, go Lost and are rebound as a set. Every
pass restores that invariant before it places anything, gathering a set
left partly Lost, by a write that failed half way or by a VMI that ended
next to held members, onto one node or sending all of it to Lost, so
each later step only ever changes whole sets.
A Lost device already counts against its tenant's budgets, so the pod of
its still running VMI is not counted again under a raw GPU key; for node
capacity it is, since the device itself holds no units.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
The GPU controller writes the usage records of held devices under a
WorkloadMonitor whose selector is a reserved label key, so the monitor
must match nothing and tenant labels cannot shape GPU usage records.
A write that adds the key to any object or subresource in a tenant
namespace is refused, whoever makes it, which also covers the
virt-launcher pods that copy a VM template's labels and the
DataVolumes created from a VM's dataVolumeTemplates.
The policy renders whether or not GPUs are on, so no tenant can place
the label before the hoster turns GPUs on. It ratchets: an object that
already carries the key can still be updated, lose its finalizers and
drop the key, so nothing labelled before the policy can get stuck. It
sees every write in those namespaces, so its expression reads only
metadata and guards each step.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Whoever orders GPUDevice objects must not touch two parts of a device.
The device finalizer keeps a released device, its binding and its usage
record until no VMI uses it, so a tenant who could drop it could take a
held device out of the usage records. The template-device label is the
mark the VMI webhook is to set on devices it creates from a
deviceTemplate, and the GPU controller deletes devices so marked with
their VMI, so on a tenant's own order it would get that order deleted.
Both are refused for everyone but platform callers, matched by group
like the other cozystack-basics policies: cozy-system and kube-system
service accounts and cluster admins, plus kube-controller-manager,
which has no group of its own. The orphan and foregroundDeletion
finalizers belong to the API server and the garbage collector and stay
free.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Held devices and their usage records name their class, so while the
class carries its protection finalizer, what it sells (vendor, model,
mode, profile, memory, maxCount and the kind of source) cannot change.
The policy relies on the GPU controller setting the finalizer on every
class, so a changed offer is a new class. resourceName and nodeSelector
stay open for host-layer changes and for moving the offer. CRD rules
cannot see finalizers, so this is an admission policy, and it reads the
stored object so the write that drops the finalizer cannot change the
spec as well.
A class being deleted waits only for the devices that already hold it,
so a second policy, evaluated once per class, refuses new orders of it.
The policies render only where the platform switched GPUs for VMs on.
Their tests evaluate the expressions read from the chart, compiled for
both the current library and the oldest served Kubernetes minor.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binder re-checks what admission checked, and cozystack-api shows
the remaining budget a later order is checked against; computing these
in three places would let them disagree. The model is pure, so each
caller builds a snapshot from its own cache.
GPU budgets cap the whole subtree of the tenant that declares them,
walked over the namespace ancestry and independent of the tenant quota
pools. A raw GPU key blocks a class until the same tenant also declares
a GPU key, and then caps devices and raw-name pods together. Pod usage
follows the scheduler's arithmetic, sidecars included, so the binder
never books a unit a raw-name pod holds.
A node takes binds only when it is schedulable, Ready and carries no
NoSchedule or NoExecute taint the pod does not tolerate; the
unschedulable, not-ready and unreachable taints are left to the first
two checks, and a taint never counts as a cordon. Gt and Lt
tolerations do not count, since the scheduler ignores them where their
feature gate is off.
Template device names are derived from the VMI and shortened with a
hash when the VMI name leaves no room, so they stay valid subdomains.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binder, the admission webhooks, the tenant chart, the policies and
cozystack-api all have to agree on the same label, annotation,
finalizer and quota keys and on the settings ConfigMap the policies
take as a parameter, so they are defined once here.
The binder in the leader is the only writer of placement; webhooks
running in every replica record an attachment and a bind request in
the device status and wait for the binder's answer, so capacity and
budgets are decided in one serialized loop. Status is the carrier
because no tenant role can write it. The template-device mark carries
no VMI name, since a VMI name can exceed a label value.
Register and Setup exist as no-ops so the controller can be wired to
them before the webhooks and reconcilers land.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
GPUs become a catalog of hoster offers, tenant reservations against
them and a per-node inventory, so a tenant can hold a GPU while its VM
is off and an order that does not fit is refused at create time.
The CRDs enforce what the apiserver alone can: a class name is a DNS
label other than total because it becomes part of a quota key, the
source is a union with dra reserved for a later member, a device's
spec is immutable and orders at least one GPU, and bound devices are
listable per node through a field selector on status.nodeName. A
class resource name must be a valid extended resource outside
gpu.cozystack.io, where it would read as a GPU budget key. The CRDs
are kept on chart uninstall, since they ship as plain templates and
deleting GPUDevice would drop every hold and usage record.
maxCount carries no schema default: it defaults to 1 for vgpu classes
and to no limit for passthrough, which a structural default cannot
express, so readers resolve it through the shared model.
A GPUNode device reports how many device-plugin devices kubelet
counts as allocatable for it, because an mdev parent serves several
from one PCI function and one healthy entry per function would
undercount the node.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin