Commit Graph
1 Commits
Author SHA1 Message Date
Aleksei Sviridkin 9da24c2aa8 feat(gpu): bind, move and release GPU devices in the leader
The binder is the only writer of a device's placement, so budget and
capacity checks that admission already ran cannot be raced past by two
replicas or two webhook calls. Every event collapses into one pass over
all GPU objects, which serializes the decisions without per-object
locks, and devices are read past the cache so a unit booked in the last
pass is never booked again from a stale view.

A device that carries an unanswered bind request also queues a pass
that only answers requests, so the webhook, which waits seconds, is not
held behind moves, usage records and status of the whole cluster. The
full pass moves held sets off nodes that cannot keep them, rebinds Lost
sets before new orders, and binds new orders onto the fullest node with
room. Finalizers and usage records run with the feature disabled too,
so turning GPU off never hangs a namespace or class deletion.

The devices one VM names move, go Lost and are rebound as a set. Every
pass restores that invariant before it places anything, gathering a set
left partly Lost, by a write that failed half way or by a VMI that ended
next to held members, onto one node or sending all of it to Lost, so
each later step only ever changes whole sets.

A Lost device already counts against its tenant's budgets, so the pod of
its still running VMI is not counted again under a raw GPU key; for node
capacity it is, since the device itself holds no units.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
2026-10-21 06:33:42 +03:00