A cluster whose cozystack-controller predates the GPU binder cannot
pick up GPUDevice by starting a second, newer instance: every
reconciler is registered unconditionally, so the second instance runs
its own copy of the workload, tenant, gateway and secret reconcilers
next to the first.
--gpu-only (with --enable-gpu) registers only the GPU binder, its
webhooks and the probes; nothing else is set up, so no other informer
or cache exists in that process. It is not meant to run beside a
controller from this tree, which runs a binder of its own.
--leader-election-id names the Lease instead of the fixed ID, whose
default stays the historical one. A GPU-only instance that elects
must name its own Lease: on the default one it would take leadership
from the main controller in a shared namespace and stop all of its
reconcilers while holding it. The logger is now set right after flag
parsing, so this and the other flag refusals reach the pod log
instead of exiting silently.
Known limit of the pairing: the older controller's WorkloadMonitor
still counts the GPUs KubeVirt requests on a virt-launcher pod, while
the binder writes a usage record per device, so a VM covered by an
application's WorkloadMonitor is counted twice until the main
controller is upgraded.
The manager wiring is now a list of named steps, so a test can assert
which steps each mode runs without starting a manager. The image
builds the command as a package, since it no longer fits in main.go.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
On a fresh install the platform turns GPUs for VMs on, and so passes
--enable-gpu, while KubeVirt's operator may not have installed the VMI,
VM and KubeVirt CRDs yet. An informer on a kind no CRD serves cannot
start, and cozystack-controller exited on it, taking every reconciler
down with the GPU paths.
Now nothing outside the GPU paths waits for KubeVirt. The binder starts
with its other watches and reads no VMI, VM or KubeVirt object while
their kinds are not served, so finalizers, usage records and placement
keep running, and joins the KubeVirt watches from a runnable once the
kinds appear. While a device is attached, an unserved kind is a gap in
discovery rather than an absent VMI, and the pass fails instead of
detaching a running VM's device.
Every replica warms the informers the webhooks read, and readiness
fails until those that need no KubeVirt have synced. Readiness does not
wait for KubeVirt itself: cozystack-api, the dashboard and the lineage
webhook wait for cozystack-controller to be ready, and must not wait
for KubeVirt, nor hang when it is not installed. The VMI and KubeVirt
informers start once their kinds are served.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The NVIDIA and mediated entries of the KubeVirt permittedHostDevices
table are GPU resource names besides the hoster's gpu.resourceNames
list, whether the hoster sets one or not. Without that, a
GPU name only the table carries, on a node whose agent reports nothing
and with no class of its own, would not be a GPU resource name: under
forbidden the raw-name policy would let a VM or a pod request it
directly, and the VMI webhook would take it for a non-GPU host device.
The binder adds those names to the effective set it publishes in the
settings ConfigMap, which the policy and cozystack-api read, and the VMI
webhook adds them when it looks for a VMI's own GPU entries. A PCI entry
of another vendor, such as a ConnectX VF, stays out. Whether a class's
resource name is in the table at all is still checked against every
PCI and mediated entry.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binder, the VMI webhook, the catalog option source and the usage
records each leave out of a pod's GPU usage the units its VMI's devices
already hold. They matched the pod to the VMI by its owner reference,
the usage records also by the device-reference label, and whoever
creates the pod writes both, so a pod naming a VMI with attached
devices could hide its GPUs from capacity, raw-key budgets and the
usage records.
A pod now counts as a VMI's launcher only when that VMI lists the pod's
UID in status.activePods; any other pod's GPUs count as raw-name usage.
One shared reader decides it for all four. The binder takes the active
pods from the VMIs it already lists, the webhook from the VMI informer
every replica warms, the usage records from a read of the named VMI,
and cozystack-api reads the named VMI only for a pod that requests a
raw GPU name, for which it gains get on virtualmachineinstances.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The chart writes the device references and the webhook reads them, in
two packages with no shared code, so a renamed field on either side
would leave VMs the webhook refuses to start. The vm-instance
chart tests already pin the rendered annotation; this test reads each
pinned value back through the webhook's parser.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binder, the admission webhooks, the catalog option source and the
tenant quota gate carried copies of the tenant release name mapping,
the resourceQuotas reader, the permittedHostDevices reader and the
per-VMI device coverage, and the copies had started to drift: the
catalog skipped a budget it could not read, where the others failed.
The VMI webhook still counted one unit per healthy GPU function when a
node had lost capacity, so it refused a VM whose devices an mdev
parent's remaining slots still served while the binder would have kept
them; the units a node serves are now counted once, for both.
They now live in the shared model, next to the tenant walk they depend
on. Coverage keeps two named readers on purpose: for budgets every
attached device covers its pod, since a Lost device still counts
itself, while for node capacity only a device holding units does.
Merging them would count a Lost device twice or not at all.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The dashboard orders GPUs by class and attaches them by device name,
so it needs the catalog and the tenant's own devices without reading
GPUClass or GPUDevice objects itself.
The gpuclass source lists every class that still takes orders, a
deleted one excluded, with its display name, a short description of
the offer, the free units and the most free units on one node from
the class status, the per-VM maximum, and the requesting tenant's
remaining budget. That budget comes from the shared model over the
tenant and each ancestor's subtree, from their declared resourceQuotas
and every device, and is 0 for a vGPU class without the entitlement.
Pods are read only when an ancestor limits a class's resource name
with a raw key, one list per namespace of the outermost such
ancestor's subtree, and a virt-launcher pod does not count the units
its attached devices already cover. No field names a node.
The gpudevice source lists the devices in the request namespace.
cozystack-api gets read access to both kinds, and get and list on
pods for the raw-key case.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Once the hoster sets gpu.rawResourceNames to forbidden, a VM gets GPUs
only through device references, and admission refuses a GPU entry
anywhere else. The gpu source would still offer raw resource names, and
the instancetype source would still offer instance types such as the
gn1 family that bring their own GPU entries, so the form would lead to
a VM that admission then refuses.
Read the setting and the GPU resource-name set from the controller's
settings ConfigMap. Under forbidden the gpu source is empty, and an
instance type with any gpus entry, or a hostDevices entry on a GPU
resource name, is left out; a NIC passed as a host device keeps its
instance type listed. An unreadable setting serves the full lists: the
pickers are advisory and admission enforces the policy.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A GPU class picker needs more than a name: the portal offers the
smallest of the class's most free units on one node, its per-VM
maximum and the tenant's remaining budget as the largest order, and
shows the class's free units. Carry them as optional integers on the
option item, so an absent field keeps meaning that nothing limits the
order there and every other option source stays unchanged on the wire.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The wording change for GPU-only tenants left one line of the comment
well past the width of its block.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The system memory limits page read "resourceQuotas other than the GPU
budgets is configured", which does not parse.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Hosters need to know what a class must satisfy before tenants can use
it: its resourceName has to be in the KubeVirt permittedHostDevices
table, since KubeVirt checks that only when it renders the virt-launcher
pod and a VM with a missing name hangs Pending instead of failing. The
section also shows how a VM takes devices, which class fields are fixed
once the controller protects a class, the GPU budget keys, how the vGPU
entitlement runs from the root down, and what gpu.rawResourceNames
forbidden changes.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A gpus entry can now reference a held GPUDevice by name or ask for a
device of a class for as long as the VM runs, next to the existing raw
resource name. References are rendered only as the device-reference
label and one JSON list in the devices annotation on the VMI template.
The GPU webhook resolves them when the VMI is created, and the label is
what selects the VMI and its virt-launcher pod for it.
The schema cannot say "exactly one of", so name becomes optional and
the template refuses an entry that sets none or several of the three,
a list mixing raw names with references, a device listed twice, and a
deviceTemplate without a class or with a count below one. A zero
deviceTemplate counts as unset, since a typed client sends one on
every entry. The dashboard offers the tenant's devices and the GPU
catalog through the gpudevice and gpuclass option sources.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Tenant admins and above create, change and delete GPUDevice objects in
their namespaces where GPUs are on. The grant renders under the same
conditions as the policy that keeps tenants off a device's finalizers
and template mark, so no tenant can order without it; the GPU webhook
checks each order's budget and entitlement. Status holds a device's
binding and attachment and stays a subresource no tenant role grants.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
cozystack-controller runs on every cluster, and its GPU paths watch
KubeVirt objects, so the chart's gpu.enabled is set only where the
GPU admission policies apply: where gpu-operator serves GPUs to VMs.
monitoring-agents scrapes the metrics those paths serve under the same
condition.
gpu.resourceNames and gpu.rawResourceNames reach the controller, which
publishes them in its settings ConfigMap for the policies and
cozystack-api. rawResourceNames also reaches cozystack-basics, whose
raw-name policy renders only under forbidden. Both are validated on
every render, so a typo fails before anyone turns GPUs on.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
With raw names forbidden a GPU reaches a VM only through a GPUDevice
reference, which makes a held device a hard guarantee. A VMI carrying a
GPU entry (any gpus entry, a hostDevices entry or a resource of a GPU
resource name) without the device-reference label is refused, which
covers a VM written by a tenant super-admin, a VMIReplicaSet and a
worker pool. A pod requesting a GPU resource name is refused unless
kubevirt-controller creates it with a VMI owner, as it does for
virt-launcher and migration target pods.
The GPU resource names come from the settings ConfigMap the controller
publishes. The binding fails closed without it, and since the parameter
is resolved before any expression runs, the bindings select tenant
namespaces only; anything wider would block every pod in the cluster,
the controller's own included, until the ConfigMap exists. The
usage-monitor label policy now selects tenant namespaces the same way,
so the two cannot disagree on what a tenant namespace is, and its
ratchet also refuses changing the label's value, which could make an
object labelled before the policy start matching the monitor.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The class capacity, bound units and Lost count are kept out of the
world-readable GPUClass status and exported as cozystack-controller
metrics instead, so platform monitoring has to scrape them. The metrics
port is authenticated, so vmagent also needs the right to GET /metrics.
Both render only with GPU enabled, as the port does.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The GPU admission webhooks need a Service, a serving certificate and
webhook configurations, and the binder's metrics need a port a scrape
can reach. All of it renders only with GPU enabled, which the platform
sets where KubeVirt serves GPUs, so clusters without KubeVirt change
nothing.
The webhooks fail closed. The two mutating ones select only VMIs and
virt-launcher pods carrying the device-reference label, the pod webhook
also requiring the virt-launcher label, so of VMIs and pods a
controller outage blocks only the start of VMs with device references;
the GPUDevice webhook blocks new orders as well. The controller then
runs two replicas behind a PDB, since a single fail-closed replica
would block those starts on every rollout.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A tenant references held devices and device templates on the VMI
template; something has to turn those references into GPUs on one node
before KubeVirt renders the launcher pod, and refuse a start that cannot
work instead of leaving a VMI Pending with no pod.
The GPUDevice webhook refuses an order that cannot be met now: an
unknown or deleted class, maxCount, the class's free units or one
node's, a budget of the tenant or an ancestor, or a vGPU class without
the entitlement.
The VMI webhook runs every refusal check before it writes anything, so
a dry run and a refused start leave no trace. It then creates the
template devices, attaches the whole set with optimistic concurrency so
two VMIs never take one device, and waits a bounded time for the
leader's binder to place the set on one node. The API server calls it
again after virt-api's mutator changes the VMI, so a recent answered
request for the same VMI is taken as its own and nothing is written
twice. It adds a gpus entry per GPU and never a resources key as
well, since the launcher pod would request the sum.
The pod webhook pins the launcher pod to the devices' node with a
required node affinity on the pod, because LiveUpdate rewrites the
VMI's affinity from the VM template.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
RBAC cannot name the caller's node, so the agent's role lets every
agent write every GPUNode, and a compromised node could rewrite the
inventory, and through resourceNames the cluster's GPU resource-name
set, of any other node. The pod-bound service account token carries
the node name as a user extra, which a ValidatingAdmissionPolicy can
compare with the object name.
A token without that extra is refused rather than let through: the
policy matches only the agent's service account, so failing closed
stops the agent and nobody else.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
GPUNode inventory is needed exactly where GPUs are served, so the
agent ships as a component of every gpu-operator variant, in the
cozy-gpu-operator namespace. It is marked privileged because its
hostPath mounts are refused below the privileged Pod Security level,
and the operator labels a namespace privileged only for privileged
components.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The inventory has to run next to kubelet: a DaemonSet reading the
host's sysfs and kubelet's pod-resources socket, both mounted
read-only. Connecting to the socket needs root, so the agent runs as
root but with no capabilities and no privilege escalation.
The manager's cached client would watch every pod in the cluster, so
the agent uses a direct client and lists only its own node's pods.
RBAC cannot name the caller's node, so the role is cluster-wide; it
holds only the verbs the agent uses, and the DaemonSet selects the
nodes node-feature-discovery marks as carrying an NVIDIA device, so
the token does not sit on nodes with nothing to report.
The image is in the root build list so that every build path stamps
its reference; the committed tag is replaced by the first build.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
GPUNode is the hoster's per-node view of GPUs and the binder's source
of device health, so something on each node has to report it. The
agent reads the GPU functions from sysfs and what kubelet's device
plugins make of them from the pod-resources API.
Plugins report devices by PCI address, mdev UUID or IOMMU group, so each
ID is resolved through sysfs by its shape rather than by knowing the
plugin. A group counts only through functions bound to vfio, so the
plain indexes NVIDIA's container plugin may report do not read as
groups. GPUs of NVIDIA's container plugin, reported by NVML UUID, cannot
be placed through sysfs, but their resource name still joins
resourceNames, since pods request GPUs by it. pod-resources carries no
health; kubelet lists only healthy devices as allocatable, which is what
the agent reads health from, and it counts them per function, because an
mdev parent serves several. Pods in a final phase are skipped because
kubelet 1.32 and 1.33 still list them with devices a new pod may hold.
The binder writes conditions into the same status, so the agent only
merge-patches its own fields, and it writes no GPUNode but its own
node's.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binding contract said the binder either gathers a set onto one
eligible node or refuses it. A set held entirely on one cordoned node is
answered Succeeded without a move in two cases the design requires: a
cordon alone does not move a set with no room elsewhere, and a device of
a class with onCordon Stay keeps its node. The admission webhook pins
the pod to the devices' current node, so it relies on this answer; the
contract now says so.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binder writes finalizers and status on GPU classes, devices and
nodes, deletes template devices with their VMI, and deletes the
unscheduled VMI of a device that must leave its node. The create right
on GPU devices is for the template devices made on behalf of a VMI.
The settings ConfigMap is the only ConfigMap the controller writes, so
its rights live in a Role in cozy-system rather than the ClusterRole:
get, update and patch are scoped to that object by name, and create,
which RBAC cannot scope by name, is scoped by the namespace.
With --metrics-secure the metrics endpoint authenticates and authorizes
each scrape through TokenReview and SubjectAccessReview, so the
controller needs create on both for the GPU metrics to be scraped.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
The GPU reconciler always runs, so GPU finalizers and usage records are
handled on every cluster, including ones that later turn GPU off;
--enable-gpu adds placement and the KubeVirt watches. GPU admission
webhooks are registered only with it, because asking the manager for
its webhook server starts it and the server needs a serving
certificate. The hoster's resource-name list and raw-name policy come
in as flags and are written to the settings ConfigMap for the
admission policies to read.
GPU admission waits for the binder only seconds, so every replica has
to answer from warm caches: with GPU enabled, the informers it needs
start before the manager on every replica rather than at the first
admission, readiness waits for exactly those to sync, and the leader
releases its lease on shutdown so a rollout does not leave bind
requests unanswered for a lease duration. The ConfigMap cache holds
only the settings object. Non-positive grace periods are refused, since
a zero attachment grace would clear every attachment before its VMI
appears.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
Each GPUDevice now has a usage record of its own, so the virt-launcher
pod of a VMI with device references would report the same GPUs a
second time. The units the VMI's attached devices hold are subtracted
after the resources annotation is merged, since that annotation may
name the GPU too; names the devices do not cover, such as a NIC next to
the GPU, stay reported.
The Workload controller already deletes a record only when its monitor
is gone, which is what lets GPU records live under a monitor that
selects nothing; a test now pins that.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binder is the only writer of a device's placement, so budget and
capacity checks that admission already ran cannot be raced past by two
replicas or two webhook calls. Every event collapses into one pass over
all GPU objects, which serializes the decisions without per-object
locks, and devices are read past the cache so a unit booked in the last
pass is never booked again from a stale view.
A device that carries an unanswered bind request also queues a pass
that only answers requests, so the webhook, which waits seconds, is not
held behind moves, usage records and status of the whole cluster. The
full pass moves held sets off nodes that cannot keep them, rebinds Lost
sets before new orders, and binds new orders onto the fullest node with
room. Finalizers and usage records run with the feature disabled too,
so turning GPU off never hangs a namespace or class deletion.
The devices one VM names move, go Lost and are rebound as a set. Every
pass restores that invariant before it places anything, gathering a set
left partly Lost, by a write that failed half way or by a VMI that ended
next to held members, onto one node or sending all of it to Lost, so
each later step only ever changes whole sets.
A Lost device already counts against its tenant's budgets, so the pod of
its still running VMI is not counted again under a raw GPU key; for node
capacity it is, since the device itself holds no units.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
With gpu.rawResourceNames set to forbidden, GPUs reach tenants only as
GPUDevices, budgeted by gpu.cozystack.io keys. A hoster sets it only
once no quota key on a GPU resource name itself is left, and tenant
validation now keeps it that way: a raw GPU key, bare or as
requests.* or limits.*, is refused, so nobody can bring the old
budget form back.
The setting and the effective GPU resource-name set are read from the
settings ConfigMap the controller writes, through the uncached client
the API already uses for quota reads, so the Deployment needs no new
flags. Without the ConfigMap raw names stay allowed. The read happens
only when the tenant declares a key that can be a resource name.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A gpu.cozystack.io/<class> or gpu.cozystack.io/total key caps the
whole subtree of the tenant that declares it and is enforced by the
GPUDevice admission and the binder, not by this gate. The tenant chart
does not render these keys into the namespace ResourceQuota, so a
tenant declaring only GPU keys gets no quota objects and draws from
its parent's pool like a tenant that declares nothing.
The declaration gate read the same values verbatim. It would carve a
child's GPU key out of its parent's and siblings' as if it were a pod
resource, and it treated a GPU-only tenant as bounded, leaving that
tenant's usage out of its parent's pool. Drop the GPU keys where the
declared quotas are parsed, so the child, the parent, the siblings
and the pool membership all see the same keys the chart renders.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The tenant declaration gate reads the parent pool's current usage from
ResourceQuota status.used under the key that carries a resource's
limit-side budget, limits.<name>. The quota controller fills limits.*
usage only for cpu, memory and ephemeral-storage. For an extended
resource, a GPU name included, and for hugepages it counts only
requests.<name>, so the limits.<name> key the chart also renders
always reports zero.
The gate therefore saw such a resource as unused and could grant a
child tenant quota the parent pool was already consuming. Read the
usage of every other resource the chart renders with a prefix under
requests.<name>, where the chart applies no ratio, so that key holds
the same declared amount, and read a key that is already a usage key
as declared.
Fixes: e3f09bab1b ("fix(apiserver): subtract parent pool usage in hierarchical tenant quota admission")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The tenant chart now renders its ResourceQuota and the default
LimitRange only for keys other than the gpu.cozystack.io/* GPU budgets,
so a tenant with only GPU budgets has no LimitRange. The comments and
the system memory limits page that explain sidecar sizing and system
defaults by that LimitRange said any declared resourceQuotas brings it.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The class and device policies render only where the platform switched
GPUs for VMs on, the usage-monitor label reservation always, and none
of them where the API server does not serve ValidatingAdmissionPolicy.
A string "false" or a null gpu block must not switch the gated ones
on.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The GPU controller writes the usage records of held devices under a
WorkloadMonitor whose selector is a reserved label key, so the monitor
must match nothing and tenant labels cannot shape GPU usage records.
A write that adds the key to any object or subresource in a tenant
namespace is refused, whoever makes it, which also covers the
virt-launcher pods that copy a VM template's labels and the
DataVolumes created from a VM's dataVolumeTemplates.
The policy renders whether or not GPUs are on, so no tenant can place
the label before the hoster turns GPUs on. It ratchets: an object that
already carries the key can still be updated, lose its finalizers and
drop the key, so nothing labelled before the policy can get stuck. It
sees every write in those namespaces, so its expression reads only
metadata and guards each step.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Whoever orders GPUDevice objects must not touch two parts of a device.
The device finalizer keeps a released device, its binding and its usage
record until no VMI uses it, so a tenant who could drop it could take a
held device out of the usage records. The template-device label is the
mark the VMI webhook is to set on devices it creates from a
deviceTemplate, and the GPU controller deletes devices so marked with
their VMI, so on a tenant's own order it would get that order deleted.
Both are refused for everyone but platform callers, matched by group
like the other cozystack-basics policies: cozy-system and kube-system
service accounts and cluster admins, plus kube-controller-manager,
which has no group of its own. The orphan and foregroundDeletion
finalizers belong to the API server and the garbage collector and stay
free.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Held devices and their usage records name their class, so while the
class carries its protection finalizer, what it sells (vendor, model,
mode, profile, memory, maxCount and the kind of source) cannot change.
The policy relies on the GPU controller setting the finalizer on every
class, so a changed offer is a new class. resourceName and nodeSelector
stay open for host-layer changes and for moving the offer. CRD rules
cannot see finalizers, so this is an admission policy, and it reads the
stored object so the write that drops the finalizer cannot change the
spec as well.
A class being deleted waits only for the devices that already hold it,
so a second policy, evaluated once per class, refuses new orders of it.
The policies render only where the platform switched GPUs for VMs on.
Their tests evaluate the expressions read from the chart, compiled for
both the current library and the oldest served Kubernetes minor.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every tenant role reads the GPUDevice objects in its namespace. No
tenant role writes them yet: nothing checks an order's budget or vGPU
entitlement until the admission webhooks exist, so the write ships
with them.
GPUClass is the hoster's published catalog and is cluster-scoped, so
tenant roles, which are bound per namespace, cannot carry its read; a
ClusterRoleBinding gives it to every authenticated user. GPUNode names
physical devices and stays out of every tenant role.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The GPU class and device admission policies in cozystack-basics apply
only where gpu-operator serves GPUs to VMs: emitted by the iaas bundle
in the default or vgpu variant. A helper reads the same bundle keys as
the iaas bundle's own gpu-operator gate to decide that.
gpu.vgpu entitles the root tenant through _cluster.gpu-vgpu, written
only when on because every release in every tenant reads that Secret.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A tenant is entitled to vGPU only when its parent is, and the root
tenant's namespace and _namespace values are rendered here rather than
by the tenant chart, so the chain has to start here. The value comes
from _cluster.gpu-vgpu, which the platform fills from its gpu.vgpu.
Like a tenant namespace, the root namespace always carries the label,
and _namespace.vgpu appears only when entitled.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A vGPU class may be ordered only by an entitled tenant, and no tenant
may grant itself the entitlement. The Tenant field gpu.vgpu is set by
the parent's administrator, so the effective value is the field and the
parent's value together, and it reaches the namespace as the
namespace.cozystack.io/vgpu label. The label is always written, so only
"true" entitles and a missing label cannot be read as consent.
_namespace.vgpu is written only when entitled, since every app in the
tenant reads that Secret and an always-present key would upgrade them
all.
gpu.cozystack.io/* budget keys are checked against GPUDevice orders,
not pods, so they are dropped before the ResourceQuota render, and a
tenant with only GPU budgets gets no ResourceQuota or LimitRange.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binder re-checks what admission checked, and cozystack-api shows
the remaining budget a later order is checked against; computing these
in three places would let them disagree. The model is pure, so each
caller builds a snapshot from its own cache.
GPU budgets cap the whole subtree of the tenant that declares them,
walked over the namespace ancestry and independent of the tenant quota
pools. A raw GPU key blocks a class until the same tenant also declares
a GPU key, and then caps devices and raw-name pods together. Pod usage
follows the scheduler's arithmetic, sidecars included, so the binder
never books a unit a raw-name pod holds.
A node takes binds only when it is schedulable, Ready and carries no
NoSchedule or NoExecute taint the pod does not tolerate; the
unschedulable, not-ready and unreachable taints are left to the first
two checks, and a taint never counts as a cordon. Gt and Lt
tolerations do not count, since the scheduler ignores them where their
feature gate is off.
Template device names are derived from the VMI and shortened with a
hash when the VMI name leaves no room, so they stay valid subdomains.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The binder, the admission webhooks, the tenant chart, the policies and
cozystack-api all have to agree on the same label, annotation,
finalizer and quota keys and on the settings ConfigMap the policies
take as a parameter, so they are defined once here.
The binder in the leader is the only writer of placement; webhooks
running in every replica record an attachment and a bind request in
the device status and wait for the binder's answer, so capacity and
budgets are decided in one serialized loop. Status is the carrier
because no tenant role can write it. The template-device mark carries
no VMI name, since a VMI name can exceed a label value.
Register and Setup exist as no-ops so the controller can be wired to
them before the webhooks and reconcilers land.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
GPUs become a catalog of hoster offers, tenant reservations against
them and a per-node inventory, so a tenant can hold a GPU while its VM
is off and an order that does not fit is refused at create time.
The CRDs enforce what the apiserver alone can: a class name is a DNS
label other than total because it becomes part of a quota key, the
source is a union with dra reserved for a later member, a device's
spec is immutable and orders at least one GPU, and bound devices are
listable per node through a field selector on status.nodeName. A
class resource name must be a valid extended resource outside
gpu.cozystack.io, where it would read as a GPU budget key. The CRDs
are kept on chart uninstall, since they ship as plain templates and
deleting GPUDevice would drop every hold and usage record.
maxCount carries no schema default: it defaults to 1 for vgpu classes
and to no limit for passthrough, which a structural default cannot
express, so readers resolve it through the shared model.
A GPUNode device reports how many device-plugin devices kubelet
counts as allocatable for it, because an mdev parent serves several
from one PCI function and one healthy entry per function would
undercount the node.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Implements the release-asset part of #4759: a release cut from a tree
that has this change gets an SBOM of its source and one keyless
signature that covers all of its assets.
These are the two CLOMonitor checks still failing for this repository,
`sbom` and `signed_releases`, after #4757 and #4899 brought the project
score to 96.69.
`hack/upload-assets.sh` now collects its asset list into an array and
hands it to the new `hack/sign-release-assets.sh` before the first
upload. The script writes three files into `_out/assets`, and they are
uploaded with the rest:
| File | What it is |
|---|---|
| `cozystack-source-sbom.spdx.json` | SPDX 2.3 SBOM made with syft, over
`git archive HEAD` (the committed release tree, without `_out/`,
`.release-tooling` or runner leftovers) and the unpacked
`cozypkg-*.tar.gz` binaries. It does **not** describe the container
images that `packages/` pins by digest (their SBOMs are #2937, A2 in
#3278) or the Talos ISO, disk, kernel and initramfs images. It is named
by what it covers, so `cozystack-sbom` stays free for an SBOM of the
release as a whole. CLOMonitor's `sbom` check matches `sbom` in an asset
name. |
| `cozystack-checksums.txt` | SHA-256 of every uploaded asset and of the
SBOM. |
| `cozystack-checksums.txt.sigstore.json` | Sigstore bundle of `cosign
sign-blob` over the checksums, keyless, signed with the job's GitHub
OIDC identity. |
Before it returns, the script runs the same `cosign verify-blob` that
`docs/release.md` gives users, with `--certificate-identity-regexp
'^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)'`:
`tags.yaml` on a `v*` tag push, `promote-rc.yaml` dispatched from
`main`. A signature that check would reject therefore fails the job
before anything is uploaded. It also refuses its own three outputs as
inputs: `promote-rc.yaml` downloads every rc asset, and a stable release
has to get freshly computed checksums and a fresh signature, since
promotion restamps several assets.
Workflows: `tags.yaml::prepare-release` and `promote-rc.yaml::promote`,
the two jobs that call `make upload_assets`, get `id-token: write` and
two setup steps:
- syft v1.54.1 from its release tarball, checked against a committed
SHA-256 (`c069905b…`, the release's checksums entry; the checksums file
verifies against anchore/syft's keyless release signature). Not
`anchore/sbom-action/download-syft`: pinned by commit, it still runs
syft's `install.sh` fetched from a tag at run time, in a job that holds
the app token, the GHCR login and `id-token: write`. v1.54.1 is past
GHSA-mw2c-m758-9v5q (fixed in v1.52.0).
`hack/ci-tool-installs-contract.bats` now fails on
`anchore/sbom-action`, on a syft install script, on a syft download
without `sha256sum -c` and on version or checksum drift between the two
workflows.
- `sigstore/cosign-installer` v4.1.2, pinned by SHA, with
`cosign-release: v3.1.3`: the version `docs/release.md` asks users for,
so the job checks its own signature the way they will. The action
verifies that binary against cosign's keyless and KMS bundles with its
digest-pinned bootstrap cosign. `hack/ci-tool-installs-contract.bats`
holds the action commit and the release equal in both workflows and
fails on a step without `cosign-release`.
`docs/release.md` gets a "Release signatures and SBOM" section with the
user-side verification: cosign v3.1.3 or later (v2.6.5 on the v2 line),
the two commands joined with `&&`, what the SBOM covers and what the
certificate identity proves. It now also says, at each place
`promote-rc.yaml` is run, that it is dispatched from `main` only.
### Decisions for the release owners
- **Fail closed.** If Fulcio or Rekor is down, the release job fails and
nothing is uploaded. I chose that so that no release goes out unsigned.
If you would rather publish unsigned than block, the change is a `||
echo ::warning::` around the call in `upload-assets.sh`.
- **Promotion runs the rc tree's script.** `promote-rc.yaml` calls `make
upload_assets` from the rc checkout, not from `.release-tooling`. The
first signed stable release is therefore the promotion of the first rc
tagged after this merges. `release-1.6` gets signatures only if this is
backported. CLOMonitor reads the latest release, so the score moves only
when that release carries the files.
- **Certificate identity pins the refs** (decided in review):
`tags.yaml@refs/tags/v…` and `promote-rc.yaml@refs/heads/main`. A
`promote-rc.yaml` dispatched from another branch fails the job's own
check at the upload step, after the staging branch, the `promotion-*`
candidate and the empty draft release exist; nothing in `parse` refuses
a non-`main` dispatch up front. Until the `v*` tag rulesets exist, a
hand-pushed `v*` tag whose tree has an edited `tags.yaml` still signs,
which `docs/release.md` says.
- **Follow-ups after the first signed release** (#4759 stays open for
them):
- `project.documentation.signature-verification` in
`security-insights.yml` (#4757);
- the SBOM row in `docs/security/self-assessment.md`, which says "Not
published at this time"; the image caveat above stays in it.
I left both alone so that they do not describe releases that don't exist
yet.
### How it was checked
- Bats 1.14.0 (the version CI installs), with gh, syft and cosign
stubbed:
- `hack/sign-release-assets.bats`, 8 tests: what the checksums cover;
that syft sees the committed tree and the cozypkg binaries but not
untracked files; that the verify identity accepts `tags.yaml` on a `v*`
tag and `promote-rc.yaml` on `main`, and rejects a fork, another
workflow, `tags.yaml.evil`, `tags.yaml` on a branch, and
`promote-rc.yaml` on `release-1.6`, on `main-evil` and on a tag; that
`docs/release.md` shows the same regexp as the script; that a failed
verification fails the script; that it refuses a missing asset, assets
split over two directories (with a same-named file in both, so the test
fails if the directory check is removed) and its own outputs as input.
- `hack/upload-assets.bats`, 4 tests, new: the exact upload list and
order (the assets, then the SBOM, the checksums and the bundle); that
the checksums name every upload but the last two; zero `gh` calls when
verification fails or an asset is missing. The old `upload-assets.sh`
fails all four, a version that signs after uploading fails the last two.
- `hack/ci-tool-installs-contract.bats`, 8 tests. Putting
`download-syft` back, dropping the `sha256sum -c` line, letting
`SYFT_VERSION` drift between the workflows, dropping `cosign-release`
from one step, letting it drift, unpinning the installer or giving one
step another installer commit each fails it.
- Putting the old any-ref regexp back in the script fails the identity
test and the docs tie; putting it back in the docs alone fails the tie.
- The meta-suites `bats-strict-setup`, `bats-no-negated-assert`,
`bats-no-exit-trap`, `bats-runner-coverage` and `bats-runner-contract`
pass: 90 tests across these eight files.
- The pinned regexp gives the same accept/reject results under `grep -E`
and Go's `regexp` (what cosign uses). cosign-installer's install script
at the pinned commit, run as written with `cosign-release: v3.1.3`,
verifies the binary's keyless and KMS bundles and installs v3.1.3 (its
SHA-256 matches `cosign_checksums.txt`). `--yes`, `--bundle`,
`--certificate-identity-regexp` and `--certificate-oidc-issuer` are
listed, not deprecated, in cosign v3.1.3's help.
- The `Set up syft` step's script, run as written with
`RUNNER_TEMP`/`GITHUB_PATH` set: the checksum passes and `syft version`
reports 1.54.1. A real syft v1.54.1 run through
`hack/sign-release-assets.sh` (cosign stubbed) over this tree and the
v1.6.5 `cozypkg-linux-amd64.tar.gz` took 10 s and wrote an SPDX 2.3
document with 920 packages: 440 golang, 373 npm, 103 github, 4 without a
purl; 60 come from the cozypkg binary.
- cosign behaviour behind the docs, with a key-signed legacy bundle
whose `cert` holds the bare public key, against the documented identity
regexp (tlog ignored, since an attacker can upload their own entry):
v2.6.4 and v3.1.2 print "Verified OK", v2.6.5 and v3.1.3 reject it.
`--new-bundle-format` makes v2.6.4 reject it but not v3.1.2, so the docs
give the minimum version and not the flag. A real keyless protobuf
bundle (syft's) verifies on v2.6.5 and v3.1.3.
- zizmor 1.26.1 `--offline` reports no findings. actionlint 1.7.12 (with
`.github/actionlint.yaml`) reports the same 13 pre-existing info-level
shellcheck notes in `tags.yaml`/`promote-rc.yaml` as before, nothing
new. shellcheck passes on both scripts. `hack/check-commit-trailers.sh`
passes.
- Not checked: a real keyless signature. It needs the OIDC token of a
job in this repository, so the first rc after the merge is the
end-to-end test. Cutting an `-alpha.N` first would exercise it without
touching an rc.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
I checked the trigger map file by file. The only entry
`hack/upload-assets.sh` can trigger is renaming a release asset, because
the website install pages hardcode asset names. This PR adds three
assets and renames none of the released ones, `cozypkg-checksums.txt`
included (the SBOM was renamed to `cozystack-source-sbom.spdx.json` in
review, before any release carried it). The workflow and docs changes
match no entry.
### Release note
```release-note
feat(release): releases carry cozystack-source-sbom.spdx.json, an SPDX SBOM of the source tree and the cozypkg binaries (not of the container or Talos images), cozystack-checksums.txt with the SHA-256 of each asset, and cozystack-checksums.txt.sigstore.json, a keyless Sigstore signature over the checksums made by the release workflow; docs/release.md shows how to verify a download with cosign v3.1.3 or later.
```
Refs #4759
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Releases now include an SPDX software bill of materials, SHA-256
checksums, and a keyless Sigstore signature bundle.
* Release assets are verified against the expected workflow identity
before upload; uploads stop if verification fails.
* **Documentation**
* Added guidance on release signatures, SBOM coverage, asset
verification, and which branch dispatches release workflows.
* **Tests**
* Added coverage for generated release files, upload behavior, and
signing and verification failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
No release carried a signature or an SBOM, so users could verify none of the ISO, disk images, kernel, initramfs, manifests and cozypkg binaries they download, and the OpenSSF Scorecard Signed-Releases check and CLOMonitor's signed_releases and sbom checks failed.
hack/upload-assets.sh now hands its asset list to the new hack/sign-release-assets.sh before the first upload. That script writes cozystack-source-sbom.spdx.json with syft over the committed tree and the cozypkg binaries, cozystack-checksums.txt with the SHA-256 of every asset and the SBOM, and cozystack-checksums.txt.sigstore.json with a keyless cosign signature over the checksums. The SBOM does not describe the container images or the Talos boot images, which is what its name says; image SBOMs are #2937. The script verifies the bundle against the identities of tags.yaml on a v* tag and promote-rc.yaml on main before returning, so a bad signature fails the job with nothing uploaded, and hack/upload-assets.bats pins the upload list and that order.
The two jobs that call make upload_assets get id-token: write, cosign v3.1.3 from the SHA-pinned cosign-installer, and syft from its release tarball checked against a committed SHA-256, because anchore/sbom-action/download-syft runs an install script fetched at run time; hack/ci-tool-installs-contract.bats now covers both. docs/release.md describes the files, what the SBOM covers, the minimum cosign version to verify a download with, what the certificate identity proves, and that promote-rc.yaml is dispatched from main only.
Refs #4759
Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Clusters with a memory limit below 512Mi and no explicit shared_buffers get the new default on upgrade, which needs a PostgreSQL restart. The changelog described the fix but not the restart.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
## What this PR does
Upgrade from v1.6.x to v1.7 leaves `backupstrategy-controller`
HelmRelease in `UpgradeFailed`: `no matches for kind "Kafka" in version
"strategy.backups.cozystack.io/v1alpha1"`, same for Rabbitmq and Redis,
and none of the new strategy CRDs get created.
CRDs of this chart are ordinary templates, and Helm maps every object of
a release through the RESTMapper before applying any of them. Default
Strategy CRs for kafka, rabbitmq and redis were gated only on the bucket
name, so they render next to their not yet served CRDs, Build() fails,
nothing is applied and every retry renders the same manifest. Same
happens on a first install with external S3 (`provisionBucket: false`)
for all older default strategies, bucket name comes from values there.
MongoDB default strategy already has the `crdEstablished` gate, this PR
adds it to all other default strategies. On an installed cluster nothing
changes. Otherwise first revision creates the CRDs and skips the CRs,
then DefaultObjectsGate forces the follow-up upgrade that renders them.
CRDs stay templates so later CRD changes still reach the cluster on
upgrade. Short explanation is added to
`docs/operations/backup-classes.md`.
To keep this from coming back there is
`hack/backupstrategy-crd-gate.bats`: it renders the chart with the
bucket gate open and no CRD served and fails on any Strategy CR from any
template, so a new `strategy-*-default.yaml` without the gate goes red
without anyone updating a list. It also checks that every CR gates on
the CRD this chart ships for its kind. helm-unittest covers both halves
of the gate for all 11 templates.
Checked on a throwaway kind cluster:
- v1.6.4 installed, then upgraded: current main fails with the error
above, this branch passes with all 11 CRDs Established and the follow-up
upgrade creates the rest
- modified CRD is applied on a later upgrade
- fresh install converges, and fresh install with external S3 converges
here while main fails with "no matches for kind" for eight kinds
Fixes#4925. Needs backport to release-1.7, both commits apply there
cleanly.
### Downstream repositories
- [x] No downstream repository is affected by this change
### Release note
```release-note
fix(backups): backupstrategy-controller release no longer gets stuck in UpgradeFailed on upgrade from v1.6 and no longer fails a fresh install with external S3, strategy CRDs are created first and default strategies follow on the next reconcile
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Default backup strategies are now created only after their required
resource types are available and the backup bucket is resolved. This
helps prevent installation and upgrade issues, including when using
externally managed S3 storage.
* **Documentation**
* Clarified when default backup strategies are created during
installation and upgrades.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The CRD gate has now been missed twice: MongoDB shipped without it, and
Kafka, RabbitMQ and Redis followed after it existed. helm-unittest only
checks the templates a case lists, so a new strategy-*-default.yaml
that forgets the gate stays green until an upgrade wedges.
Render the chart with the bucket gate open and no CRD served, and fail
on any strategy.backups.cozystack.io object from any template. Also
check that every rendered Strategy CR gates on the CRD this chart ships
for its kind, because a gate naming the wrong CRD passes the first
check and keeps the CR out of every live render.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Aleksei Sviridkin