mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 02:27:01 +09:00
test/gpudevice-e2e
7636
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
e894a26fcf |
test(e2e): add a GPUClaim suite on a fake extended resource
Nothing exercised GPUClaim end to end: ordering and binding, budget and capacity refusals at admission, attachment to a VMI with the launcher pod released from its scheduling gate onto the claim's node, the usage record the claim owns, a template claim that waits for its tenant's budget with its pod held at the gate and starts once the budget frees, deletion of that claim with its VMI, and deletion of a tenant still holding a claim. CI has no GPUs, so the suite advertises two units of a fake extended resource on one node and restores the node at cleanup. No device plugin serves the fake name, so virt-handler never hands the guest a device and the VMI does not sync. The VM checks stop at what admission, the controller and the scheduler decide; the guest side stays manual. The suite creates VMs on a VMDisk, so both VM application sources and gpu-operator select it, and its install closure keeps gpu-operator. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
5a9c19ce20 |
test(e2e): enable gpu-operator in the e2e platform install
The GPU paths of cozystack-controller (the GPUClaim, VMI and pod webhooks, the launcher scheduling gate, the GPU policies) render only when gpu-operator serves GPUs to VMs, and the e2e install never enabled it, so no lane could reach them. Enable it whenever the selection keeps the package, and permit a fake device name in KubeVirt as an external provider: a VM using a class on a name outside the table is refused at admission, and CI has no GPUs to advertise a real one. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
57f709de9a |
docs(gpu): list the GPU budget keys in the tenant quota reference
A tenant sets its GPU budgets through resourceQuotas, but the tenant chart's quota reference did not list the keys, did not say that a child's key never lifts an ancestor's, and did not say that a tenant with only GPU budgets gets no ResourceQuota or LimitRange. The memory-limits page said the tenant LimitRange renders when "resourceQuotas other than the GPU budgets is configured"; it renders when resourceQuotas carries any key that is not a GPU budget, which the sentence now says. The barman sidecar comment is reflowed where that edit left a ragged line. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
3cee3d70ef |
feat(api): add gpuclass and gpuclaim option sources
The dashboard orders GPUs by class and attaches them by claim name, so it needs the catalog and the tenant's own claims without reading GPUClass or GPUClaim objects itself. The gpuclass source lists every class that still takes orders, a deleted one excluded, with its display name, a short description of the offer, the free units and the most free units on one node from the class status, the per-VM maximum, and the requesting tenant's remaining budget. That budget comes from the shared model over the tenant and each ancestor's subtree, from their declared resourceQuotas and every claim, and is 0 for a vGPU class without the entitlement. Pods are read only when an ancestor limits a class's resource name with a raw key, one list per namespace of the outermost such ancestor's subtree, and a virt-launcher pod does not count the units its attached claims already cover. No field names a node. The gpuclaim source lists the claims in the request namespace that a VM can name, leaving out claims made from a claimTemplate. cozystack-api gets read access to both kinds, get and list on pods for the raw-key case, and get on the VMI a counted pod names. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
ea233c5ed7 |
feat(api): hide raw GPU options when raw names are forbidden
Once the hoster sets gpu.rawResourceNames to forbidden, a VM gets GPUs only through claim references, and admission refuses a GPU entry anywhere else. The gpu source would still offer raw resource names, and the instancetype source would still offer instance types such as the gn1 family that bring their own GPU entries, so the form would lead to a VM that admission then refuses. Read the setting and the GPU resource-name set from the controller's settings ConfigMap. Under forbidden the gpu source is empty, and an instance type with any gpus entry, or a hostDevices entry on a GPU resource name, is left out; a NIC passed as a host device keeps its instance type listed. An unreadable setting serves the full lists: the pickers are advisory and admission enforces the policy. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
714231d2fe |
feat(api): add GPU order bounds to OptionItem
A GPU class picker needs more than a name: the portal offers the smallest of the class's most free units on one node, its per-VM maximum and the tenant's remaining budget as the largest order, and shows the class's free units. Carry them as optional integers on the option item, so an absent field keeps meaning that nothing limits the order there and every other option source stays unchanged on the wire. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
c84768a4c6 |
feat(vm-instance): attach GPUClaims through claim and claimTemplate
A gpus entry now names a raw resource, a held GPUClaim or a claimTemplate that orders a claim for as long as the VM runs. The chart renders references only as the claim-reference label and the claims annotation on the VMI template; the GPU webhook resolves them at VMI creation and writes the gpus entries itself, so a list holds raw names or references, never both. Where gpu-operator does not serve GPUs to VMs no webhook would resolve a reference and the VM would boot without its GPUs, so the platform publishes the switch in _cluster and the chart refuses references without it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
67fc342e79 |
feat(cozystack-basics): let tenant admins order and release GPUClaims
Tenant admins and above can now create, change and delete GPUClaim objects in their namespaces where GPUs are on. The order webhook judges each order against its class, capacity, budgets and the vGPU entitlement, and the claim tamper policy keeps tenants off a claim's finalizers and template mark, so the write is granted only where that policy renders. Status, which carries the binding and the attachment, stays the controller's. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
352f726196 |
feat(platform): thread the GPU switch and settings to the controller
cozystack-controller runs on every cluster, and its GPU paths watch KubeVirt objects, so the chart's gpu.enabled is set only where the GPU admission policies apply: where gpu-operator serves GPUs to VMs. monitoring-agents scrapes the metrics those paths serve under the same condition. gpu.resourceNames and gpu.rawResourceNames reach the controller, which publishes them in its settings ConfigMap for the policies and cozystack-api. rawResourceNames also reaches cozystack-basics, whose raw-name policy renders only under forbidden. Both are validated on every render, so a typo fails before anyone turns GPUs on. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
3256155926 |
feat(cozystack-basics): enforce gpu.rawResourceNames forbidden
With raw names forbidden a GPU reaches a VM only through a GPUClaim reference, which makes a held claim a hard guarantee. A VMI carrying a GPU entry (any gpus entry, a hostDevices entry or a resource of a GPU resource name) without the claim-reference label is refused, which covers a VM written by a tenant super-admin, a VMIReplicaSet and a worker pool. A pod requesting a GPU resource name is refused unless kubevirt-controller creates it with a VMI owner, as it does for virt-launcher and migration target pods. The GPU resource names come from the settings ConfigMap the controller publishes. The binding fails closed without it, and since the parameter is resolved before any expression runs, the bindings select tenant namespaces only; anything wider would block every pod in the cluster, the controller's own included, until the ConfigMap exists. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
1b4c021ebe |
feat(monitoring-agents): scrape the GPU controller's metrics
The class capacity, bound units and Lost count are kept out of the world-readable GPUClass status and exported as cozystack-controller metrics instead, so platform monitoring has to scrape them. The metrics port is authenticated, so vmagent also needs the right to GET /metrics. Both render only with GPU enabled, as the port does. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d0aa92af63 |
feat(cozystack-controller): serve the GPU webhooks with GPU enabled
With gpu.enabled the chart passes the GPU flags, serves the three GPU admission webhooks with a cert-manager certificate, and exposes the controller's metrics over HTTPS, where each scrape is authorised through the token and access reviews the controller is granted. The webhooks fail closed, so the controller runs two replicas spread over nodes, under a disruption budget, and a replica is ready only once the caches the webhooks read have synced. The pod webhook selects virt-launcher pods with claim references and leaves migration targets out, which KubeVirt deletes when they stay pending. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7f6d6ae9a4 |
feat(gpu): serve the GPUClaim, VMI and virt-launcher pod webhooks
The VMI webhook resolves a VMI's claim references, refuses what needs no placement to judge, records the attachment of each held claim and writes one gpus entry per unit with the class's resource name. It binds, moves and creates nothing: budgets, capacity and template claims are the controller's, so a VM whose set does not fit yet waits at start instead of being refused into KubeVirt's backoff. The pod webhook holds the launcher pod, and the temporary pod for WaitForFirstConsumer volumes, at the claims' scheduling gate and adds the tolerations of the claims' classes, which binding already counted on. A repeated call adds neither twice, since a duplicate gate fails pod validation. Migration targets are left alone, because KubeVirt deletes a target pod that stays pending. The order webhook refuses a tenant's GPUClaim that cannot be met now: class, maxCount, published free units, budgets and the vGPU entitlement. Template claims skip it; the tamper policy keeps their mark out of tenants' hands. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
ca0c8944b2 |
refactor(gpu): move the KubeVirt table, quota and coverage readers to the model
The admission webhooks and the option sources need the same readings the controller makes: the permittedHostDevices table KubeVirt checks GPU entries against and the table's GPU names, a tenant's declared resourceQuotas, and the units a VMI's attached claims cover for budgets. They move into the model unchanged, and the controller reads them from there, so each has one copy. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
ebca80fe38 |
feat(gpu-node-agent): restrict the agent to its own node's GPUNode
RBAC cannot name the caller's node, so the agent's role lets every agent write every GPUNode, and a compromised node could rewrite the inventory, and through resourceNames the cluster's GPU resource-name set, of any other node. The pod-bound service account token carries the node name as a user extra, which a ValidatingAdmissionPolicy can compare with the object name. A token without that extra is refused rather than let through: the policy matches only the agent's service account, so failing closed stops the agent and nobody else. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9d34f69e96 |
feat(platform): run the GPU node agent with gpu-operator
GPUNode inventory is needed exactly where GPUs are served, so the agent ships as a component of every gpu-operator variant, in the cozy-gpu-operator namespace. It is marked privileged because its hostPath mounts are refused below the privileged Pod Security level, and the operator labels a namespace privileged only for privileged components. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
c28f09f9b5 |
feat(gpu-node-agent): add the agent command, image and chart
The inventory has to run next to kubelet: a DaemonSet reading the host's sysfs and kubelet's pod-resources socket, both mounted read-only. Connecting to the socket needs root, so the agent runs as root but with no capabilities and no privilege escalation. The manager's cached client would watch every pod in the cluster, so the agent uses a direct client and lists only its own node's pods. RBAC cannot name the caller's node, so the role is cluster-wide; it holds only the verbs the agent uses, and the DaemonSet selects the nodes node-feature-discovery marks as carrying an NVIDIA device, so the token does not sit on nodes with nothing to report. The image is in the root build list so that every build path stamps its reference; the committed tag is replaced by the first build. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
5b7ae8001e |
feat(gpu): add the GPU node agent inventory
GPUNode is the hoster's per-node view of GPUs and the binder's source of device health, so something on each node has to report it. The agent reads the GPU functions from sysfs and what kubelet's device plugins make of them from the pod-resources API. Plugins report devices by PCI address, mdev UUID or IOMMU group, so each ID is resolved through sysfs by its shape rather than by knowing the plugin. A group counts only through functions bound to vfio, so the plain indexes NVIDIA's container plugin may report do not read as groups. GPUs of NVIDIA's container plugin, reported by NVML UUID, cannot be placed through sysfs, but their resource name still joins resourceNames, since pods request GPUs by it. pod-resources carries no health; kubelet lists only healthy devices as allocatable, which is what the agent reads health from, and it counts them per function, because an mdev parent serves several. A device named by its IOMMU group counts once, on the group's lowest-addressed GPU, however many GPUs the group holds. Pods in a final phase are skipped because kubelet 1.32 and 1.33 still list them with devices a new pod may hold. The binder writes conditions into the same status, so the agent only merge-patches its own fields, and it writes no GPUNode but its own node's. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
eb3508a7dd |
feat(cozystack-controller): grant the GPU controller its writes
The binder writes finalizers and status on GPU classes, claims and nodes, creates and deletes the template claims of a VMI's claimTemplate entries, updates a gated launcher pod to pin and release it, deletes a pinned launcher pod its node refuses, and deletes the unscheduled VMI of a claim that must leave its node once that VMI's pod is pinned. RBAC cannot narrow the pod rights to launcher pods. It reports on a VMI, through events.k8s.io, why its pod stays gated. The settings ConfigMap is the only ConfigMap the controller writes, so its rights live in a Role in cozy-system rather than the ClusterRole: get, update and patch are scoped to that object by name, and create, which RBAC cannot scope by name, is scoped by the namespace. With --metrics-secure the metrics endpoint authenticates and authorizes each scrape through TokenReview and SubjectAccessReview, so the controller needs create on both for the GPU metrics to be scraped. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d07dcb0b78 |
feat(cozystack-controller): wire the GPU controller and its flags
The GPU reconciler always runs, so GPU finalizers and usage records are handled on every cluster, including ones that later turn GPU off; --enable-gpu adds placement and the KubeVirt watches. GPU admission webhooks are registered only with it, because asking the manager for its webhook server starts it and the server needs a serving certificate. The hoster's resource-name list and raw-name policy come in as flags and are written to the settings ConfigMap for the admission policies to read. GPU admission has to answer from warm caches, so with GPU enabled every replica starts the informers it needs before the manager rather than at the first admission, and readiness waits for exactly those to sync. It does not wait for KubeVirt, whose kinds a fresh install serves only after this controller is ready; those informers start once the kinds are served. The leader releases its lease on shutdown so a rollout does not leave starting VMs gated for a lease duration. The ConfigMap cache holds only the settings object. Non-positive grace periods are refused, since a zero attachment grace would clear every attachment before its VMI appears. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
47c8288786 |
feat(cozystack-controller): stop counting claim GPUs on launcher pods
Each GPUClaim has a usage record of its own, so the virt-launcher pod of a VMI with claim references would report the same GPUs a second time. The units the VMI's attached claims hold are subtracted after the resources annotation is merged, since that annotation may name the GPU too; names the claims do not cover, such as a NIC next to the GPU, stay reported. Only a pod the VMI lists in status.activePods is covered, since whoever creates a pod sets its owner reference. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4225be12cc |
feat(gpu): bind, move and release GPU claims in the leader
The binder is the only writer of a claim's placement, so budget and capacity checks that admission already ran cannot be raced past by two replicas. Every event collapses into one pass over all GPU objects, which serializes the decisions without per-object locks, and claims are read past the cache so a unit booked in the last pass is never booked again from a stale view. A VM starts through its launcher pod, which the pod webhook holds at a scheduling gate. The pass creates the VMI's template claims, gathers the whole set onto one node that every class of the set tolerates and has room on, then pins the pod there and removes the gate in one update, since the API server lets a pod's node affinity change only while the pod is gated. A set that cannot be placed keeps its pod gated with a condition on its claims and a Warning event on the VMI, once per change of reason, so the VM waits for capacity instead of being refused into KubeVirt's backoff. The set goes only to a node the pod's own node selector and required affinity accept and, when the VMI has a volume bound to a node's PersistentVolume, that volume reaches. Gated pods go before moves and new orders. A pin cannot be undone, so a pinned pod its node refuses for lack of resources is deleted and the node is skipped for that set for ten minutes, or until its allocatable, taints or readiness change. The template claims of the set go with its failed VMI, so the controller also keeps the rejection in memory for the VM's next VMI. Only a VMI whose pod is already pinned is deleted to free a claim whose node cannot keep it; a VMI whose pod is gated has its set gathered instead. The full pass also moves held sets off nodes that cannot keep them, rebinds Lost sets before new orders, and binds new orders onto the fullest node with room. Finalizers and usage records run with the feature disabled too, so turning GPU off never hangs a namespace or class deletion. Each usage record is owned by its claim and sits under no WorkloadMonitor, so no tenant label can add or drop one, and garbage collection removes it with the claim. Without a monitor label the Workload controller would take such a record for an orphan, so it now leaves a record owned by a GPUClaim to that owner. The claims one VM names move, go Lost and are rebound as a set. Every pass restores that invariant before it places anything, gathering a set left partly Lost, by a write that failed half way or by a VMI that ended next to held members, onto one node or sending all of it to Lost, so each later step only ever changes whole sets. A Lost claim already counts against its tenant's budgets, so the pod of its still running VMI is not counted again under a raw GPU key; for node capacity it is, since the claim itself holds no units. A pod is a VMI's launcher only when the VMI lists it in status.activePods, because whoever creates a pod sets its owner reference. The KubeVirt watches start once KubeVirt serves its kinds, whose CRDs a fresh install adds after this controller starts. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
a883541176 |
feat(api): refuse raw GPU quota keys when raw names are forbidden
With gpu.rawResourceNames set to forbidden, GPUs reach tenants only as GPUClaims, budgeted by gpu.cozystack.io keys. A hoster sets it only once no quota key on a GPU resource name itself is left, and tenant validation now keeps it that way: a raw GPU key, bare or as requests.* or limits.*, is refused, so nobody can bring the old budget form back. The setting and the effective GPU resource-name set are read from the settings ConfigMap the controller writes, through the uncached client the API already uses for quota reads, so the Deployment needs no new flags. Without the ConfigMap raw names stay allowed. The read happens only when the tenant declares a key that can be a resource name. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d760d9ee68 |
feat(api): keep GPU budget keys out of the tenant quota gate
A gpu.cozystack.io/<class> or gpu.cozystack.io/total key caps the whole subtree of the tenant that declares it and is enforced by the GPUClaim admission and the GPU controller, not by this gate. The tenant chart does not render these keys into the namespace ResourceQuota, so a tenant declaring only GPU keys gets no quota objects and draws from its parent's pool like a tenant that declares nothing. The declaration gate read the same values verbatim. It would carve a child's GPU key out of its parent's and siblings' as if it were a pod resource, and it treated a GPU-only tenant as bounded, leaving that tenant's usage out of its parent's pool. Drop the GPU keys where the declared quotas are parsed, so the child, the parent, the siblings and the pool membership all see the same keys the chart renders. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
34e2c3d17d |
fix(api): read pool usage under the quota key the controller counts
The tenant declaration gate reads the parent pool's current usage from
ResourceQuota status.used under the key that carries a resource's
limit-side budget, limits.<name>. The quota controller fills limits.*
usage only for cpu, memory and ephemeral-storage. For an extended
resource, a GPU name included, and for hugepages it counts only
requests.<name>, so the limits.<name> key the chart also renders
always reports zero.
The gate therefore saw such a resource as unused and could grant a
child tenant quota the parent pool was already consuming. Read the
usage of every other resource the chart renders with a prefix under
requests.<name>, where the chart applies no ratio, so that key holds
the same declared amount, and read a key that is already a usage key
as declared.
Fixes:
|
||
|
Aleksei Sviridkin
|
9b9729ca18 |
docs(tenant): say which resourceQuotas bring the tenant LimitRange
The tenant chart now renders its ResourceQuota and the default LimitRange only for keys other than the gpu.cozystack.io/* GPU budgets, so a tenant with only GPU budgets has no LimitRange. The comments and the system memory limits page that explain sidecar sizing and system defaults by that LimitRange said any declared resourceQuotas brings it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
70d4891a7b |
test(cozystack-basics): pin when the GPU admission policies render
The class and claim policies render only where the platform switched GPUs for VMs on, and not where the API server does not serve ValidatingAdmissionPolicy. A string "false" or a null gpu block must not switch them on. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f6a79815a4 |
feat(cozystack-basics): keep tenants off GPUClaim finalizers and mark
Whoever orders GPUClaim objects must not touch two parts of a claim. The claim finalizer keeps a released claim, its binding and its usage record until no VMI uses it, so a tenant who could drop it could take a held claim out of the usage records. The template-claim label is the mark the GPU controller sets on claims it creates from a claimTemplate, and it deletes claims so marked with their VMI, so on a tenant's own order it would get that order deleted. Both are refused for everyone but platform callers, matched by group like the other cozystack-basics policies: cozy-system and kube-system service accounts and cluster admins, plus kube-controller-manager, which has no group of its own. The orphan and foregroundDeletion finalizers belong to the API server and the garbage collector and stay free. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
efe038023d |
feat(cozystack-basics): freeze a held GPUClass, refuse deleted orders
Held claims and their usage records name their class, so while the class carries its protection finalizer, what it sells (vendor, model, profile, memory, maxCount and the kind of source) cannot change. The policy relies on the GPU controller setting the finalizer on every class, so a changed offer is a new class. resourceName, nodeSelector and tolerations stay open for host-layer changes and for moving the offer, and mode is left to the CRD, which keeps it fixed whether or not the class is held. CRD rules cannot see finalizers, so this is an admission policy, and it reads the stored object so the write that drops the finalizer cannot change the spec as well. A class being deleted waits only for the claims that already hold it, so a second policy, evaluated once per class, refuses new orders of it. The policies render only where the platform switched GPUs for VMs on. Their tests evaluate the expressions read from the chart, compiled for both the current library and the oldest served Kubernetes minor. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
b71648c753 |
feat(cozystack-basics): let tenants read GPUClaim, everyone GPUClass
Every tenant role reads the GPUClaim objects in its namespace. No tenant role writes them yet: nothing checks an order's budget or vGPU entitlement until the admission webhooks exist, so the write ships with them. GPUClass is the hoster's published catalog and is cluster-scoped, so tenant roles, which are bound per namespace, cannot carry its read; a ClusterRoleBinding gives it to every authenticated user. GPUNode names physical devices and stays out of every tenant role. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
fc5f9ecce9 |
feat(platform): gate the GPU policies on gpu-operator, add gpu.vgpu
The GPU class and claim admission policies in cozystack-basics apply only where gpu-operator serves GPUs to VMs: emitted by the iaas bundle in the default or vgpu variant. A helper reads the same bundle keys as the iaas bundle's own gpu-operator gate to decide that. gpu.vgpu entitles the root tenant through _cluster.gpu-vgpu, written only when on because every release in every tenant reads that Secret. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9211bba23e |
feat(cozystack-basics): take the root tenant vGPU flag from the platform
A tenant is entitled to vGPU only when its parent is, and the root tenant's namespace and _namespace values are rendered here rather than by the tenant chart, so the chain has to start here. The value comes from _cluster.gpu-vgpu, which the platform fills from its gpu.vgpu. Like a tenant namespace, the root namespace always carries the label, and _namespace.vgpu appears only when entitled. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
52a37f0fa8 |
feat(tenant): add the vGPU entitlement, keep GPU budgets out of quotas
A vGPU class may be ordered only by an entitled tenant, and no tenant may grant itself the entitlement. The Tenant field gpu.vgpu is set by the parent's administrator, so the effective value is the field and the parent's value together, and it reaches the namespace as the namespace.cozystack.io/vgpu label. The label is always written, so only "true" entitles and a missing label cannot be read as consent. _namespace.vgpu is written only when entitled, since every app in the tenant reads that Secret and an always-present key would upgrade them all. gpu.cozystack.io/* budget keys are checked against GPUClaim orders, not pods, so they are dropped before the ResourceQuota render, and a tenant with only GPU budgets gets no ResourceQuota or LimitRange. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
46ff481549 |
feat(gpu): add the shared GPU placement and budget model
The binder re-checks what admission checked, and cozystack-api shows the remaining budget a later order is checked against; computing these in three places would let them disagree. The model is pure, so each caller builds a snapshot from its own cache. GPU budgets cap the whole subtree of the tenant that declares them, walked over the namespace ancestry and independent of the tenant quota pools. A raw GPU key blocks a class until the same tenant also declares a GPU key, and then caps claims and raw-name pods together. Pod usage follows the scheduler's arithmetic, sidecars included, so the binder never books a unit a raw-name pod holds. A node serves a class only when it is schedulable, Ready and carries no NoSchedule or NoExecute taint the class does not tolerate. The tolerations are the class's, because a claim is bound before the pod that will use it exists. The unschedulable, not-ready and unreachable taints are left to the first two checks, and a taint never counts as a cordon. Gt and Lt tolerations do not count, since the scheduler ignores them where their feature gate is off. Template claim names are derived from the VMI and shortened with a hash when the VMI name leaves no room, so they stay valid subdomains. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
46eb90a74d |
feat(gpu): define the shared GPU keys, settings and start contract
The controller, the admission webhooks, the tenant chart, the policies and cozystack-api all have to agree on the same label, annotation, finalizer, scheduling gate and quota keys and on the settings ConfigMap the policies take as a parameter, so they are defined once here. The VMI webhook only checks a start and records it: the attachments of the held claims, and the gpus entries, because KubeVirt renders the launcher pod's requests from a VMI spec that cannot change later. Placement happens in the controller while the launcher pod waits at a scheduling gate, so no admission call waits on the binder, and a set that does not fit leaves the VM starting instead of refusing it into KubeVirt's start backoff. The pin and the ungating are one pod update, because the API server lets a pod's node affinity gain requirements only while the pod being updated is still gated. A usage record is owned by its claim through an ownerReference, so no WorkloadMonitor and no reserved label have to guard it. The template-claim mark carries no VMI name, since a VMI name can exceed a label value. Register and Setup exist as no-ops so the controller can be wired to them before the webhooks and reconcilers land. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
8f4036c0da |
feat(api): add gpu.cozystack.io GPUClass, GPUClaim and GPUNode
GPUs become a catalog of hoster offers, tenant claims against them and a per-node inventory, so a tenant can hold a GPU while its VM is off and an order that does not fit is refused at create time. The CRDs enforce what the apiserver alone can: a class name is a DNS label other than total because it becomes part of a quota key, the source is a union with dra reserved for a later member, a class's mode never changes because a pool that names the class takes its worker template from it, a claim's spec is immutable and orders at least one GPU, and bound claims are listable per node through a field selector on status.nodeName. A class resource name must have the shape of an extended resource name and lie outside gpu.cozystack.io, where it would read as a GPU budget key. The CRDs are kept on chart uninstall, since they ship as plain templates and deleting GPUClaim would drop every hold and usage record. A class carries the tolerations of its VMs. A claim is bound when it is ordered, before any pod exists, so a dedicated taint on GPU nodes can only be checked against the class; without them such a node would serve no class at all. The tolerations are copied onto launcher pods, so the class takes only what pod validation accepts on every cluster: Exists and Equal, since Gt and Lt are refused where their feature gate is off, and the key, value, effect and tolerationSeconds rules of a pod toleration. They are a type of their own rather than the core one, because the core type carries no schema to put length limits on, and without them the checks would not fit the CEL cost budget. A claim records the nodes that refused its set's pinned launcher pod, each with an expiry, in status, so a leader change does not forget them and the controller does not pin the set to the same node again on its next pass. maxCount carries no schema default: it defaults to 1 for vgpu classes and to no limit for passthrough, which a structural default cannot express, so readers resolve it through the shared model. A GPUNode device reports how many device-plugin devices kubelet counts as allocatable for it, because an mdev parent serves several from one PCI function and one healthy entry per function would undercount the node. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
myasnikovdaniil
|
d1d4e92acb |
feat(release): sign the release assets and attach a source SBOM (#4861)
## What this PR does Implements the release-asset part of #4759: a release cut from a tree that has this change gets an SBOM of its source and one keyless signature that covers all of its assets. These are the two CLOMonitor checks still failing for this repository, `sbom` and `signed_releases`, after #4757 and #4899 brought the project score to 96.69. `hack/upload-assets.sh` now collects its asset list into an array and hands it to the new `hack/sign-release-assets.sh` before the first upload. The script writes three files into `_out/assets`, and they are uploaded with the rest: | File | What it is | |---|---| | `cozystack-source-sbom.spdx.json` | SPDX 2.3 SBOM made with syft, over `git archive HEAD` (the committed release tree, without `_out/`, `.release-tooling` or runner leftovers) and the unpacked `cozypkg-*.tar.gz` binaries. It does **not** describe the container images that `packages/` pins by digest (their SBOMs are #2937, A2 in #3278) or the Talos ISO, disk, kernel and initramfs images. It is named by what it covers, so `cozystack-sbom` stays free for an SBOM of the release as a whole. CLOMonitor's `sbom` check matches `sbom` in an asset name. | | `cozystack-checksums.txt` | SHA-256 of every uploaded asset and of the SBOM. | | `cozystack-checksums.txt.sigstore.json` | Sigstore bundle of `cosign sign-blob` over the checksums, keyless, signed with the job's GitHub OIDC identity. | Before it returns, the script runs the same `cosign verify-blob` that `docs/release.md` gives users, with `--certificate-identity-regexp '^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)'`: `tags.yaml` on a `v*` tag push, `promote-rc.yaml` dispatched from `main`. A signature that check would reject therefore fails the job before anything is uploaded. It also refuses its own three outputs as inputs: `promote-rc.yaml` downloads every rc asset, and a stable release has to get freshly computed checksums and a fresh signature, since promotion restamps several assets. Workflows: `tags.yaml::prepare-release` and `promote-rc.yaml::promote`, the two jobs that call `make upload_assets`, get `id-token: write` and two setup steps: - syft v1.54.1 from its release tarball, checked against a committed SHA-256 (`c069905b…`, the release's checksums entry; the checksums file verifies against anchore/syft's keyless release signature). Not `anchore/sbom-action/download-syft`: pinned by commit, it still runs syft's `install.sh` fetched from a tag at run time, in a job that holds the app token, the GHCR login and `id-token: write`. v1.54.1 is past GHSA-mw2c-m758-9v5q (fixed in v1.52.0). `hack/ci-tool-installs-contract.bats` now fails on `anchore/sbom-action`, on a syft install script, on a syft download without `sha256sum -c` and on version or checksum drift between the two workflows. - `sigstore/cosign-installer` v4.1.2, pinned by SHA, with `cosign-release: v3.1.3`: the version `docs/release.md` asks users for, so the job checks its own signature the way they will. The action verifies that binary against cosign's keyless and KMS bundles with its digest-pinned bootstrap cosign. `hack/ci-tool-installs-contract.bats` holds the action commit and the release equal in both workflows and fails on a step without `cosign-release`. `docs/release.md` gets a "Release signatures and SBOM" section with the user-side verification: cosign v3.1.3 or later (v2.6.5 on the v2 line), the two commands joined with `&&`, what the SBOM covers and what the certificate identity proves. It now also says, at each place `promote-rc.yaml` is run, that it is dispatched from `main` only. ### Decisions for the release owners - **Fail closed.** If Fulcio or Rekor is down, the release job fails and nothing is uploaded. I chose that so that no release goes out unsigned. If you would rather publish unsigned than block, the change is a `|| echo ::warning::` around the call in `upload-assets.sh`. - **Promotion runs the rc tree's script.** `promote-rc.yaml` calls `make upload_assets` from the rc checkout, not from `.release-tooling`. The first signed stable release is therefore the promotion of the first rc tagged after this merges. `release-1.6` gets signatures only if this is backported. CLOMonitor reads the latest release, so the score moves only when that release carries the files. - **Certificate identity pins the refs** (decided in review): `tags.yaml@refs/tags/v…` and `promote-rc.yaml@refs/heads/main`. A `promote-rc.yaml` dispatched from another branch fails the job's own check at the upload step, after the staging branch, the `promotion-*` candidate and the empty draft release exist; nothing in `parse` refuses a non-`main` dispatch up front. Until the `v*` tag rulesets exist, a hand-pushed `v*` tag whose tree has an edited `tags.yaml` still signs, which `docs/release.md` says. - **Follow-ups after the first signed release** (#4759 stays open for them): - `project.documentation.signature-verification` in `security-insights.yml` (#4757); - the SBOM row in `docs/security/self-assessment.md`, which says "Not published at this time"; the image caveat above stays in it. I left both alone so that they do not describe releases that don't exist yet. ### How it was checked - Bats 1.14.0 (the version CI installs), with gh, syft and cosign stubbed: - `hack/sign-release-assets.bats`, 8 tests: what the checksums cover; that syft sees the committed tree and the cozypkg binaries but not untracked files; that the verify identity accepts `tags.yaml` on a `v*` tag and `promote-rc.yaml` on `main`, and rejects a fork, another workflow, `tags.yaml.evil`, `tags.yaml` on a branch, and `promote-rc.yaml` on `release-1.6`, on `main-evil` and on a tag; that `docs/release.md` shows the same regexp as the script; that a failed verification fails the script; that it refuses a missing asset, assets split over two directories (with a same-named file in both, so the test fails if the directory check is removed) and its own outputs as input. - `hack/upload-assets.bats`, 4 tests, new: the exact upload list and order (the assets, then the SBOM, the checksums and the bundle); that the checksums name every upload but the last two; zero `gh` calls when verification fails or an asset is missing. The old `upload-assets.sh` fails all four, a version that signs after uploading fails the last two. - `hack/ci-tool-installs-contract.bats`, 8 tests. Putting `download-syft` back, dropping the `sha256sum -c` line, letting `SYFT_VERSION` drift between the workflows, dropping `cosign-release` from one step, letting it drift, unpinning the installer or giving one step another installer commit each fails it. - Putting the old any-ref regexp back in the script fails the identity test and the docs tie; putting it back in the docs alone fails the tie. - The meta-suites `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage` and `bats-runner-contract` pass: 90 tests across these eight files. - The pinned regexp gives the same accept/reject results under `grep -E` and Go's `regexp` (what cosign uses). cosign-installer's install script at the pinned commit, run as written with `cosign-release: v3.1.3`, verifies the binary's keyless and KMS bundles and installs v3.1.3 (its SHA-256 matches `cosign_checksums.txt`). `--yes`, `--bundle`, `--certificate-identity-regexp` and `--certificate-oidc-issuer` are listed, not deprecated, in cosign v3.1.3's help. - The `Set up syft` step's script, run as written with `RUNNER_TEMP`/`GITHUB_PATH` set: the checksum passes and `syft version` reports 1.54.1. A real syft v1.54.1 run through `hack/sign-release-assets.sh` (cosign stubbed) over this tree and the v1.6.5 `cozypkg-linux-amd64.tar.gz` took 10 s and wrote an SPDX 2.3 document with 920 packages: 440 golang, 373 npm, 103 github, 4 without a purl; 60 come from the cozypkg binary. - cosign behaviour behind the docs, with a key-signed legacy bundle whose `cert` holds the bare public key, against the documented identity regexp (tlog ignored, since an attacker can upload their own entry): v2.6.4 and v3.1.2 print "Verified OK", v2.6.5 and v3.1.3 reject it. `--new-bundle-format` makes v2.6.4 reject it but not v3.1.2, so the docs give the minimum version and not the flag. A real keyless protobuf bundle (syft's) verifies on v2.6.5 and v3.1.3. - zizmor 1.26.1 `--offline` reports no findings. actionlint 1.7.12 (with `.github/actionlint.yaml`) reports the same 13 pre-existing info-level shellcheck notes in `tags.yaml`/`promote-rc.yaml` as before, nothing new. shellcheck passes on both scripts. `hack/check-commit-trailers.sh` passes. - Not checked: a real keyless signature. It needs the OIDC token of a job in this repository, so the first rc after the merge is the end-to-end test. Cutting an `-alpha.N` first would exercise it without touching an rc. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: I checked the trigger map file by file. The only entry `hack/upload-assets.sh` can trigger is renaming a release asset, because the website install pages hardcode asset names. This PR adds three assets and renames none of the released ones, `cozypkg-checksums.txt` included (the SBOM was renamed to `cozystack-source-sbom.spdx.json` in review, before any release carried it). The workflow and docs changes match no entry. ### Release note ```release-note feat(release): releases carry cozystack-source-sbom.spdx.json, an SPDX SBOM of the source tree and the cozypkg binaries (not of the container or Talos images), cozystack-checksums.txt with the SHA-256 of each asset, and cozystack-checksums.txt.sigstore.json, a keyless Sigstore signature over the checksums made by the release workflow; docs/release.md shows how to verify a download with cosign v3.1.3 or later. ``` Refs #4759 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Releases now include an SPDX software bill of materials, SHA-256 checksums, and a keyless Sigstore signature bundle. * Release assets are verified against the expected workflow identity before upload; uploads stop if verification fails. * **Documentation** * Added guidance on release signatures, SBOM coverage, asset verification, and which branch dispatches release workflows. * **Tests** * Added coverage for generated release files, upload behavior, and signing and verification failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Marian Koreniuk
|
bc24b588b2 |
feat(release): sign the release assets and attach a source SBOM
No release carried a signature or an SBOM, so users could verify none of the ISO, disk images, kernel, initramfs, manifests and cozypkg binaries they download, and the OpenSSF Scorecard Signed-Releases check and CLOMonitor's signed_releases and sbom checks failed. hack/upload-assets.sh now hands its asset list to the new hack/sign-release-assets.sh before the first upload. That script writes cozystack-source-sbom.spdx.json with syft over the committed tree and the cozypkg binaries, cozystack-checksums.txt with the SHA-256 of every asset and the SBOM, and cozystack-checksums.txt.sigstore.json with a keyless cosign signature over the checksums. The SBOM does not describe the container images or the Talos boot images, which is what its name says; image SBOMs are #2937. The script verifies the bundle against the identities of tags.yaml on a v* tag and promote-rc.yaml on main before returning, so a bad signature fails the job with nothing uploaded, and hack/upload-assets.bats pins the upload list and that order. The two jobs that call make upload_assets get id-token: write, cosign v3.1.3 from the SHA-pinned cosign-installer, and syft from its release tarball checked against a committed SHA-256, because anchore/sbom-action/download-syft runs an install script fetched at run time; hack/ci-tool-installs-contract.bats now covers both. docs/release.md describes the files, what the SBOM covers, the minimum cosign version to verify a download with, what the certificate identity proves, and that promote-rc.yaml is dispatched from main only. Refs #4759 Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
myasnikovdaniil
|
aed7fa896e |
docs(release): add changelog for v1.6.5 (#4940)
This PR adds the changelog for release `v1.6.5`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.6.5.md`.
|
||
|
Myasnikov Daniil
|
4814ccfed9 |
docs(release): note the PostgreSQL restart on upgrade in the v1.6.5 changelog
Clusters with a memory limit below 512Mi and no explicit shared_buffers get the new default on upgrade, which needs a PostgreSQL restart. The changelog described the fix but not the restart. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-by: LLM |
||
|
cozystack-ci[bot]
|
13f612d55e |
docs: add changelog for v1.6.5
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
myasnikovdaniil
|
8bd6d91dbb |
fix(backups): gate every default Strategy CR on its CRD (#4929)
## What this PR does Upgrade from v1.6.x to v1.7 leaves `backupstrategy-controller` HelmRelease in `UpgradeFailed`: `no matches for kind "Kafka" in version "strategy.backups.cozystack.io/v1alpha1"`, same for Rabbitmq and Redis, and none of the new strategy CRDs get created. CRDs of this chart are ordinary templates, and Helm maps every object of a release through the RESTMapper before applying any of them. Default Strategy CRs for kafka, rabbitmq and redis were gated only on the bucket name, so they render next to their not yet served CRDs, Build() fails, nothing is applied and every retry renders the same manifest. Same happens on a first install with external S3 (`provisionBucket: false`) for all older default strategies, bucket name comes from values there. MongoDB default strategy already has the `crdEstablished` gate, this PR adds it to all other default strategies. On an installed cluster nothing changes. Otherwise first revision creates the CRDs and skips the CRs, then DefaultObjectsGate forces the follow-up upgrade that renders them. CRDs stay templates so later CRD changes still reach the cluster on upgrade. Short explanation is added to `docs/operations/backup-classes.md`. To keep this from coming back there is `hack/backupstrategy-crd-gate.bats`: it renders the chart with the bucket gate open and no CRD served and fails on any Strategy CR from any template, so a new `strategy-*-default.yaml` without the gate goes red without anyone updating a list. It also checks that every CR gates on the CRD this chart ships for its kind. helm-unittest covers both halves of the gate for all 11 templates. Checked on a throwaway kind cluster: - v1.6.4 installed, then upgraded: current main fails with the error above, this branch passes with all 11 CRDs Established and the follow-up upgrade creates the rest - modified CRD is applied on a later upgrade - fresh install converges, and fresh install with external S3 converges here while main fails with "no matches for kind" for eight kinds Fixes #4925. Needs backport to release-1.7, both commits apply there cleanly. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(backups): backupstrategy-controller release no longer gets stuck in UpgradeFailed on upgrade from v1.6 and no longer fails a fresh install with external S3, strategy CRDs are created first and default strategies follow on the next reconcile ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Default backup strategies are now created only after their required resource types are available and the backup bucket is resolved. This helps prevent installation and upgrade issues, including when using externally managed S3 storage. * **Documentation** * Clarified when default backup strategies are created during installation and upgrades. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
f47bab7280 |
test(backups): fail CI on any default Strategy CR rendered without its CRD gate
The CRD gate has now been missed twice: MongoDB shipped without it, and Kafka, RabbitMQ and Redis followed after it existed. helm-unittest only checks the templates a case lists, so a new strategy-*-default.yaml that forgets the gate stays green until an upgrade wedges. Render the chart with the bucket gate open and no CRD served, and fail on any strategy.backups.cozystack.io object from any template. Also check that every rendered Strategy CR gates on the CRD this chart ships for its kind, because a gate naming the wrong CRD passes the first check and keeps the CR out of every live render. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
7204992452 |
fix(backups): gate every default Strategy CR on its CRD
The chart ships its strategy CRDs as ordinary templates, and Helm maps every document of a release through the RESTMapper before it applies any. A default Strategy CR gated only on the bucket name therefore fails Build() with "no matches for kind" whenever the bucket name resolves before the CRD is served, the release applies nothing, the CRDs are never created, and every retry renders the same manifest. That happens on the upgrade that first ships a kind: the Kafka, Rabbitmq and Redis CRs leave a v1.6 cluster's HelmRelease stuck in UpgradeFailed on the way to v1.7. It also happens on the first render of an external-S3 install (provisionBucket=false), where the bucket name comes from values and every older default CR renders next to its absent CRD. MongoDB already gates on crdEstablished for this reason. Give every default Strategy the same gate. On an installed cluster the CRDs are established and the render is unchanged; otherwise the first revision creates the CRDs and skips the CRs, and DefaultObjectsGate, which reads the cozy-default BackupClass routes, forces the follow-up upgrade that renders them once the kinds are served. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
11b19635e0 |
fix(monitoring): send alerts to every Alertmanager replica (#4889)
## What this PR does vmalert sends every alert to every Alertmanager replica instead of to the shared Service. The chart gave vmalert a single notifier, `http://vmalertmanager-alertmanager.<namespace>.svc:9093`. That Service is headless and resolves to the three Alertmanager pods, but each alert reached one replica at a time and moved to another one whenever vmalert reconnected. The Alertmanager README asks for the opposite. It says "Do not load balance traffic between Prometheus and its Alertmanagers" and to point the sender at every instance, because the cluster deduplicates notifications only when each replica receives each alert. What this looked like on three clusters running 1.6.4, read through the Alertmanager API of each pod. - On all three, every active alert sat on one replica and the other two had none. The counts were 0, 0 and 10 on the first, 0, 0 and 23 on the second, and 1, 0 and 9 on the third. - On the third one, which also has an email receiver, the active alert count of each replica changed 163 to 180 times in 24 hours. The three replicas sent 63, 67 and 62 emails in that window. Over the same week only three alert names with a severity routed to email fired there. An alert that leaves a replica expires there and goes out as resolved, and the replica that receives it next sends it as a new firing notification. So a single long-running alert keeps producing firing and resolved notifications, and a receiver that rejects concurrent duplicates sees them as conflicts. The change. - `vm/vmalert.yaml` renders one notifier per Alertmanager pod, `http://vmalertmanager-alertmanager-<i>.vmalertmanager-alertmanager.<namespace>.svc:9093`. That is the address the operator builds for StatefulSet pods (`VMAlertmanager.asPodFQDN`), and the replicas already reach each other through the same per-pod names in their `--cluster.peer` arguments. - The replica count comes from a new `monitoring.alertmanager.replicas` helper, which now also sets `spec.replicaCount` of the VMAlertmanager. The notifier list cannot drift from the number of pods. Why not the operator's notifier selector. `VMAlertNotifierSpec.selector` expands matched VMAlertmanagers into the same per-pod addresses, but only when the VMAlert is reconciled. The VMAlert controller does not watch VMAlertmanager objects, so a VMAlert reconciled before its VMAlertmanager exists, which can happen on a fresh install, would get no notifier until something else triggers a reconcile. A static list renders the same addresses without depending on that order. Upgrade impact. The VMAlertmanager is unchanged, it renders `replicaCount: 3` as before. The VMAlert gets new notifier arguments, so the operator restarts the vmalert pod once. Testing. - `helm unittest` passes for the monitoring chart, 24 suites and 304 tests. The new suite `tests/vmalert_notifiers_test.yaml` checks the three notifier addresses, the replica count of the VMAlertmanager and the release namespace in the addresses. - Each of these breakages turns at least one case of the new suite red. The notifier back on the Service address, the helper set to 2, a hardcoded namespace in the address, and a hardcoded `replicaCount` that no longer follows the helper. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Walked the trigger map in `docs/agents/contributing.md` against the diff. It changes two templates and the helpers of `packages/system/monitoring` and adds a test file. No values, schema, README, package name, make target or `ApplicationDefinition` changes, so no entry applies. ### Release note ```release-note fix(monitoring): vmalert now sends alerts to every Alertmanager replica instead of the shared Service, so the Alertmanager cluster deduplicates them and a long-running alert no longer produces repeated firing and resolved notifications. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * vmalert now configures a notifier URL for each of the three Alertmanager replicas, rather than using a single shared URL. * Alertmanager’s replica count is managed consistently with the notifier configuration. * **Tests** * Added checks for the replica count, all three notifier URLs, and the configured namespace. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
e05c56524e |
fix(backups): prepare standalone VMDisk restore targets and add VM backup/restore e2e round-trips (#3427)
## What this PR does Closes the uncovered VM backup path from #3236. `hack/e2e-chainsaw/backup/` held only a disabled Chainsaw port of the old `backup.bats`, written against the retired API (`kind: VirtualMachine`, `BackupJob.storageRef`/`strategyRef`, `velero-strategy-default`), so it guarded nothing. This removes it, fixes the in-place restore of a standalone VMDisk in the backup controller, and adds two real round-trips in the `vminstance` suite that drive the `examples/backups/{vmi,vmdisk}` scripts, so the documented demo and the automated test stay in step. ### Controller: standalone VMDisk in-place restore `collectUnderlyingResources` in `internal/backupcontroller/velerostrategy_controller.go` returned nothing for any kind but VMInstance, so `prepareForRestore` did nothing for a standalone VMDisk: no HelmRelease suspend, no DataVolume delete, no `keepOriginalPVC` rename. An in-place VMDisk restore then raced Flux and CDI over the PVC name the data mover was repopulating. A standalone VMDisk now reports its own disk as its underlying resource, so the restore preparation already used for VMInstance disks suspends the `vm-disk-<name>` HelmRelease, deletes its DataVolume and renames the live PVC to `<name>-orig-<hash>` before Velero recreates it. Unit tests cover both the VMDisk and the VMInstance branch. A VMInstance is halted before its disks are touched, but a VMDisk target is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a disk that a running VM mounts would leave the original `Terminating` behind `pvc-protection`; Velero's PVC restore action skips a PVC that still exists, so the RestoreJob would report `Succeeded` having moved nothing, and the VM would lose its disk on the next restart. An in-place VMDisk restore therefore fails before changing anything when a Pod that has not terminated mounts the disk's PVC. The message names the Pod and tells the user to stop the VM or restore the VMInstance instead. `TestPrepareForRestore_VMDiskInUse` covers a running mounting Pod (refused, PVC not renamed, HelmRelease not suspended), a finished Pod, a Pod on another disk, and a VMInstance target whose own launcher Pod must not refuse it. A VMDisk Backup taken before the disk was recorded as its underlying resource has none, so the disk is derived from the application name (`vm-disk-<name>`): such a restore is prepared and checked for a mounting Pod too. The per-disk `orLabelSelectors` stay a VMInstance concern: a VMDisk strategy already selects its disk, and Velero rejects a Backup or Restore carrying both `labelSelector` and `orLabelSelectors`, so appending them for a VMDisk would break a strategy written with `labelSelector`. When an in-place restore fails after preparation, nothing is rolled back. Resuming the disk's HelmRelease while its PVC is renamed and its DataVolume deleted would have CDI import a fresh disk under the original name, which looks healthy and hides that the data sits in the `-orig-` PVC. The failure message and a Warning event instead name the HelmReleases preparation suspended and the `-orig-` PVCs holding the original data, read back from the cluster so that a failure before a step ran does not name what it would have left, and say that a HelmRelease Velero already restored is resumed, so the disk has to be checked; `examples/backups/vmdisk/README.md` describes the recovery. Tests cover both selector paths, the pre-upgrade Backup, a failed Pod list (the restore requeues and suspends nothing), and all three failures after preparation: a `PartiallyFailed` Velero Restore, several Restores owning the job, and a Restore that cannot be created. Because preparation takes the target apart before Velero validates the Restore, the RestoreJob is refused before preparation, with nothing changed, when the Velero Backup is gone (expired or deleted) or not `Completed`. This is the same single check #4672 added, now run on every pass behind the preparation failure path, so a Backup that expires after preparation started names the leftovers too. A Velero Restore that ends in `FailedValidation` fails the RestoreJob like `Failed`, with Velero's validation errors and the leftovers in the message, instead of polling forever. A HelmRelease suspend or a VM halt that does not happen now stops preparation (requeue) rather than letting the rename and DataVolume delete run under a live release or VM, and a VMInstance's disks are renamed only once no Pod mounts them. A VMDisk held only by a Pod already being deleted is waited for rather than refused, for up to 30 minutes after the RestoreJob started, so a Pod stuck terminating on a lost node fails the job instead of keeping it `Running`. Each of these is pinned by a test that turns red when the guard is removed. Between renaming a kept disk's PVC and deleting its DataVolume, CDI can recreate an empty PVC under the original name for the still-present DataVolume. Velero then skips that PVC as already present ("PVC already exists. Skip restore this PVC."), the RestoreJob reports `Succeeded`, and the VM boots a blank disk. The round-trip's new DataDownload check caught this on a VMInstance restore, where the uid, label and `-orig-` checks had all passed. Once the DataVolume is gone (the wait #4672 added), preparation now deletes a PVC found under the original name (read uncached) before the Velero Restore is created; `TestPrepareForRestore_ClearsPVCRecreatedAfterRename` reproduces the recreation inside the DataVolume delete. Preparation spans several reconciles, so the RestoreJob records a `PreparationStarted` condition before it first changes the target; if recording it fails, preparation retries without changing anything, so a later failure cannot report leftovers that were never made. A refusal on a later pass (the Velero Backup expired, a VM started on the disk) then names what preparation left behind instead of saying nothing was changed, and a preparation still waiting 30 minutes after it started fails the same way instead of polling forever. The vmdisk script starts its consumer Pod only on a PVC that carries the restore label, so it cannot pin a PVC CDI recreated in the rename window. Now that a VMDisk target is prepared, `keepOriginalPVC: false` reaches it as well. Its DataVolume is deleted with orphan propagation, and the bound PVC is deleted and waited for before the Restore (the path #4672 added for VMInstance disks). Left in place, Velero would skip the PVC and the CDI webhook would refuse the DataVolume the resumed HelmRelease creates, leaving the disk's HelmRelease stuck. `TestPrepareForRestore_VMDiskWithoutKeepDeletesOriginalPVC` pins it. ### `vminstance-2-backup-roundtrip` (un-gated in CI) A tiny cirros guest (`u1.nano`, 2Gi, in-place restore only) runs the current API end to end: Velero strategy, BackupClass, a booted VMInstance, an ad-hoc `BackupJob` and an in-place `RestoreJob`, each waited to `Succeeded` and failing fast on `Failed`. The VM must come back `Running`, and its disk PVC must have been created by the restore: a new `metadata.uid`, Velero's `velero.io/restore-name` label, and the `vm-disk-<name>-orig-<hash>` PVC that the rename leaves behind. The label alone does not prove the data moved, because with `existingResourcePolicy: update` Velero also stamps it on a PVC it only patched, and the patch path never reaches the data mover. An in-guest signal is not available on this platform: the KubeVirt CR disables serial-console logging cluster-wide and cirros ships no qemu-guest-agent. ### `vmdisk-2-backup-roundtrip` (un-gated in CI) The disk-only companion: import a 5Gi Ubuntu disk, back it up, restore it in place relying on the controller fix above, and require the same three signals (new uid, restore label, `-orig-` PVC), a `Completed` DataDownload for the restore that wrote bytes, and a Ready HelmRelease. On a `WaitForFirstConsumer` StorageClass, such as the `local` class the CI lane uses, the Velero data mover waits until a scheduled Pod has selected a node for the restored PVC (`WaitPVCConsumed` in Velero's generic restore exposer). A VMInstance gets that from its virt-launcher Pod; a detached disk has none, so its `RestoreJob` stays `Running` until the 30m data-mover prepare timeout fails it. The script therefore starts a restricted-profile `pause` Pod on the new PVC once Velero has created it, standing in for the VM that would attach the disk, and removes it after the restore. `examples/backups/vmdisk/README.md` documents this for operators restoring a detached disk. ### Harness details - The disk `storageClass` is an override (default `replicated`). The Chainsaw steps pass the lane's `COZY_E2E_STORAGE_CLASS` (`local` on the DRBD-off PR gate). - Each harness stands up a throwaway in-cluster BSL (`cozy-e2e-vmi`/`cozy-e2e-vmdisk`) at `seaweedfs-s3.tenant-root:8333` with `insecureSkipTLSVerify` and `checksumAlgorithm: ""` (the #3236 fixes), reusing the platform bucket and `cozy-backups-creds` under a distinct prefix, because the platform `cozy-default` BSL derives the external S3 ingress endpoint, unroutable in the CI cluster. - The script waits are bounded so that their worst case stays under the 40m op timeout (38m for the VMDisk step), and a hung wait trips the script's own named timeout rather than a bare `signal: killed`. - On failure, `catch` also describes the Velero `Restore`, `DataDownload` and `DataUpload` objects in `cozy-velero`. - `skipDelete` plus a `finally` that resumes the suspended HelmReleases, reclaims the `-orig-` PVC and its LINSTOR volume (the PV is flipped back to `Delete` first), and drains the Velero backups on the throwaway BSL, branching on kubectl's exit status, before deleting it. When the drain is not confirmed, the BSL is kept and the step fails, so the Velero Backups are not orphaned behind a green step. - HelmRelease waits use the shared `wait_hr_ready`, which fails fast on `Stalled`. After the VMDisk restore the script first waits for `observedGeneration` to reach the restored spec's `generation`, because the `Ready=True` from before the suspend would otherwise satisfy the wait at once. The `vmi` example is CI-drivable: `run-all.sh`/`cleanup.sh` are non-interactive, the VM is waited to boot before the backup, `volumeSnapshotLocations` is dropped from the demo strategies (`snapshotMoveData` routes data through the BSL), and `BACKUP_STORAGE_LOCATION` defaults to `cozy-default`. `select-e2e.sh` maps `examples/backups/{vmi,vmdisk}` edits to the `vminstance` suite, except `07-restore-to-copy.sh`, which the suite skips (`SKIP_RESTORE_TO_COPY=1`) and so stays docs-only. A `packages/system/velero` change selects the `vminstance` suite as well, since these are the only Tests that run a Velero backup and restore. ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff (backup controller restore preparation for VMDisk, shell scripts under `examples/backups/`, Chainsaw tests, `hack/select-e2e.sh` routing, a deleted stale suite, one doc line): no `packages/apps|extra` add/rename, no `values.schema.json`/enum/default changes, no CRD/ApplicationDefinition/label/annotation/namespace/metric changes. The controller change alters what a VMDisk `RestoreJob` does in the cluster, not any API surface a downstream repository consumes. `cozystack/examples` holds Terraform for bare nodes, unrelated to the in-repo `examples/` tree. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(backups): an in-place RestoreJob for a standalone VMDisk now suspends the disk's HelmRelease, deletes its DataVolume and (with keepOriginalPVC) renames the original PVC before Velero restores it, instead of racing Flux and CDI over the same PVC; without keepOriginalPVC the original PVC is deleted so Velero recreates it from the backup. Such a restore fails before changing anything when a running Pod, such as the launcher of a VM that uses the disk, mounts the disk's PVC: stop the VM or restore the VMInstance instead. If such a restore fails after the disk was prepared, its message and a Warning event name the HelmRelease preparation suspended and the PVC holding the original data. On a WaitForFirstConsumer StorageClass the data mover still starts only once a Pod consumes the restored PVC, for example when the disk is attached to a VMInstance. An in-place restore with keepOriginalPVC no longer reports Succeeded over an empty PVC that CDI recreated under the disk's name while the original was being kept aside. Adds Chainsaw backup/restore round-trips for VMInstance and standalone VMDisk that run a real Velero backup and restore in CI. ``` |
||
|
Aleksei Sviridkin
|
588d8292e7 |
fix(seaweedfs): build SeaweedFS in-tree with a bounded S3 read-ahead (#4819)
## What this PR does Large objects served through a Bucket's S3 endpoint stop tens of megabytes into the download (cozystack/cozystack#3782). The cause is not TLS: the SeaweedFS S3 gateway is OOM-killed mid-response. Its read-ahead for a sequential GET is meant to stay four chunks in front of the reader, but SeaweedFS counts only newly started chunk downloads toward that window and re-runs the prefetch after every 256 KiB read, so the gateway pulls up to about a gigabyte of the object into memory and exceeds the 512Mi limit the S3 deployment gets from its resources preset. The client sees `unexpected eof` over HTTP/1.1 and `INTERNAL_ERROR` over HTTP/2, which is what the CDI importer reported. Upgrading does not help yet: 4.31, 4.45 and the latest release 4.48 all carry the defect. The defect is reported upstream in seaweedfs/seaweedfs#11672, and the fix with its regression test is merged there in seaweedfs/seaweedfs#11674, after 4.48. Until a SeaweedFS release carrying it is vendored, this PR builds SeaweedFS in-tree the way #3343 builds cilium-operator: `images/seaweedfs` downloads the 4.45 source archive and checks it against a pinned checksum, applies `patches/reader-cache-prefetch-window.diff`, runs the tests of the packages the patch touches (including the regression test the patch adds) once, compiles `weed` with the Go toolchain upstream's own builder uses for the tag, and lays it over the upstream `chrislusf/seaweedfs:4.45` image pinned by digest, so everything else in the image stays the upstream build. `weed version` reports `cozystack-patched` in the commit field. All SeaweedFS components run the in-tree image. `make update` re-pins the Dockerfile from the vendored chart: the version, the source checksum, the upstream image digest and the builder toolchain. It fails when the patch no longer applies, which is how the first release carrying the upstream fix will show that the patch is ready to drop. Renovate is disabled for that Dockerfile, since it would move those pins one at a time, and whitespace checks are off for the patch. Tests: - `tests/image_test.yaml` pins every component (master, volume, filer, S3) to the in-tree image with a digest. - `make test-image-version` fails when the Dockerfile's version drifts from the vendored chart's appVersion. - `make update-image`, which `make update` runs, was checked to be idempotent on the current chart, to restore deliberately corrupted pins, and to fail with a clear message when the patch context no longer matches. - `hack/build-matrix_test.bats` now expects seaweedfs to be a build unit next to objectstorage-controller. - The Go regression test inside the patch fails on unpatched 4.45 (read-ahead reaches 60 chunks past the reader) and passes with the patch; the image build runs it. Verified on a test cluster against an 840 MiB object, with every SeaweedFS component running the image this package builds and the S3 gateway at the production limits (512Mi, 1 CPU). Unpatched 4.31, 4.45 and 4.48 gateways were OOM-killed on every full GET. The patched gateway served nine full GETs (six sequential over HTTP/1.1 and HTTP/2, three in parallel) with matching checksums and no restarts; it read from the volume servers about as many bytes as it sent, its live heap after a GET was about 40 MiB, and three parallel GETs peaked at about 140 MiB. The image this PR pins reports `4.45 cozystack-patched` and was rerun through the same checks. Drop the patch, and the in-tree build with it, once a SeaweedFS release carrying the upstream fix is vendored. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(seaweedfs): stop the S3 gateway from being OOM-killed while serving large objects, which cut downloads (including CDI imports from a Bucket URL) off tens of megabytes in. SeaweedFS is now built in-tree with a patch that bounds the gateway's read-ahead. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * SeaweedFS now runs a Cozystack-built image based on version 4.45, pinned to a specific image digest. * **Bug Fixes** * Sequential small reads now keep read-ahead within the configured prefetch window, preventing it from extending unexpectedly. * **Build & Maintenance** * SeaweedFS image updates are aligned with the chart version and verify the source archive, builder image, and patches before building. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
58b4343e5a |
ci(hack): route examples/backups/{vmi,vmdisk} edits to the vminstance suite
The two backup round-trips run the examples/backups/{vmi,vmdisk} scripts as their
harness, but select-e2e.sh mapped an examples/backups/<dir> edit to a suite only
when <dir> was itself a suite name. The dirs are vmi/vmdisk while the suite is
vminstance, so a harness-only edit selected no suite and the round-trip that runs
it never ran. Translate both dirs to vminstance, matching the vm-disk-application
entry in src_to_suites, and pin it with a bats case. The suite runs the vmi demo
with SKIP_RESTORE_TO_COPY=1, so 07-restore-to-copy.sh stays docs-only: selecting
the suite for it would pass without executing it. Both round-trips write to
the in-cluster seaweedfs-s3 endpoint, so a seaweedfs change selects the
vminstance suite as well. Also correct the controller-
owned-teardown note in docs/agents/e2e-testing.md (only Backup/RestoreJob
finalizers reach Velero; a BackupJob has none).
The two round-trips are also the only Tests that run a Velero backup and
restore, so the velero PackageSource maps to the vminstance suite; it
otherwise maps to a velero suite that does not exist, and a Velero change
ran neither round-trip.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey Kolkov
|
9cc3a9e459 |
test(vmi): add VMInstance and standalone VMDisk backup/restore e2e round-trips
The disabled hack/e2e-chainsaw/backup/ suite was a Chainsaw port of the retired
backup.bats, written against the old backup API, so the VMInstance and standalone
VMDisk Velero paths had no e2e coverage. Replace it with two round-trips in the
vminstance suite that drive the validated examples/backups/{vmi,vmdisk} scripts,
so the documented demo and the automated test stay in step.
Both run un-gated in CI:
- vminstance-2-backup-roundtrip boots a tiny cirros VM (u1.nano, 2Gi), backs it
up via BackupClass + BackupJob, restores in place, and asserts the VM returns
Running off a disk PVC the restore created: a new uid, Velero's restore-name
label and the -orig- PVC keepOriginalPVC leaves behind. The label alone is not
enough, because existingResourcePolicy: update also stamps it on a PVC Velero
only patched, which moves no data. A guest-side boot proof is unavailable: the
KubeVirt CR disables serial-console logging and cirros ships no guest-agent.
- vmdisk-2-backup-roundtrip imports a 5Gi disk, backs it up, and restores it in
place with the same three assertions. The controller prepares the target
(suspend HelmRelease, delete DataVolume, rename PVC). On a WaitForFirstConsumer
class the Velero data mover waits for a Pod to select a node for the restored
PVC, which a detached disk never gets, so the script starts a pause Pod on the
new PVC, standing in for the VM that would attach it.
The disk storageClass is an override (default replicated; the CI steps pass the
lane's COZY_E2E_STORAGE_CLASS, 'local' on the DRBD-off gate) and the script waits
are bounded so their worst case stays under the 40m op timeout, sized to keep the
whole parallel:1 suite inside the 215m job cap. On failure the catch also dumps
the Velero Restore, DataDownload and DataUpload objects. Each harness stands up a
throwaway in-cluster BSL (the platform cozy-default BSL derives the external S3
endpoint, unroutable in kind); its finally resumes the suspended HelmReleases,
reclaims the keepOriginalPVC orig PVC and its LINSTOR volume, and drains the
Velero backups before deleting the BSL. The example scripts are non-interactive
so the harness can drive them.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey Kolkov
|
9af1c0f902 |
fix(backups): prepare standalone VMDisk restore targets before the data mover runs
collectUnderlyingResources returned nothing for a non-VMInstance kind, so prepareForRestore did nothing for a standalone VMDisk: no HelmRelease suspend and no DataVolume delete. An in-place VMDisk restore then raced Flux/CDI — the restored HelmRelease reinstalled and CDI recreated the DataVolume over the same PVC name the data mover was repopulating, wedging the RestoreJob. Report a standalone VMDisk's own disk as its underlying resource, so the existing (VMInstance-proven) restore preparation suspends the vm-disk-<name> HelmRelease, deletes its DataVolume and — with keepOriginalPVC — renames the live PVC before the data mover recreates and repopulates it. No VM, so no IP/MAC is collected. A VMInstance is halted before its disks are touched; a VMDisk target is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a disk a running VM mounts leaves the original Terminating behind pvc-protection, Velero skips a PVC that still exists, and the RestoreJob would report Succeeded having moved nothing while the VM loses its disk on the next restart. An in-place VMDisk restore now fails before changing anything when a Pod that has not terminated mounts the disk's PVC, naming the Pod and telling the user to stop the VM or restore the VMInstance instead. A VMDisk Backup taken before the disk was recorded as its underlying resource has none, so its disk is now derived from the application name and such a restore is prepared and checked for a mounting Pod as well. The per-disk orLabelSelectors stay a VMInstance concern. A VMDisk strategy already selects its disk, and Velero rejects a Backup or Restore that carries both labelSelector and orLabelSelectors, so appending them for a VMDisk would break a strategy written with labelSelector. When an in-place restore fails after preparation, nothing is rolled back: resuming the disk's HelmRelease while its PVC is renamed and its DataVolume deleted would have CDI import a fresh disk under the original name, which looks healthy and hides that the data sits in the -orig- PVC. The failure message and a Warning event now name the suspended HelmReleases and the -orig- PVCs so the user can recover them, read back from the cluster, so a failure before a step ran does not name what it would have left. Preparation takes the target apart before Velero validates the Restore, so it is refused up front when the Velero Backup is gone or not Completed. That check now runs on every pass behind the same failure path as the rest of preparation, and a FailedValidation Restore fails the RestoreJob like a Failed one instead of leaving it Running. A HelmRelease suspend or a VM halt that did not happen now stops preparation instead of renaming and deleting under a running release or VM, and a VMInstance's disks are renamed only once its launcher Pod has let go of them. A VMDisk held only by a Pod already being deleted is waited for rather than refused, for up to 30 minutes after the RestoreJob started. Between renaming a kept disk's PVC and deleting its DataVolume, CDI can recreate an empty PVC under the original name. Velero then skips the PVC as already present, the RestoreJob reports Succeeded and the VM boots a blank disk. Once the DataVolume is gone, preparation now deletes a PVC found under the original name, read uncached, before the Velero Restore is created. Preparation spans several reconciles, so the RestoreJob records a PreparationStarted condition before it first changes the target, and retries without changing anything when recording it fails. A refusal on a later pass (the Velero Backup expired, a VM started on the disk) then names what preparation left behind instead of claiming that nothing changed, and a preparation still waiting 30 minutes after it started fails the same way instead of polling forever. Now that a VMDisk target is prepared, keepOriginalPVC=false reaches it too: its DataVolume is deleted with orphan propagation and the bound PVC is deleted before the Restore, as for a VMInstance disk. Left in place, Velero would skip it and CDI would refuse the DataVolume the resumed HelmRelease creates. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |